You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by eo...@apache.org on 2021/05/13 07:45:54 UTC
[pulsar] branch 2.7.2_ds_tmp created (now e7670be)
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a change to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git.
at e7670be Upgrade Lombok to 1.18.20 (#10259)
This branch includes the following new commits:
new 37d2dfa Start 2.7.2_1.0.0 Luna Streaming Disto Fork
new 65a1acd auth token for debezium and kafka connect adaptor
new d6f9538 upgrade presto version to 334 and JDK11
new 6761fcb update LICENSE for presto 334 upversion
new e3081b3 Issue 9130: Pulsar-admin sinks create: bad error message "java.lang.NullPointerException: path is 'null'." in case of missing "--name" parameter
new 65f744f suppress nashorn warning in JDK11 runtime
new 759c995 update ds distro required io connectors
new 830d798 Switch Docker image to Java 11
new c9e071c Update path for TTL config in Java 11
new b56a6d8 Always return from trigger even if read from output topic times out
new b30ee3d Update command descriptions from old 'property/cluster/namespace' format to current 'tenant/namespace' format
new e03a65f Update "sample" tenant on standalone to stop using old property/cluster/namespace naming convention.
new 186e63b Changing default function subscription to be function name, not FQFN
new 421fd82 [pulsar-functions] enhance kubernetes manifest customizer with default options (#9445)
new 01405a5 Fix pom file
new 98d4e75 remove underscore from version, it is not valid for deb package
new 4070e67 Fix test case for Pulsar Auth Token in Kafka Adapter Config
new 0550df6 More fixes about running tests on JDK11 (#9893)
new 56e127d [Branch-2.7] Fix deadlock on Monitoring thread blocked by LeaderService.isLeader() (#10512)
new e275813 Bump Debezium version to latest in series
new 5bc23c6 Enable Conscrypt for Jetty in the Broker and in the Proxy (#10541)
new 4d1a8d7 Fix checkstyle
new 6fe93f5 Add presto password authenticators plugin to enable password auth in Pulsar SQL
new 23b2859 Implement GenericObject - Allow GenericRecord to wrap any Java Object (#10057)
new ab3caca GenericObject - support KeyValue in Message#getValue() (#10107)
new b116efd Pulsar IO: Allow to develop Sinks that support Schema but without setting it at build time (Sink<GenericObject>) (#10034)
new 72295b6 GenericObject: handle KeyValue with SEPARATED encoding and add more tests (#10186)
new ec3ad85 Sink<GenericObject> unwrap internal AutoConsumeSchema and allow to handle topics with KeyValue schema (#10211)
new ff59244 Java Client: MessageImpl - ensure that AutoConsumeSchema downloaded the schema before decoding the payload (#10248)
new ea5b676 [Security] Upgrade vertx to 3.9.7, addresses CVE-2018-12541 (#10261)
new fe980e7 Add JsonRecordBuilder implementation (#10052)
new 1264c47 [Java client] Fix behaviour of Schema.AUTO_CONSUME() with KeyValueSchema and multi versions (#10492)
new 2932b10 Add Schema.getNativeSchema (#10076)
new b1ee56c Add JsonRecordBuilder implementation (#10052)
new 56c2f6b cleaned some code in GenericJsonRecord (#10527)
new 4acd720 PIP-85 Add Schema Information to Message in Java Client API (#10476)
new 32e3bb8 Fix some conflicts
new c73cdae AutoConsumeSchema: handle schema NONE as BYTES (#10277)
new 53e8c3e fix build
new 7e57cbe Add streaming dispatcher. (#9056)
new a1e1388 [pulsar-broker] Allow broker to discover and unblock stuck subscription (#9789)
new 83d10a1 [Broker] Make Persistent*DispatcherMultipleConsumers.readMoreEntries synchronized (#10413)
new a26de54 [pulsar-broker] Dispatch messaages to consumer with permits (#10417)
new e2c5245 Made OpAddEntry.toString() more robust to nulls to prevent NPEs (#10548)
new 567d077 Fix build
new e7670be Upgrade Lombok to 1.18.20 (#10259)
The 46 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
[pulsar] 31/46: Add JsonRecordBuilder implementation (#10052)
Posted by eo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit fe980e72bd4d188bbbd018551fd7694f571d7b36
Author: Vincent Royer <vr...@strapdata.com>
AuthorDate: Tue Apr 20 04:07:37 2021 +0200
Add JsonRecordBuilder implementation (#10052)
Provide a JSON GenericRecord builder allowing to produce JsonGenericRecord from the GenericJsonSchema.
Add a new class org.apache.pulsar.client.impl.schema.generic.JsonRecordBuilderImpl
Modify the org.apache.pulsar.client.impl.schema.generic.GenericJsonSchema.newRecordBuilder()
Add a unit test in org.apache.pulsar.client.impl.schema.JSONSchemaTest
(cherry picked from commit 08cbdc7c6e0df236d4ee2ab708ea1500063cb212)
---
.../client/impl/schema/FieldSchemaBuilderImpl.java | 7 ++
.../impl/schema/generic/GenericJsonSchema.java | 16 ++-
.../impl/schema/generic/JsonRecordBuilderImpl.java | 108 +++++++++++++++++++
.../pulsar/client/impl/schema/JSONSchemaTest.java | 117 +++++++++++++++++++++
4 files changed, 247 insertions(+), 1 deletion(-)
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/FieldSchemaBuilderImpl.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/FieldSchemaBuilderImpl.java
index 88283bd..eb4df16 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/FieldSchemaBuilderImpl.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/FieldSchemaBuilderImpl.java
@@ -142,6 +142,13 @@ class FieldSchemaBuilderImpl implements FieldSchemaBuilder<FieldSchemaBuilderImp
case TIMESTAMP:
baseSchema = LogicalTypes.timestampMillis().addToSchema(Schema.create(Schema.Type.LONG));
break;
+ case JSON:
+ checkArgument(genericSchema.getSchemaInfo().getType() == SchemaType.JSON,
+ "The field is expected to be using JSON schema but "
+ + genericSchema.getSchemaInfo().getType() + " schema is found");
+ AvroBaseStructSchema genericJsonSchema = (AvroBaseStructSchema) genericSchema;
+ baseSchema = genericJsonSchema.getAvroSchema();
+ break;
case AVRO:
checkArgument(genericSchema.getSchemaInfo().getType() == SchemaType.AVRO,
"The field is expected to be using AVRO schema but "
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericJsonSchema.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericJsonSchema.java
index 87a1790..24b7777 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericJsonSchema.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericJsonSchema.java
@@ -19,6 +19,8 @@
package org.apache.pulsar.client.impl.schema.generic;
import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.schema.GenericRecord;
import org.apache.pulsar.client.api.schema.GenericRecordBuilder;
import org.apache.pulsar.common.schema.SchemaInfo;
@@ -41,6 +43,18 @@ public class GenericJsonSchema extends GenericSchemaImpl {
@Override
public GenericRecordBuilder newRecordBuilder() {
- throw new UnsupportedOperationException("Json Schema doesn't support record builder yet");
+ return new JsonRecordBuilderImpl(this);
+ }
+
+ public boolean supportSchemaVersioning() {
+ return true;
+ }
+
+ public Schema<GenericRecord> clone() {
+ Schema<GenericRecord> schema = of(this.schemaInfo, ((AbstractMultiVersionGenericReader)this.reader).useProvidedSchemaAsReaderSchema);
+ if (this.schemaInfoProvider != null) {
+ schema.setSchemaInfoProvider(this.schemaInfoProvider);
+ }
+ return schema;
}
}
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/JsonRecordBuilderImpl.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/JsonRecordBuilderImpl.java
new file mode 100644
index 0000000..39249e7
--- /dev/null
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/JsonRecordBuilderImpl.java
@@ -0,0 +1,108 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.impl.schema.generic;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.pulsar.client.api.schema.Field;
+import org.apache.pulsar.client.api.schema.GenericRecord;
+import org.apache.pulsar.client.api.schema.GenericRecordBuilder;
+
+import java.util.HashMap;
+import java.util.Map;
+
+public class JsonRecordBuilderImpl implements GenericRecordBuilder {
+
+ private static ObjectMapper objectMapper = new ObjectMapper();
+
+ private final GenericSchemaImpl genericSchema;
+ private Map<String, Object> map = new HashMap<>();
+
+ public JsonRecordBuilderImpl(GenericSchemaImpl genericSchema) {
+ this.genericSchema = genericSchema;
+ }
+
+ /**
+ * Sets the value of a field.
+ *
+ * @param fieldName the name of the field to set.
+ * @param value the value to set.
+ * @return a reference to the RecordBuilder.
+ */
+ @Override
+ public GenericRecordBuilder set(String fieldName, Object value) {
+ if (value instanceof GenericRecord) {
+ if (!(value instanceof GenericJsonRecord))
+ throw new IllegalArgumentException("JSON Record Builder doesn't support non-JSON record as a field");
+ GenericJsonRecord genericJsonRecord = (GenericJsonRecord) value;
+ value = genericJsonRecord.getJsonNode();
+ }
+
+ map.put(fieldName, value);
+ return this;
+ }
+
+ /**
+ * Sets the value of a field.
+ *
+ * @param field the field to set.
+ * @param value the value to set.
+ * @return a reference to the RecordBuilder.
+ */
+ @Override
+ public GenericRecordBuilder set(Field field, Object value) {
+ set(field.getName(), value);
+ return this;
+ }
+
+ /**
+ * Clears the value of the given field.
+ *
+ * @param fieldName the name of the field to clear.
+ * @return a reference to the RecordBuilder.
+ */
+ @Override
+ public GenericRecordBuilder clear(String fieldName) {
+ map.remove(fieldName);
+ return this;
+ }
+
+ /**
+ * Clears the value of the given field.
+ *
+ * @param field the field to clear.
+ * @return a reference to the RecordBuilder.
+ */
+ @Override
+ public GenericRecordBuilder clear(Field field) {
+ clear(field.getName());
+ return this;
+ }
+
+ @Override
+ public GenericRecord build() {
+ JsonNode jn = objectMapper.valueToTree(map);
+ return new GenericJsonRecord(
+ null,
+ genericSchema.getFields(),
+ jn,
+ null
+ );
+ }
+}
diff --git a/pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/JSONSchemaTest.java b/pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/JSONSchemaTest.java
index d17bbf0..172bae3 100644
--- a/pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/JSONSchemaTest.java
+++ b/pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/JSONSchemaTest.java
@@ -20,18 +20,30 @@ package org.apache.pulsar.client.impl.schema;
import java.util.Collections;
import java.util.List;
+import static org.apache.pulsar.client.impl.schema.SchemaTestUtils.FOO_FIELDS;
+import static org.apache.pulsar.client.impl.schema.SchemaTestUtils.SCHEMA_JSON_ALLOW_NULL;
+import static org.apache.pulsar.client.impl.schema.SchemaTestUtils.SCHEMA_JSON_NOT_ALLOW_NULL;
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertTrue;
+import static org.testng.AssertJUnit.assertSame;
import io.netty.buffer.ByteBuf;
import io.netty.buffer.ByteBufAllocator;
import lombok.extern.slf4j.Slf4j;
import org.apache.avro.Schema;
import org.apache.pulsar.client.api.SchemaSerializationException;
+import org.apache.pulsar.client.api.schema.Field;
+import org.apache.pulsar.client.api.schema.GenericRecord;
+import org.apache.pulsar.client.api.schema.RecordSchemaBuilder;
+import org.apache.pulsar.client.api.schema.SchemaBuilder;
import org.apache.pulsar.client.api.schema.SchemaDefinition;
import org.apache.pulsar.client.impl.schema.SchemaTestUtils.Bar;
import org.apache.pulsar.client.impl.schema.SchemaTestUtils.DerivedFoo;
import org.apache.pulsar.client.impl.schema.SchemaTestUtils.Foo;
import org.apache.pulsar.client.impl.schema.SchemaTestUtils.NestedBar;
import org.apache.pulsar.client.impl.schema.SchemaTestUtils.NestedBarList;
+import org.apache.pulsar.client.impl.schema.generic.GenericSchemaImpl;
+import org.apache.pulsar.common.schema.SchemaInfo;
import org.apache.pulsar.common.schema.SchemaType;
import org.skyscreamer.jsonassert.JSONAssert;
import org.testng.Assert;
@@ -332,4 +344,109 @@ public class JSONSchemaTest {
assertEquals(jsonSchema.decode(byteBuf), foo1);
}
+
+ @Data
+ @AllArgsConstructor
+ @NoArgsConstructor
+ private static class Seller {
+ public String state;
+ public String street;
+ public long zipCode;
+ }
+
+ @Data
+ @AllArgsConstructor
+ @NoArgsConstructor
+ private static class PC {
+ public String brand;
+ public String model;
+ public int year;
+ public GPU gpu;
+ public Seller seller;
+ }
+
+ private enum GPU {
+ AMD, NVIDIA
+ }
+
+ @Test
+ public void testEncodeAndDecodeObject() throws JsonProcessingException {
+ JSONSchema<PC> jsonSchema = JSONSchema.of(SchemaDefinition.<PC>builder().withPojo(PC.class).build());
+ PC pc = new PC("dell", "alienware", 2021, GPU.AMD,
+ new Seller("WA", "street", 98004));
+ byte[] encoded = jsonSchema.encode(pc);
+ PC roundtrippedPc = jsonSchema.decode(encoded);
+ assertEquals(roundtrippedPc, pc);
+ }
+
+ @Test
+ public void testGetNativeSchema() throws SchemaValidationException {
+ JSONSchema<PC> schema2 = JSONSchema.of(PC.class);
+ org.apache.avro.Schema avroSchema2 = (Schema) schema2.getNativeSchema().get();
+ assertSame(schema2.schema, avroSchema2);
+ }
+
+ @Test
+ public void testJsonGenericRecordBuilder() {
+ JSONSchema<Seller> sellerJsonSchema = JSONSchema.of(Seller.class);
+
+ RecordSchemaBuilder sellerSchemaBuilder = SchemaBuilder.record("seller");
+ sellerSchemaBuilder.field("state").type(SchemaType.STRING);
+ sellerSchemaBuilder.field("street").type(SchemaType.STRING);
+ sellerSchemaBuilder.field("zipCode").type(SchemaType.INT64);
+ SchemaInfo sellerSchemaInfo = sellerSchemaBuilder.build(SchemaType.JSON);
+ GenericSchemaImpl sellerGenericSchema = GenericSchemaImpl.of(sellerSchemaInfo);
+
+ JSONSchema<PC> pcJsonSchema = JSONSchema.of(PC.class);
+
+ RecordSchemaBuilder pcSchemaBuilder = SchemaBuilder.record("pc");
+ pcSchemaBuilder.field("brand").type(SchemaType.STRING);
+ pcSchemaBuilder.field("model").type(SchemaType.STRING);
+ pcSchemaBuilder.field("gpu").type(SchemaType.STRING);
+ pcSchemaBuilder.field("year").type(SchemaType.INT64);
+ pcSchemaBuilder.field("seller", sellerGenericSchema).type(SchemaType.JSON).optional();
+ SchemaInfo pcGenericSchemaInfo = pcSchemaBuilder.build(SchemaType.JSON);
+ GenericSchemaImpl pcGenericSchema = GenericSchemaImpl.of(pcGenericSchemaInfo);
+
+ Seller seller = new Seller("USA","oakstreet",9999);
+ PC pc = new PC("dell","g3",2020, GPU.AMD, seller);
+
+ byte[] bytes = pcJsonSchema.encode(pc);
+ Assert.assertTrue(bytes.length > 0);
+
+ Object pc2 = pcJsonSchema.decode(bytes);
+ assertEquals(pc, pc2);
+
+ GenericRecord sellerRecord = sellerGenericSchema.newRecordBuilder()
+ .set("state", "USA")
+ .set("street", "oakstreet")
+ .set("zipCode", 9999)
+ .build();
+
+ GenericRecord pcRecord = pcGenericSchema.newRecordBuilder()
+ .set("brand", "dell")
+ .set("model","g3")
+ .set("year", 2020)
+ .set("gpu", GPU.AMD)
+ .set("seller", sellerRecord)
+ .build();
+
+ byte[] bytes3 = pcGenericSchema.encode(pcRecord);
+ Assert.assertTrue(bytes3.length > 0);
+ GenericRecord pc3Record = pcGenericSchema.decode(bytes3);
+
+ for(Field field : pc3Record.getFields()) {
+ assertTrue(pcGenericSchema.getFields().contains(field));
+ }
+ assertEquals("dell", pc3Record.getField("brand"));
+ assertEquals("g3", pc3Record.getField("model"));
+ assertEquals(2020, pc3Record.getField("year"));
+ assertEquals(GPU.AMD.toString(), pc3Record.getField("gpu"));
+
+
+ GenericRecord seller3Record = (GenericRecord) pc3Record.getField("seller");
+ assertEquals("USA", seller3Record.getField("state"));
+ assertEquals("oakstreet", seller3Record.getField("street"));
+ assertEquals(9999, seller3Record.getField("zipCode"));
+ }
}
[pulsar] 21/46: Enable Conscrypt for Jetty in the Broker and in the
Proxy (#10541)
Posted by eo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 5bc23c6aead321fa1c6142d96b4563a2243e108a
Author: Lari Hotari <lh...@users.noreply.github.com>
AuthorDate: Wed May 12 07:02:56 2021 +0300
Enable Conscrypt for Jetty in the Broker and in the Proxy (#10541)
* Add jetty-alpn-conscrypt-server jar
* Enable Conscrypt / OpenSSL provider if it is available
* Use conscrypt 2.5.2
* Set default hostname verifier for Conscrypt
* Move TlsHostnameVerifier for pulsar-client to pulsar-common
* Use Pulsar's TlsHostnameVerifier with Conscrypt
* Add conscrypt.version property for managing conscrypt version in pom.xml
* Update LICENSE file
* Add comments about HostnameVerifier in Conscrypt
---
distribution/server/src/assemble/LICENSE.bin.txt | 7 +-
pom.xml | 245 +++++++++++----------
pulsar-broker/pom.xml | 5 +
.../pulsar/broker/admin/AdminApiTlsAuthTest.java | 2 +-
.../AuthenticationTlsHostnameVerificationTest.java | 4 +-
.../client/impl/AdminApiKeyStoreTlsAuthTest.java | 2 +-
.../org/apache/pulsar/client/impl/ClientCnx.java | 4 +-
.../org/apache/pulsar/common}/tls/DomainType.java | 2 +-
.../pulsar/common}/tls/InetAddressUtils.java | 2 +-
.../pulsar/common}/tls/NoopHostnameVerifier.java | 2 +-
.../pulsar/common}/tls/PublicSuffixList.java | 2 +-
.../pulsar/common}/tls/PublicSuffixMatcher.java | 2 +-
.../org/apache/pulsar/common}/tls/SubjectName.java | 2 +-
.../pulsar/common}/tls/TlsHostnameVerifier.java | 2 +-
.../apache/pulsar/common/util/SecurityUtility.java | 117 +++++++++-
.../util/keystoretls/KeyStoreSSLContext.java | 16 +-
.../SslContextFactoryWithAutoRefresh.java | 3 +
pulsar-discovery-service/pom.xml | 5 +
pulsar-functions/worker/pom.xml | 5 +
pulsar-proxy/pom.xml | 5 +
.../pulsar/proxy/server/DirectProxyHandler.java | 2 +-
tiered-storage/file-system/pom.xml | 8 +-
22 files changed, 304 insertions(+), 140 deletions(-)
diff --git a/distribution/server/src/assemble/LICENSE.bin.txt b/distribution/server/src/assemble/LICENSE.bin.txt
index 56be02c..6808141 100644
--- a/distribution/server/src/assemble/LICENSE.bin.txt
+++ b/distribution/server/src/assemble/LICENSE.bin.txt
@@ -321,9 +321,10 @@ The Apache Software License, Version 2.0
- com.fasterxml.jackson.module-jackson-module-jaxb-annotations-2.11.1.jar
- com.fasterxml.jackson.module-jackson-module-jsonSchema-2.11.1.jar
* Caffeine -- com.github.ben-manes.caffeine-caffeine-2.6.2.jar
- * Proto Google Common Protos -- com.google.api.grpc-proto-google-common-protos-1.12.0.jar
* Joda -- org.joda-joda-convert-2.2.1.jar
- * Bitbucket -- org.bitbucket.b_c-jose4j-0.7.2.jar
+ * Conscrypt -- org.conscrypt-conscrypt-openjdk-uber-2.5.2.jar
+ * Proto Google Common Protos -- com.google.api.grpc-proto-google-common-protos-1.17.0.jar
+ * Bitbucket -- org.bitbucket.b_c-jose4j-0.7.6.jar
* Gson
- com.google.code.gson-gson-2.8.6.jar
- io.gsonfire-gson-fire-1.8.4.jar
@@ -447,6 +448,8 @@ The Apache Software License, Version 2.0
- org.eclipse.jetty.websocket-websocket-common-9.4.39.v20210325.jar
- org.eclipse.jetty.websocket-websocket-server-9.4.39.v20210325.jar
- org.eclipse.jetty.websocket-websocket-servlet-9.4.39.v20210325.jar
+ - org.eclipse.jetty-jetty-alpn-conscrypt-server-9.4.39.v20210325.jar
+ - org.eclipse.jetty-jetty-alpn-server-9.4.39.v20210325.jar
* SnakeYaml -- org.yaml-snakeyaml-1.26.jar
* RocksDB - org.rocksdb-rocksdbjni-6.10.2.jar
* Google Error Prone Annotations - com.google.errorprone-error_prone_annotations-2.3.4.jar
diff --git a/pom.xml b/pom.xml
index d415f8b..348ef7a 100644
--- a/pom.xml
+++ b/pom.xml
@@ -28,121 +28,123 @@
<version>18</version>
</parent>
<groupId>org.apache.pulsar</groupId>
- <artifactId>pulsar</artifactId>
- <version>2.7.2.1.0.0</version>
- <name>Pulsar</name>
- <description>Pulsar is a distributed pub-sub messaging platform with a very
- flexible messaging model and an intuitive client API.</description>
- <url>https://github.com/apache/pulsar</url>
-
- <organization>
- <name>Apache Software Foundation</name>
- <url>http://www.apache.org/</url>
- </organization>
- <inceptionYear>2017</inceptionYear>
-
- <developers>
- <developer>
- <organization>Apache Pulsar developers</organization>
- <organizationUrl>http://pulsar.apache.org/</organizationUrl>
- </developer>
- </developers>
-
- <licenses>
- <license>
- <name>Apache License, Version 2.0</name>
- <url>https://www.apache.org/licenses/LICENSE-2.0.txt</url>
- <distribution>repo</distribution>
- </license>
- </licenses>
-
- <scm>
- <url>https://github.com/apache/pulsar</url>
- <connection>scm:git:https://github.com/apache/pulsar.git</connection>
- <developerConnection>scm:git:ssh://git@github.com:apache/pulsar.git</developerConnection>
- </scm>
-
- <ciManagement>
- <system>Travis</system>
- <url>https://travis-ci.org/apache/pulsar</url>
- </ciManagement>
-
- <issueManagement>
- <system>Github</system>
- <url>https://github.com/apache/pulsar/issues</url>
- </issueManagement>
-
- <properties>
- <!--config keys to congiure test selection -->
- <include>*</include>
- <exclude/>
- <groups/>
- <excludedGroups/>
-
- <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
- <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
- <redirectTestOutputToFile>true</redirectTestOutputToFile>
- <testReuseFork>true</testReuseFork>
- <testForkCount>4</testForkCount>
- <testRealAWS>false</testRealAWS>
- <testRetryCount>1</testRetryCount>
- <docker.organization>apachepulsar</docker.organization>
-
- <!-- pin the protobuf-shaded version to make the pulsar build friendly to intellij -->
- <pulsar.protobuf.shaded.version>2.1.0-incubating</pulsar.protobuf.shaded.version>
-
- <!-- apache commons -->
- <commons-compress.version>1.19</commons-compress.version>
-
- <bookkeeper.version>4.12.0</bookkeeper.version>
- <zookeeper.version>3.5.7</zookeeper.version>
- <netty.version>4.1.60.Final</netty.version>
- <netty-tc-native.version>2.0.36.Final</netty-tc-native.version>
- <jetty.version>9.4.39.v20210325</jetty.version>
- <jersey.version>2.31</jersey.version>
- <athenz.version>1.10.9</athenz.version>
- <prometheus.version>0.5.0</prometheus.version>
- <aspectj.version>1.9.2</aspectj.version>
- <vertx.version>3.5.3</vertx.version>
- <rocksdb.version>6.10.2</rocksdb.version>
- <slf4j.version>1.7.25</slf4j.version>
- <commons.collections.version>3.2.2</commons.collections.version>
- <log4j2.version>2.10.0</log4j2.version>
- <bouncycastle.version>1.68</bouncycastle.version>
- <bouncycastlefips.version>1.0.2</bouncycastlefips.version>
- <jackson.version>2.11.1</jackson.version>
- <jackson.databind.version>2.11.1</jackson.databind.version>
- <reflections.version>0.9.11</reflections.version>
- <swagger.version>1.5.21</swagger.version>
- <puppycrawl.checkstyle.version>8.37</puppycrawl.checkstyle.version>
- <dockerfile-maven.version>1.4.13</dockerfile-maven.version>
- <typetools.version>0.5.0</typetools.version>
- <protobuf2.version>2.4.1</protobuf2.version>
- <protobuf3.version>3.11.4</protobuf3.version>
- <protoc3.version>${protobuf3.version}</protoc3.version>
- <grpc.version>1.18.0</grpc.version>
- <protoc-gen-grpc-java.version>${grpc.version}</protoc-gen-grpc-java.version>
- <gson.version>2.8.6</gson.version>
- <sketches.version>0.8.3</sketches.version>
- <hbc-core.version>2.2.0</hbc-core.version>
- <cassandra-driver-core.version>3.6.0</cassandra-driver-core.version>
- <aerospike-client.version>4.4.8</aerospike-client.version>
- <kafka-client.version>2.3.0</kafka-client.version>
- <rabbitmq-client.version>5.1.1</rabbitmq-client.version>
- <aws-sdk.version>1.11.774</aws-sdk.version>
- <avro.version>1.9.1</avro.version>
- <joda.version>2.10.1</joda.version>
- <jclouds.version>2.2.1</jclouds.version>
- <sqlite-jdbc.version>3.8.11.2</sqlite-jdbc.version>
- <mysql-jdbc.version>8.0.11</mysql-jdbc.version>
- <postgresql-jdbc.version>42.2.12</postgresql-jdbc.version>
- <clickhouse-jdbc.version>0.2.4</clickhouse-jdbc.version>
- <mariadb-jdbc.version>2.6.0</mariadb-jdbc.version>
- <hdfs-offload-version3>3.2.0</hdfs-offload-version3>
- <org.eclipse.jetty-hdfs-offload>9.3.24.v20180605</org.eclipse.jetty-hdfs-offload>
- <elasticsearch.version>7.9.1</elasticsearch.version>
- <presto.version>334</presto.version>
- <flink.version>1.6.0</flink.version>
+
+ <artifactId>pulsar</artifactId>
+ <version>2.7.2.1.0.0</version>
+ <name>Pulsar</name>
+ <description>Pulsar is a distributed pub-sub messaging platform with a very
+flexible messaging model and an intuitive client API.</description>
+ <url>https://github.com/apache/pulsar</url>
+
+ <organization>
+ <name>Apache Software Foundation</name>
+ <url>http://www.apache.org/</url>
+ </organization>
+ <inceptionYear>2017</inceptionYear>
+
+ <developers>
+ <developer>
+ <organization>Apache Pulsar developers</organization>
+ <organizationUrl>http://pulsar.apache.org/</organizationUrl>
+ </developer>
+ </developers>
+
+ <licenses>
+ <license>
+ <name>Apache License, Version 2.0</name>
+ <url>https://www.apache.org/licenses/LICENSE-2.0.txt</url>
+ <distribution>repo</distribution>
+ </license>
+ </licenses>
+
+ <scm>
+ <url>https://github.com/apache/pulsar</url>
+ <connection>scm:git:https://github.com/apache/pulsar.git</connection>
+ <developerConnection>scm:git:ssh://git@github.com:apache/pulsar.git</developerConnection>
+ </scm>
+
+ <ciManagement>
+ <system>Travis</system>
+ <url>https://travis-ci.org/apache/pulsar</url>
+ </ciManagement>
+
+ <issueManagement>
+ <system>Github</system>
+ <url>https://github.com/apache/pulsar/issues</url>
+ </issueManagement>
+
+ <properties>
+ <!--config keys to congiure test selection -->
+ <include>*</include>
+ <exclude/>
+ <groups/>
+ <excludedGroups/>
+
+ <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
+ <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
+ <redirectTestOutputToFile>true</redirectTestOutputToFile>
+ <testReuseFork>true</testReuseFork>
+ <testForkCount>4</testForkCount>
+ <testRealAWS>false</testRealAWS>
+ <testRetryCount>1</testRetryCount>
+ <docker.organization>apachepulsar</docker.organization>
+
+ <!-- pin the protobuf-shaded version to make the pulsar build friendly to intellij -->
+ <pulsar.protobuf.shaded.version>2.1.0-incubating</pulsar.protobuf.shaded.version>
+
+ <!-- apache commons -->
+ <commons-compress.version>1.19</commons-compress.version>
+
+ <bookkeeper.version>4.12.0</bookkeeper.version>
+ <zookeeper.version>3.5.7</zookeeper.version>
+ <netty.version>4.1.60.Final</netty.version>
+ <netty-tc-native.version>2.0.36.Final</netty-tc-native.version>
+ <jetty.version>9.4.39.v20210325</jetty.version>
+ <conscrypt.version>2.5.2</conscrypt.version>
+ <jersey.version>2.31</jersey.version>
+ <athenz.version>1.10.9</athenz.version>
+ <prometheus.version>0.5.0</prometheus.version>
+ <aspectj.version>1.9.2</aspectj.version>
+ <vertx.version>3.5.3</vertx.version>
+ <rocksdb.version>6.10.2</rocksdb.version>
+ <slf4j.version>1.7.25</slf4j.version>
+ <commons.collections.version>3.2.2</commons.collections.version>
+ <log4j2.version>2.10.0</log4j2.version>
+ <bouncycastle.version>1.68</bouncycastle.version>
+ <bouncycastlefips.version>1.0.2</bouncycastlefips.version>
+ <jackson.version>2.11.1</jackson.version>
+ <jackson.databind.version>2.11.1</jackson.databind.version>
+ <reflections.version>0.9.11</reflections.version>
+ <swagger.version>1.5.21</swagger.version>
+ <puppycrawl.checkstyle.version>8.37</puppycrawl.checkstyle.version>
+ <dockerfile-maven.version>1.4.13</dockerfile-maven.version>
+ <typetools.version>0.5.0</typetools.version>
+ <protobuf2.version>2.4.1</protobuf2.version>
+ <protobuf3.version>3.11.4</protobuf3.version>
+ <protoc3.version>${protobuf3.version}</protoc3.version>
+ <grpc.version>1.18.0</grpc.version>
+ <protoc-gen-grpc-java.version>${grpc.version}</protoc-gen-grpc-java.version>
+ <gson.version>2.8.6</gson.version>
+ <sketches.version>0.8.3</sketches.version>
+ <hbc-core.version>2.2.0</hbc-core.version>
+ <cassandra-driver-core.version>3.6.0</cassandra-driver-core.version>
+ <aerospike-client.version>4.4.8</aerospike-client.version>
+ <kafka-client.version>2.3.0</kafka-client.version>
+ <rabbitmq-client.version>5.1.1</rabbitmq-client.version>
+ <aws-sdk.version>1.11.774</aws-sdk.version>
+ <avro.version>1.9.1</avro.version>
+ <joda.version>2.10.1</joda.version>
+ <jclouds.version>2.2.1</jclouds.version>
+ <sqlite-jdbc.version>3.8.11.2</sqlite-jdbc.version>
+ <mysql-jdbc.version>8.0.11</mysql-jdbc.version>
+ <postgresql-jdbc.version>42.2.12</postgresql-jdbc.version>
+ <clickhouse-jdbc.version>0.2.4</clickhouse-jdbc.version>
+ <mariadb-jdbc.version>2.6.0</mariadb-jdbc.version>
+ <hdfs-offload-version3>3.2.0</hdfs-offload-version3>
+ <org.eclipse.jetty-hdfs-offload>9.3.24.v20180605</org.eclipse.jetty-hdfs-offload>
+ <elasticsearch.version>7.9.1</elasticsearch.version>
+ <presto.version>334</presto.version>
+ <flink.version>1.6.0</flink.version>
<scala.binary.version>2.11</scala.binary.version>
<scala-library.version>2.11.12</scala-library.version>
<debezium.version>1.0.3.Final</debezium.version>
@@ -422,6 +424,18 @@
<dependency>
<groupId>org.eclipse.jetty</groupId>
+ <artifactId>jetty-alpn-conscrypt-server</artifactId>
+ <version>${jetty.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.conscrypt</groupId>
+ <artifactId>conscrypt-openjdk-uber</artifactId>
+ <version>${conscrypt.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-servlet</artifactId>
<version>${jetty.version}</version>
</dependency>
@@ -1744,4 +1758,3 @@
</repositories>
</project>
-
diff --git a/pulsar-broker/pom.xml b/pulsar-broker/pom.xml
index c3c3dc5..b3209ec 100644
--- a/pulsar-broker/pom.xml
+++ b/pulsar-broker/pom.xml
@@ -184,6 +184,11 @@
<dependency>
<groupId>org.eclipse.jetty</groupId>
+ <artifactId>jetty-alpn-conscrypt-server</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-servlet</artifactId>
</dependency>
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApiTlsAuthTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApiTlsAuthTest.java
index b623725..75b4564 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApiTlsAuthTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApiTlsAuthTest.java
@@ -41,7 +41,7 @@ import org.apache.pulsar.client.admin.internal.JacksonConfigurator;
import org.apache.pulsar.client.api.Producer;
import org.apache.pulsar.client.api.PulsarClient;
import org.apache.pulsar.client.api.Schema;
-import org.apache.pulsar.client.impl.tls.NoopHostnameVerifier;
+import org.apache.pulsar.common.tls.NoopHostnameVerifier;
import org.apache.pulsar.common.policies.data.AuthAction;
import org.apache.pulsar.common.policies.data.ClusterData;
import org.apache.pulsar.common.policies.data.TenantInfo;
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/AuthenticationTlsHostnameVerificationTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/AuthenticationTlsHostnameVerificationTest.java
index cf04a4f..54a847f 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/AuthenticationTlsHostnameVerificationTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/AuthenticationTlsHostnameVerificationTest.java
@@ -32,8 +32,8 @@ import org.apache.pulsar.broker.authentication.AuthenticationProviderBasic;
import org.apache.pulsar.broker.authentication.AuthenticationProviderTls;
import org.apache.pulsar.client.admin.PulsarAdmin;
import org.apache.pulsar.client.impl.auth.AuthenticationTls;
-import org.apache.pulsar.client.impl.tls.PublicSuffixMatcher;
-import org.apache.pulsar.client.impl.tls.TlsHostnameVerifier;
+import org.apache.pulsar.common.tls.PublicSuffixMatcher;
+import org.apache.pulsar.common.tls.TlsHostnameVerifier;
import org.apache.pulsar.common.policies.data.ClusterData;
import org.apache.pulsar.common.policies.data.TenantInfo;
import org.slf4j.Logger;
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/AdminApiKeyStoreTlsAuthTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/AdminApiKeyStoreTlsAuthTest.java
index 7752f81..898c656 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/AdminApiKeyStoreTlsAuthTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/AdminApiKeyStoreTlsAuthTest.java
@@ -46,7 +46,7 @@ import org.apache.pulsar.client.admin.PulsarAdminException;
import org.apache.pulsar.client.admin.internal.JacksonConfigurator;
import org.apache.pulsar.client.api.ProducerConsumerBase;
import org.apache.pulsar.client.impl.auth.AuthenticationKeyStoreTls;
-import org.apache.pulsar.client.impl.tls.NoopHostnameVerifier;
+import org.apache.pulsar.common.tls.NoopHostnameVerifier;
import org.apache.pulsar.common.policies.data.ClusterData;
import org.apache.pulsar.common.policies.data.TenantInfo;
import org.apache.pulsar.common.util.keystoretls.KeyStoreSSLContext;
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ClientCnx.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ClientCnx.java
index 8c1f8cc..4fb10e4 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ClientCnx.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ClientCnx.java
@@ -60,7 +60,9 @@ import org.apache.pulsar.client.api.PulsarClientException;
import org.apache.pulsar.client.api.PulsarClientException.TimeoutException;
import org.apache.pulsar.client.impl.BinaryProtoLookupService.LookupDataResult;
import org.apache.pulsar.client.impl.conf.ClientConfigurationData;
-import org.apache.pulsar.client.impl.tls.TlsHostnameVerifier;
+import org.apache.pulsar.common.tls.TlsHostnameVerifier;
+import org.apache.pulsar.client.impl.transaction.TransactionBufferHandler;
+import org.apache.pulsar.client.util.TimedCompletableFuture;
import org.apache.pulsar.common.api.AuthData;
import org.apache.pulsar.common.api.proto.PulsarApi;
import org.apache.pulsar.common.protocol.Commands;
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/tls/DomainType.java b/pulsar-common/src/main/java/org/apache/pulsar/common/tls/DomainType.java
similarity index 95%
rename from pulsar-client/src/main/java/org/apache/pulsar/client/impl/tls/DomainType.java
rename to pulsar-common/src/main/java/org/apache/pulsar/common/tls/DomainType.java
index 5ed4f67..e4527b5 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/tls/DomainType.java
+++ b/pulsar-common/src/main/java/org/apache/pulsar/common/tls/DomainType.java
@@ -17,7 +17,7 @@
* under the License.
*/
-package org.apache.pulsar.client.impl.tls;
+package org.apache.pulsar.common.tls;
/**
* Domain types differentiated by Mozilla Public Suffix List.
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/tls/InetAddressUtils.java b/pulsar-common/src/main/java/org/apache/pulsar/common/tls/InetAddressUtils.java
similarity index 99%
rename from pulsar-client/src/main/java/org/apache/pulsar/client/impl/tls/InetAddressUtils.java
rename to pulsar-common/src/main/java/org/apache/pulsar/common/tls/InetAddressUtils.java
index 1a9bf61..65f9b37 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/tls/InetAddressUtils.java
+++ b/pulsar-common/src/main/java/org/apache/pulsar/common/tls/InetAddressUtils.java
@@ -21,7 +21,7 @@
* From Apache HTTP client
*/
-package org.apache.pulsar.client.impl.tls;
+package org.apache.pulsar.common.tls;
import java.util.regex.Pattern;
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/tls/NoopHostnameVerifier.java b/pulsar-common/src/main/java/org/apache/pulsar/common/tls/NoopHostnameVerifier.java
similarity index 96%
rename from pulsar-client/src/main/java/org/apache/pulsar/client/impl/tls/NoopHostnameVerifier.java
rename to pulsar-common/src/main/java/org/apache/pulsar/common/tls/NoopHostnameVerifier.java
index 7962e3e..edc734c 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/tls/NoopHostnameVerifier.java
+++ b/pulsar-common/src/main/java/org/apache/pulsar/common/tls/NoopHostnameVerifier.java
@@ -21,7 +21,7 @@
* From Apache HTTP client
*/
-package org.apache.pulsar.client.impl.tls;
+package org.apache.pulsar.common.tls;
import javax.net.ssl.HostnameVerifier;
import javax.net.ssl.SSLSession;
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/tls/PublicSuffixList.java b/pulsar-common/src/main/java/org/apache/pulsar/common/tls/PublicSuffixList.java
similarity index 97%
rename from pulsar-client/src/main/java/org/apache/pulsar/client/impl/tls/PublicSuffixList.java
rename to pulsar-common/src/main/java/org/apache/pulsar/common/tls/PublicSuffixList.java
index eb1671a..4cc3415 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/tls/PublicSuffixList.java
+++ b/pulsar-common/src/main/java/org/apache/pulsar/common/tls/PublicSuffixList.java
@@ -21,7 +21,7 @@
* From Apache HTTP client
*/
-package org.apache.pulsar.client.impl.tls;
+package org.apache.pulsar.common.tls;
import java.util.Collections;
import java.util.List;
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/tls/PublicSuffixMatcher.java b/pulsar-common/src/main/java/org/apache/pulsar/common/tls/PublicSuffixMatcher.java
similarity index 99%
rename from pulsar-client/src/main/java/org/apache/pulsar/client/impl/tls/PublicSuffixMatcher.java
rename to pulsar-common/src/main/java/org/apache/pulsar/common/tls/PublicSuffixMatcher.java
index 05d2eca..45c2bd3 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/tls/PublicSuffixMatcher.java
+++ b/pulsar-common/src/main/java/org/apache/pulsar/common/tls/PublicSuffixMatcher.java
@@ -21,7 +21,7 @@
* From Apache HTTP client
*/
-package org.apache.pulsar.client.impl.tls;
+package org.apache.pulsar.common.tls;
import java.net.IDN;
import java.util.Collection;
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/tls/SubjectName.java b/pulsar-common/src/main/java/org/apache/pulsar/common/tls/SubjectName.java
similarity index 96%
rename from pulsar-client/src/main/java/org/apache/pulsar/client/impl/tls/SubjectName.java
rename to pulsar-common/src/main/java/org/apache/pulsar/common/tls/SubjectName.java
index 2885b88..04034e3 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/tls/SubjectName.java
+++ b/pulsar-common/src/main/java/org/apache/pulsar/common/tls/SubjectName.java
@@ -21,7 +21,7 @@
* From Apache HTTP client
*/
-package org.apache.pulsar.client.impl.tls;
+package org.apache.pulsar.common.tls;
import lombok.Data;
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/tls/TlsHostnameVerifier.java b/pulsar-common/src/main/java/org/apache/pulsar/common/tls/TlsHostnameVerifier.java
similarity index 99%
rename from pulsar-client/src/main/java/org/apache/pulsar/client/impl/tls/TlsHostnameVerifier.java
rename to pulsar-common/src/main/java/org/apache/pulsar/common/tls/TlsHostnameVerifier.java
index 5357656..9a2964d 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/tls/TlsHostnameVerifier.java
+++ b/pulsar-common/src/main/java/org/apache/pulsar/common/tls/TlsHostnameVerifier.java
@@ -21,7 +21,7 @@
* From Apache HTTP client
*/
-package org.apache.pulsar.client.impl.tls;
+package org.apache.pulsar.common.tls;
import java.net.InetAddress;
import java.net.UnknownHostException;
diff --git a/pulsar-common/src/main/java/org/apache/pulsar/common/util/SecurityUtility.java b/pulsar-common/src/main/java/org/apache/pulsar/common/util/SecurityUtility.java
index aba2f92..fcf2b19 100644
--- a/pulsar-common/src/main/java/org/apache/pulsar/common/util/SecurityUtility.java
+++ b/pulsar-common/src/main/java/org/apache/pulsar/common/util/SecurityUtility.java
@@ -29,6 +29,8 @@ import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
+import java.lang.reflect.Method;
+import java.nio.charset.StandardCharsets;
import java.security.GeneralSecurityException;
import java.security.KeyFactory;
import java.security.KeyManagementException;
@@ -49,6 +51,8 @@ import java.security.spec.PKCS8EncodedKeySpec;
import java.util.Base64;
import java.util.Collection;
import java.util.Set;
+import java.util.concurrent.ScheduledExecutorService;
+import javax.net.ssl.HostnameVerifier;
import javax.net.ssl.KeyManager;
import javax.net.ssl.KeyManagerFactory;
import javax.net.ssl.SSLContext;
@@ -58,6 +62,9 @@ import javax.net.ssl.TrustManagerFactory;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
+import org.apache.pulsar.common.classification.InterfaceAudience;
+import org.apache.pulsar.common.classification.InterfaceStability;
+import org.apache.pulsar.common.tls.TlsHostnameVerifier;
import org.eclipse.jetty.util.ssl.SslContextFactory;
/**
@@ -69,6 +76,9 @@ public class SecurityUtility {
public static final Provider BC_PROVIDER = getProvider();
public static final String BC_FIPS_PROVIDER_CLASS = "org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider";
public static final String BC_NON_FIPS_PROVIDER_CLASS = "org.bouncycastle.jce.provider.BouncyCastleProvider";
+ public static final String CONSCRYPT_PROVIDER_CLASS = "org.conscrypt.OpenSSLProvider";
+ public static final Provider CONSCRYPT_PROVIDER = loadConscryptProvider();
+
// Security.getProvider("BC") / Security.getProvider("BCFIPS").
// also used to get Factories. e.g. CertificateFactory.getInstance("X.509", "BCFIPS")
public static final String BC_FIPS = "BCFIPS";
@@ -106,6 +116,54 @@ public class SecurityUtility {
}
}
+ private static Provider loadConscryptProvider() {
+ Provider provider;
+ try {
+ provider = (Provider) Class.forName(CONSCRYPT_PROVIDER_CLASS).newInstance();
+ } catch (ReflectiveOperationException e) {
+ log.warn("Unable to get security provider for class {}", CONSCRYPT_PROVIDER_CLASS, e);
+ return null;
+ }
+
+ // Configure Conscrypt's default hostname verifier to use Pulsar's TlsHostnameVerifier which
+ // is more relaxed than the Conscrypt HostnameVerifier checking for RFC 2818 conformity.
+ //
+ // Certificates used in Pulsar docs and examples aren't strictly RFC 2818 compliant since they use the
+ // deprecated way of specifying the hostname in the CN field of the subject DN of the certificate.
+ // RFC 2818 recommends the use of SAN (subjectAltName) extension for specifying the hostname in the dNSName
+ // field of the subjectAltName extension.
+ //
+ // Conscrypt's default HostnameVerifier has dropped support for the deprecated method of specifying the hostname
+ // in the CN field. Pulsar's TlsHostnameVerifier continues to support the CN field.
+ //
+ // more details of Conscrypt's hostname verification:
+ // https://github.com/google/conscrypt/blob/master/IMPLEMENTATION_NOTES.md#hostname-verification
+ // there's a bug in Conscrypt while setting a custom HostnameVerifier,
+ // https://github.com/google/conscrypt/issues/1015 and therefore this solution alone
+ // isn't sufficient to configure Conscrypt's hostname verifier. The method processConscryptTrustManager
+ // contains the workaround.
+ try {
+ HostnameVerifier hostnameVerifier = new TlsHostnameVerifier();
+ Class<?> conscryptClazz = Class.forName("org.conscrypt.Conscrypt");
+ Object wrappedHostnameVerifier = conscryptClazz
+ .getMethod("wrapHostnameVerifier",
+ new Class[]{HostnameVerifier.class}).invoke(null, hostnameVerifier);
+ Method setDefaultHostnameVerifierMethod =
+ conscryptClazz
+ .getMethod("setDefaultHostnameVerifier",
+ new Class[]{Class.forName("org.conscrypt.ConscryptHostnameVerifier")});
+ setDefaultHostnameVerifierMethod.invoke(null, wrappedHostnameVerifier);
+ } catch (Exception e) {
+ log.warn("Unable to set default hostname verifier for Conscrypt", e);
+ }
+
+ Security.addProvider(provider);
+ if (log.isDebugEnabled()) {
+ log.debug("Added security provider '{}' from class {}", provider.getName(), CONSCRYPT_PROVIDER_CLASS);
+ }
+ return provider;
+ }
+
/**
* Get Bouncy Castle provider from classpath, and call Security.addProvider.
* Throw Exception if failed.
@@ -209,10 +267,11 @@ public class SecurityUtility {
TrustManager[] trustManagers = null;
KeyManager[] keyManagers = null;
- trustManagers = setupTrustCerts(ksh, allowInsecureConnection, trustCertficates);
+ trustManagers = setupTrustCerts(ksh, allowInsecureConnection, trustCertficates, CONSCRYPT_PROVIDER);
keyManagers = setupKeyManager(ksh, privateKey, certificates);
- SSLContext sslCtx = SSLContext.getInstance("TLS");
+ SSLContext sslCtx = CONSCRYPT_PROVIDER != null ? SSLContext.getInstance("TLS", CONSCRYPT_PROVIDER)
+ : SSLContext.getInstance("TLS");
sslCtx.init(keyManagers, trustManagers, new SecureRandom());
sslCtx.getDefaultSSLParameters();
return sslCtx;
@@ -231,12 +290,15 @@ public class SecurityUtility {
}
private static TrustManager[] setupTrustCerts(KeyStoreHolder ksh, boolean allowInsecureConnection,
- Certificate[] trustCertficates) throws NoSuchAlgorithmException, KeyStoreException {
+ Certificate[] trustCertficates, Provider securityProvider)
+ throws NoSuchAlgorithmException, KeyStoreException {
TrustManager[] trustManagers;
if (allowInsecureConnection) {
trustManagers = InsecureTrustManagerFactory.INSTANCE.getTrustManagers();
} else {
- TrustManagerFactory tmf = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
+ TrustManagerFactory tmf = securityProvider != null
+ ? TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm(), securityProvider)
+ : TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
if (trustCertficates == null || trustCertficates.length == 0) {
tmf.init((KeyStore) null);
@@ -248,10 +310,54 @@ public class SecurityUtility {
}
trustManagers = tmf.getTrustManagers();
+
+ for (TrustManager trustManager : trustManagers) {
+ processConscryptTrustManager(trustManager);
+ }
+ }
+ return trustManagers;
+ }
+
+ /***
+ * Conscrypt TrustManager instances will be configured to use the Pulsar {@link TlsHostnameVerifier}
+ * class.
+ * This method is used as a workaround for https://github.com/google/conscrypt/issues/1015
+ * when Conscrypt / OpenSSL is used as the TLS security provider.
+ *
+ * @param trustManagers the array of TrustManager instances to process.
+ * @return same instance passed as parameter
+ */
+ @InterfaceAudience.Private
+ public static TrustManager[] processConscryptTrustManagers(TrustManager[] trustManagers) {
+ for (TrustManager trustManager : trustManagers) {
+ processConscryptTrustManager(trustManager);
}
return trustManagers;
}
+ // workaround https://github.com/google/conscrypt/issues/1015
+ private static void processConscryptTrustManager(TrustManager trustManager) {
+ if (trustManager.getClass().getName().equals("org.conscrypt.TrustManagerImpl")) {
+ try {
+ Class<?> conscryptClazz = Class.forName("org.conscrypt.Conscrypt");
+ Object hostnameVerifier = conscryptClazz.getMethod("getHostnameVerifier",
+ new Class[]{TrustManager.class}).invoke(null, trustManager);
+ if (hostnameVerifier == null) {
+ Object defaultHostnameVerifier = conscryptClazz.getMethod("getDefaultHostnameVerifier",
+ new Class[]{TrustManager.class}).invoke(null, trustManager);
+ if (defaultHostnameVerifier != null) {
+ conscryptClazz.getMethod("setHostnameVerifier", new Class[]{
+ TrustManager.class,
+ Class.forName("org.conscrypt.ConscryptHostnameVerifier")
+ }).invoke(null, trustManager, defaultHostnameVerifier);
+ }
+ }
+ } catch (ReflectiveOperationException e) {
+ log.warn("Unable to set hostname verifier for Conscrypt TrustManager implementation", e);
+ }
+ }
+ }
+
public static X509Certificate[] loadCertificatesFromPemFile(String certFilePath) throws KeyManagementException {
X509Certificate[] certificates = null;
@@ -411,6 +517,9 @@ public class SecurityUtility {
super();
sslCtxRefresher = new DefaultSslContextBuilder(tlsAllowInsecureConnection, tlsTrustCertsFilePath,
tlsCertificateFilePath, tlsKeyFilePath, tlsRequireTrustedClientCertOnConnect, certRefreshInSec);
+ if (CONSCRYPT_PROVIDER != null) {
+ setProvider(CONSCRYPT_PROVIDER.getName());
+ }
}
@Override
diff --git a/pulsar-common/src/main/java/org/apache/pulsar/common/util/keystoretls/KeyStoreSSLContext.java b/pulsar-common/src/main/java/org/apache/pulsar/common/util/keystoretls/KeyStoreSSLContext.java
index e6c1af8..c7b4cfe 100644
--- a/pulsar-common/src/main/java/org/apache/pulsar/common/util/keystoretls/KeyStoreSSLContext.java
+++ b/pulsar-common/src/main/java/org/apache/pulsar/common/util/keystoretls/KeyStoreSSLContext.java
@@ -40,6 +40,7 @@ import javax.net.ssl.SSLException;
import javax.net.ssl.TrustManagerFactory;
import lombok.Getter;
import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.common.util.SecurityUtility;
import org.eclipse.jetty.util.ssl.SslContextFactory;
/**
@@ -152,7 +153,9 @@ public class KeyStoreSSLContext {
if (this.allowInsecureConnection) {
trustManagerFactory = InsecureTrustManagerFactory.INSTANCE;
} else {
- trustManagerFactory = TrustManagerFactory.getInstance(tmfAlgorithm);
+ trustManagerFactory = sslProviderString != null
+ ? TrustManagerFactory.getInstance(tmfAlgorithm, sslProviderString)
+ : TrustManagerFactory.getInstance(tmfAlgorithm);
KeyStore trustStore = KeyStore.getInstance(trustStoreTypeString);
char[] passwordChars = trustStorePassword.toCharArray();
trustStore.load(new FileInputStream(trustStorePath), passwordChars);
@@ -160,7 +163,9 @@ public class KeyStoreSSLContext {
}
// init
- sslContext.init(keyManagers, trustManagerFactory.getTrustManagers(), new SecureRandom());
+ sslContext.init(keyManagers, SecurityUtility
+ .processConscryptTrustManagers(trustManagerFactory.getTrustManagers()),
+ new SecureRandom());
this.sslContext = sslContext;
return sslContext;
}
@@ -336,6 +341,13 @@ public class KeyStoreSSLContext {
throws GeneralSecurityException, SSLException, FileNotFoundException, IOException {
SslContextFactory sslCtxFactory;
+ if (sslProviderString == null) {
+ Provider provider = SecurityUtility.CONSCRYPT_PROVIDER;
+ if (provider != null) {
+ sslProviderString = provider.getName();
+ }
+ }
+
sslCtxFactory = new SslContextFactoryWithAutoRefresh(
sslProviderString,
keyStoreTypeString,
diff --git a/pulsar-common/src/main/java/org/apache/pulsar/common/util/keystoretls/SslContextFactoryWithAutoRefresh.java b/pulsar-common/src/main/java/org/apache/pulsar/common/util/keystoretls/SslContextFactoryWithAutoRefresh.java
index e18e8c6..0882a3a 100644
--- a/pulsar-common/src/main/java/org/apache/pulsar/common/util/keystoretls/SslContextFactoryWithAutoRefresh.java
+++ b/pulsar-common/src/main/java/org/apache/pulsar/common/util/keystoretls/SslContextFactoryWithAutoRefresh.java
@@ -54,6 +54,9 @@ public class SslContextFactoryWithAutoRefresh extends SslContextFactory {
trustStorePassword,
requireTrustedClientCertOnConnect,
certRefreshInSec);
+ if (sslProviderString != null) {
+ setProvider(sslProviderString);
+ }
}
@Override
diff --git a/pulsar-discovery-service/pom.xml b/pulsar-discovery-service/pom.xml
index 065a177..720bfc6 100644
--- a/pulsar-discovery-service/pom.xml
+++ b/pulsar-discovery-service/pom.xml
@@ -76,6 +76,11 @@
<dependency>
<groupId>org.eclipse.jetty</groupId>
+ <artifactId>jetty-alpn-conscrypt-server</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-servlet</artifactId>
</dependency>
diff --git a/pulsar-functions/worker/pom.xml b/pulsar-functions/worker/pom.xml
index fd3a43d..19ce6a4 100644
--- a/pulsar-functions/worker/pom.xml
+++ b/pulsar-functions/worker/pom.xml
@@ -90,6 +90,11 @@
<dependency>
<groupId>org.eclipse.jetty</groupId>
+ <artifactId>jetty-alpn-conscrypt-server</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-servlet</artifactId>
</dependency>
diff --git a/pulsar-proxy/pom.xml b/pulsar-proxy/pom.xml
index 3e8fab9..ab42b36 100644
--- a/pulsar-proxy/pom.xml
+++ b/pulsar-proxy/pom.xml
@@ -73,6 +73,11 @@
<dependency>
<groupId>org.eclipse.jetty</groupId>
+ <artifactId>jetty-alpn-conscrypt-server</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-servlet</artifactId>
</dependency>
diff --git a/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/DirectProxyHandler.java b/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/DirectProxyHandler.java
index 3e3d668..43d6879 100644
--- a/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/DirectProxyHandler.java
+++ b/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/DirectProxyHandler.java
@@ -59,7 +59,7 @@ import org.apache.pulsar.PulsarVersion;
import org.apache.pulsar.client.api.Authentication;
import org.apache.pulsar.client.api.AuthenticationDataProvider;
import org.apache.pulsar.client.api.PulsarClientException;
-import org.apache.pulsar.client.impl.tls.TlsHostnameVerifier;
+import org.apache.pulsar.common.tls.TlsHostnameVerifier;
import org.apache.pulsar.common.allocator.PulsarByteBufAllocator;
import org.apache.pulsar.common.api.AuthData;
import org.apache.pulsar.common.api.proto.PulsarApi.CommandAuthChallenge;
diff --git a/tiered-storage/file-system/pom.xml b/tiered-storage/file-system/pom.xml
index 2e201d9..d0bc0f7 100644
--- a/tiered-storage/file-system/pom.xml
+++ b/tiered-storage/file-system/pom.xml
@@ -80,19 +80,21 @@
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-server</artifactId>
- <version>${jetty.version}</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.eclipse.jetty</groupId>
+ <artifactId>jetty-alpn-conscrypt-server</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-servlet</artifactId>
- <version>${jetty.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-util</artifactId>
- <version>${jetty.version}</version>
<scope>test</scope>
</dependency>
</dependencies>
[pulsar] 02/46: auth token for debezium and kafka connect adaptor
Posted by eo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 65a1acd73340c2124a9d6ea06cff23f08dc45b82
Author: Ming Luo <it...@gmail.com>
AuthorDate: Wed May 27 23:38:34 2020 -0400
auth token for debezium and kafka connect adaptor
---
.../pulsar/io/debezium/PulsarDatabaseHistory.java | 22 +++++++++++++++++++---
.../io/kafka/connect/PulsarKafkaWorkerConfig.java | 10 ++++++++++
.../io/kafka/connect/PulsarOffsetBackingStore.java | 14 +++++++++++---
3 files changed, 40 insertions(+), 6 deletions(-)
diff --git a/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/PulsarDatabaseHistory.java b/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/PulsarDatabaseHistory.java
index ed7de1f..81459f3 100644
--- a/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/PulsarDatabaseHistory.java
+++ b/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/PulsarDatabaseHistory.java
@@ -37,6 +37,8 @@ import lombok.extern.slf4j.Slf4j;
import org.apache.kafka.common.config.ConfigDef.Importance;
import org.apache.kafka.common.config.ConfigDef.Type;
import org.apache.kafka.common.config.ConfigDef.Width;
+import org.apache.pulsar.client.api.AuthenticationFactory;
+import org.apache.pulsar.client.api.ClientBuilder;
import org.apache.pulsar.client.api.Message;
import org.apache.pulsar.client.api.MessageId;
import org.apache.pulsar.client.api.Producer;
@@ -69,14 +71,24 @@ public final class PulsarDatabaseHistory extends AbstractDatabaseHistory {
.withDescription("Pulsar service url")
.withValidation(Field::isRequired);
+ public static final Field PULSAR_TOKEN = Field.create(CONFIGURATION_FIELD_PREFIX_STRING + "pulsar.token")
+ .withDisplayName("Pulsar auth token")
+ .withType(Type.STRING)
+ .withWidth(Width.LONG)
+ .withImportance(Importance.HIGH)
+ .withDescription("Pulsar authentication token")
+ .withValidation(Field::isOptional);
+
public static Field.Set ALL_FIELDS = Field.setOf(
TOPIC,
SERVICE_URL,
+ PULSAR_TOKEN,
DatabaseHistory.NAME);
private final DocumentReader reader = DocumentReader.defaultReader();
private String topicName;
private String serviceUrl;
+ private String token;
private String dbHistoryName;
private volatile PulsarClient pulsarClient;
private volatile Producer<String> producer;
@@ -95,6 +107,7 @@ public final class PulsarDatabaseHistory extends AbstractDatabaseHistory {
}
this.topicName = config.getString(TOPIC);
this.serviceUrl = config.getString(SERVICE_URL);
+ this.token = config.getString(PULSAR_TOKEN);
// Copy the relevant portions of the configuration and add useful defaults ...
this.dbHistoryName = config.getString(DatabaseHistory.NAME, UUID.randomUUID().toString());
@@ -118,9 +131,12 @@ public final class PulsarDatabaseHistory extends AbstractDatabaseHistory {
void setupClientIfNeeded() {
if (null == this.pulsarClient) {
try {
- pulsarClient = PulsarClient.builder()
- .serviceUrl(serviceUrl)
- .build();
+ ClientBuilder builder = PulsarClient.builder().serviceUrl(serviceUrl);
+
+ if (token != null && token != "") {
+ builder = builder.authentication(AuthenticationFactory.token(token));
+ }
+ pulsarClient = builder.build();
} catch (PulsarClientException e) {
throw new RuntimeException("Failed to create pulsar client to pulsar cluster at "
+ serviceUrl, e);
diff --git a/pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarKafkaWorkerConfig.java b/pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarKafkaWorkerConfig.java
index 624c59a..92092741 100644
--- a/pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarKafkaWorkerConfig.java
+++ b/pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarKafkaWorkerConfig.java
@@ -45,6 +45,12 @@ public class PulsarKafkaWorkerConfig extends WorkerConfig {
private static final String PULSAR_SERVICE_URL_CONFIG_DOC = "pulsar service url";
/**
+ * <code>pulsar.auth.token</code>
+ */
+ public static final String PULSAR_AUTH_TOKEN_CONFIG = "pulsar.auth.token";
+ private static final String PULSAR_AUTH_TOKEN_CONFIG_DOC = "pulsar auth token";
+
+ /**
* <code>topic.namespace</code>
*/
public static final String TOPIC_NAMESPACE_CONFIG = "topic.namespace";
@@ -60,6 +66,10 @@ public class PulsarKafkaWorkerConfig extends WorkerConfig {
Type.STRING,
Importance.HIGH,
PULSAR_SERVICE_URL_CONFIG_DOC)
+ .define(PULSAR_AUTH_TOKEN_CONFIG,
+ Type.STRING,
+ Importance.HIGH,
+ PULSAR_AUTH_TOKEN_CONFIG_DOC)
.define(TOPIC_NAMESPACE_CONFIG,
Type.STRING,
"public/default",
diff --git a/pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarOffsetBackingStore.java b/pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarOffsetBackingStore.java
index d7daf82..99303b8 100644
--- a/pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarOffsetBackingStore.java
+++ b/pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarOffsetBackingStore.java
@@ -36,6 +36,8 @@ import lombok.extern.slf4j.Slf4j;
import org.apache.kafka.connect.runtime.WorkerConfig;
import org.apache.kafka.connect.storage.OffsetBackingStore;
import org.apache.kafka.connect.util.Callback;
+import org.apache.pulsar.client.api.AuthenticationFactory;
+import org.apache.pulsar.client.api.ClientBuilder;
import org.apache.pulsar.client.api.Message;
import org.apache.pulsar.client.api.MessageId;
import org.apache.pulsar.client.api.Producer;
@@ -54,6 +56,7 @@ public class PulsarOffsetBackingStore implements OffsetBackingStore {
private PulsarClient client;
private String serviceUrl;
private String topic;
+ private String token;
private Producer<byte[]> producer;
private Reader<byte[]> reader;
private volatile CompletableFuture<Void> outstandingReadToEnd = null;
@@ -63,6 +66,7 @@ public class PulsarOffsetBackingStore implements OffsetBackingStore {
this.topic = workerConfig.getString(PulsarKafkaWorkerConfig.OFFSET_STORAGE_TOPIC_CONFIG);
checkArgument(!isBlank(topic), "Offset storage topic must be specified");
this.serviceUrl = workerConfig.getString(PulsarKafkaWorkerConfig.PULSAR_SERVICE_URL_CONFIG);
+ this.token = workerConfig.getString(PulsarKafkaWorkerConfig.PULSAR_AUTH_TOKEN_CONFIG);
checkArgument(!isBlank(serviceUrl), "Pulsar service url must be specified at `"
+ WorkerConfig.BOOTSTRAP_SERVERS_CONFIG + "`");
this.data = new HashMap<>();
@@ -137,9 +141,13 @@ public class PulsarOffsetBackingStore implements OffsetBackingStore {
@Override
public void start() {
try {
- client = PulsarClient.builder()
- .serviceUrl(serviceUrl)
- .build();
+ ClientBuilder builder = PulsarClient.builder().serviceUrl(serviceUrl);
+
+ if (token != null && token != "") {
+ builder = builder.authentication(AuthenticationFactory.token(token));
+ }
+ client = builder.build();
+
log.info("Successfully created pulsar client to {}", serviceUrl);
producer = client.newProducer(Schema.BYTES)
.topic(topic)
[pulsar] 13/46: Changing default function subscription to be
function name, not FQFN
Posted by eo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 186e63b27bf2662f52a32a8769b95ce0a125acd7
Author: Chris Bartholomew <c_...@yahoo.com>
AuthorDate: Tue Nov 26 19:18:54 2019 -0500
Changing default function subscription to be function name, not FQFN
(cherry picked from commit 4bb90560ad7c22f136c50252041c3906ae2e28d4)
---
.../java/org/apache/pulsar/functions/instance/InstanceUtils.java | 4 +++-
pulsar-functions/instance/src/main/python/python_instance.py | 5 ++---
2 files changed, 5 insertions(+), 4 deletions(-)
diff --git a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/InstanceUtils.java b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/InstanceUtils.java
index 024599c..b0cb5e6 100644
--- a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/InstanceUtils.java
+++ b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/InstanceUtils.java
@@ -117,7 +117,9 @@ public class InstanceUtils {
}
public static String getDefaultSubscriptionName(String tenant, String namespace, String name) {
- return FunctionCommon.getFullyQualifiedName(tenant, namespace, name);
+ // Hacking this to just return the function name. Using a FQFN breaks the REST endpoints
+ // that deal with subscriptions
+ return name;
}
public static String getDefaultSubscriptionName(Function.FunctionDetails functionDetails) {
diff --git a/pulsar-functions/instance/src/main/python/python_instance.py b/pulsar-functions/instance/src/main/python/python_instance.py
index fecde7a..1fd67a7 100644
--- a/pulsar-functions/instance/src/main/python/python_instance.py
+++ b/pulsar-functions/instance/src/main/python/python_instance.py
@@ -136,9 +136,8 @@ class PythonInstance(object):
if self.instance_config.function_details.source.subscriptionType == Function_pb2.SubscriptionType.Value("FAILOVER"):
mode = pulsar._pulsar.ConsumerType.Failover
- subscription_name = str(self.instance_config.function_details.tenant) + "/" + \
- str(self.instance_config.function_details.namespace) + "/" + \
- str(self.instance_config.function_details.name)
+ # Default subscription to function name
+ subscription_name = str(self.instance_config.function_details.name)
properties = util.get_properties(util.getFullyQualifiedFunctionName(
self.instance_config.function_details.tenant,
[pulsar] 22/46: Fix checkstyle
Posted by eo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 4d1a8d79200bae83d0d26f530c46ef87d4fd0081
Author: Enrico Olivelli <eo...@apache.org>
AuthorDate: Wed May 12 11:50:43 2021 +0200
Fix checkstyle
---
.../src/main/resources/pulsar/checkstyle.xml | 28 ++++++++++++----------
.../org/apache/pulsar/client/impl/ClientCnx.java | 1 -
2 files changed, 16 insertions(+), 13 deletions(-)
diff --git a/buildtools/src/main/resources/pulsar/checkstyle.xml b/buildtools/src/main/resources/pulsar/checkstyle.xml
index 1d01c81..ecfcdb0 100644
--- a/buildtools/src/main/resources/pulsar/checkstyle.xml
+++ b/buildtools/src/main/resources/pulsar/checkstyle.xml
@@ -36,7 +36,7 @@ page at http://checkstyle.sourceforge.net/config.html -->
<module name="LineLength">
<!-- Checks if a line is too long. -->
- <property name="max" value="120"/>
+ <property name="max" value="200"/>
<property name="severity" value="error"/>
<!--
@@ -86,8 +86,9 @@ page at http://checkstyle.sourceforge.net/config.html -->
<property name="file" value="${checkstyle.suppressions.file}" default="suppressions.xml" />
</module>
- <!-- Check that every module has a package-info.java -->
+ <!-- Check that every module has a package-info.java
<module name="JavadocPackage"/>
+ -->
<!-- All Java AST specific tests live under TreeWalker module. -->
<module name="TreeWalker">
@@ -154,12 +155,14 @@ page at http://checkstyle.sourceforge.net/config.html -->
<property name="message" value="Static import functions from Guava Preconditions"/>
</module>
+ <!-- This rules makes harder cherry picks
<module name="UnusedImports">
<property name="severity" value="error"/>
<property name="processJavadoc" value="true"/>
<message key="import.unused"
value="Unused import: {0}."/>
</module>
+ -->
<!--
@@ -185,11 +188,12 @@ page at http://checkstyle.sourceforge.net/config.html -->
<property name="allowMissingParamTags" value="true"/>
</module>
+ <!-- Disabled rules, in order to ease cherry picks
<module name="JavadocStyle">
<property name="severity" value="error"/>
<property name="checkHtml" value="true"/>
</module>
-
+ -->
<!--
NAMING CHECKS
@@ -252,13 +256,15 @@ page at http://checkstyle.sourceforge.net/config.html -->
<property name="severity" value="error"/>
</module>
+
+ <!--
<module name="MethodNameCheck">
- <!-- Validates identifiers for method names. -->
<metadata name="altname" value="MethodName"/>
<property name="format" value="(^[a-z][a-zA-Z0-9]*(_[a-zA-Z0-9]+)*$|Void)"/>
<property name="severity" value="error"/>
</module>
-
+ -->
+
<module name="ParameterName">
<!-- Validates identifiers for method parameters against the
expression "^[a-z][a-zA-Z0-9]*$". -->
@@ -355,11 +361,8 @@ page at http://checkstyle.sourceforge.net/config.html -->
-->
+ <!-- disabled rule, in order to ease cherry picks
<module name="WhitespaceAround">
- <!-- Checks that various tokens are surrounded by whitespace.
- This includes most binary operators and keywords followed
- by regular or curly braces.
- -->
<property name="tokens" value="ASSIGN, BAND, BAND_ASSIGN, BOR,
BOR_ASSIGN, BSR, BSR_ASSIGN, BXOR, BXOR_ASSIGN, COLON, DIV, DIV_ASSIGN,
EQUAL, GE, GT, LAND, LE, LITERAL_CATCH, LITERAL_DO, LITERAL_ELSE,
@@ -369,6 +372,7 @@ page at http://checkstyle.sourceforge.net/config.html -->
SL, SL_ASSIGN, SR_ASSIGN, STAR, STAR_ASSIGN"/>
<property name="severity" value="error"/>
</module>
+ -->
<module name="WhitespaceAfter">
<!-- Checks that commas, semicolons and typecasts are followed by
@@ -396,15 +400,15 @@ page at http://checkstyle.sourceforge.net/config.html -->
<property name="severity" value="error"/>
</module>
+
+ <!-- disabled rule
<module name="OperatorWrap">
- <!-- Checks that operators like + and ? appear at newlines rather than
- at the end of the previous line.
- -->
<property name="option" value="NL"/>
<property name="tokens" value="BAND, BOR, BSR, BXOR, DIV, EQUAL,
GE, GT, LAND, LE, LITERAL_INSTANCEOF, LOR, LT, MINUS, MOD,
NOT_EQUAL, PLUS, QUESTION, SL, SR, STAR "/>
</module>
+ -->
<module name="OperatorWrap">
<!-- Checks that assignment operators are at the end of the line. -->
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ClientCnx.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ClientCnx.java
index 4fb10e4..91896ab 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ClientCnx.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ClientCnx.java
@@ -62,7 +62,6 @@ import org.apache.pulsar.client.impl.BinaryProtoLookupService.LookupDataResult;
import org.apache.pulsar.client.impl.conf.ClientConfigurationData;
import org.apache.pulsar.common.tls.TlsHostnameVerifier;
import org.apache.pulsar.client.impl.transaction.TransactionBufferHandler;
-import org.apache.pulsar.client.util.TimedCompletableFuture;
import org.apache.pulsar.common.api.AuthData;
import org.apache.pulsar.common.api.proto.PulsarApi;
import org.apache.pulsar.common.protocol.Commands;
[pulsar] 10/46: Always return from trigger even if read from output
topic times out
Posted by eo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit b56a6d86f9dd3f728f0e33e8b157060580dd1ef6
Author: Chris Bartholomew <c_...@yahoo.com>
AuthorDate: Sat Nov 23 18:49:49 2019 -0500
Always return from trigger even if read from output topic times out
(cherry picked from commit d5b70adb742b02f3f3ea3ef31aa956994cbea9be)
---
.../functions/worker/rest/api/ComponentImpl.java | 35 +++++++++++-----------
1 file changed, 18 insertions(+), 17 deletions(-)
diff --git a/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/rest/api/ComponentImpl.java b/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/rest/api/ComponentImpl.java
index 3bcea1b..41db4c9 100644
--- a/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/rest/api/ComponentImpl.java
+++ b/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/rest/api/ComponentImpl.java
@@ -958,6 +958,7 @@ public abstract class ComponentImpl {
log.error("Function in trigger function is not ready @ /{}/{}/{}", tenant, namespace, functionName);
throw new RestException(Status.BAD_REQUEST, "Function in trigger function is not ready");
}
+
String outputTopic = functionMetaData.getFunctionDetails().getSink().getTopic();
Reader<byte[]> reader = null;
Producer<byte[]> producer = null;
@@ -986,25 +987,22 @@ public abstract class ComponentImpl {
if (reader == null) {
return null;
}
- long curTime = System.currentTimeMillis();
- long maxTime = curTime + 1000;
- while (curTime < maxTime) {
- Message msg = reader.readNext(10000, TimeUnit.MILLISECONDS);
- if (msg == null)
- break;
- if (msg.getProperties().containsKey("__pfn_input_msg_id__")
- && msg.getProperties().containsKey("__pfn_input_topic__")) {
- MessageId newMsgId = MessageId.fromByteArray(
- Base64.getDecoder().decode((String) msg.getProperties().get("__pfn_input_msg_id__")));
-
- if (msgId.equals(newMsgId)
- && msg.getProperties().get("__pfn_input_topic__").equals(TopicName.get(inputTopicToWrite).toString())) {
- return new String(msg.getData());
- }
+
+ Message msg = reader.readNext(2500, TimeUnit.MILLISECONDS);
+
+ if (msg == null) {
+ return new String("No Message On Output Topic");
+ }
+
+ if (msg.getProperties().containsKey("__pfn_input_msg_id__")
+ && msg.getProperties().containsKey("__pfn_input_topic__")) {
+ MessageId newMsgId = MessageId.fromByteArray(
+ Base64.getDecoder().decode((String) msg.getProperties().get("__pfn_input_msg_id__")));
+ if (msgId.equals(newMsgId)
+ && msg.getProperties().get("__pfn_input_topic__").equals(TopicName.get(inputTopicToWrite).toString())) {
+ return new String(msg.getData());
}
- curTime = System.currentTimeMillis();
}
- throw new RestException(Status.REQUEST_TIMEOUT, "Request Timed Out");
} catch (SchemaSerializationException e) {
throw new RestException(Status.BAD_REQUEST, String.format("Failed to serialize input with error: %s. Please check if input data conforms with the schema of the input topic.", e.getMessage()));
} catch (IOException e) {
@@ -1017,6 +1015,9 @@ public abstract class ComponentImpl {
producer.closeAsync();
}
}
+
+ return null;
+
}
public FunctionState getFunctionState(final String tenant,
[pulsar] 17/46: Fix test case for Pulsar Auth Token in Kafka
Adapter Config
Posted by eo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 4070e67af2d06cb29302e6ffdb6721835bca7f6d
Author: Enrico Olivelli <eo...@apache.org>
AuthorDate: Mon May 10 12:56:51 2021 +0200
Fix test case for Pulsar Auth Token in Kafka Adapter Config
---
.../java/org/apache/pulsar/io/kafka/connect/PulsarKafkaWorkerConfig.java | 1 +
1 file changed, 1 insertion(+)
diff --git a/pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarKafkaWorkerConfig.java b/pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarKafkaWorkerConfig.java
index 92092741..7044b48 100644
--- a/pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarKafkaWorkerConfig.java
+++ b/pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarKafkaWorkerConfig.java
@@ -68,6 +68,7 @@ public class PulsarKafkaWorkerConfig extends WorkerConfig {
PULSAR_SERVICE_URL_CONFIG_DOC)
.define(PULSAR_AUTH_TOKEN_CONFIG,
Type.STRING,
+ "",
Importance.HIGH,
PULSAR_AUTH_TOKEN_CONFIG_DOC)
.define(TOPIC_NAMESPACE_CONFIG,
[pulsar] 37/46: Fix some conflicts
Posted by eo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 32e3bb87cc6863274e46a6897bbbb968d0dc1fde
Author: Enrico Olivelli <eo...@apache.org>
AuthorDate: Wed May 12 18:16:24 2021 +0200
Fix some conflicts
---
.../pulsar/client/api/SimpleProducerConsumerTest.java | 1 +
.../java/org/apache/pulsar/client/api/SimpleSchemaTest.java | 1 +
.../src/test/java/org/apache/pulsar/schema/SchemaTest.java | 5 +----
.../org/apache/pulsar/client/api/schema/GenericRecord.java | 1 +
.../java/org/apache/pulsar/client/impl/MessageImpl.java | 11 ++---------
.../apache/pulsar/client/impl/schema/AutoConsumeSchema.java | 13 -------------
.../client/impl/schema/generic/GenericAvroRecord.java | 1 +
.../client/impl/schema/generic/GenericJsonRecord.java | 2 ++
.../impl/schema/generic/GenericProtobufNativeRecord.java | 1 +
.../apache/pulsar/client/impl/schema/JSONSchemaTest.java | 7 ++++---
.../client/impl/schema/generic/GenericJsonRecordTest.java | 13 ++++++++++---
.../schema/generic/GenericProtobufNativeReaderTest.java | 3 ++-
.../apache/pulsar/functions/source/PulsarSourceTest.java | 2 +-
13 files changed, 27 insertions(+), 34 deletions(-)
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleProducerConsumerTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleProducerConsumerTest.java
index e522908..4613cb3 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleProducerConsumerTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleProducerConsumerTest.java
@@ -100,6 +100,7 @@ import org.apache.pulsar.common.compression.CompressionCodecProvider;
import org.apache.pulsar.common.naming.TopicName;
import org.apache.pulsar.common.util.FutureUtil;
import org.apache.pulsar.shaded.com.google.protobuf.v241.ByteString;
+import org.apache.pulsar.common.schema.SchemaType;
import org.awaitility.Awaitility;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleSchemaTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleSchemaTest.java
index 0e1d756..d594688 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleSchemaTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleSchemaTest.java
@@ -21,6 +21,7 @@ package org.apache.pulsar.client.api;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
+import lombok.Cleanup;
import lombok.NoArgsConstructor;
import static java.nio.charset.StandardCharsets.UTF_8;
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/schema/SchemaTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/schema/SchemaTest.java
index 9542a36..0897e67 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/schema/SchemaTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/schema/SchemaTest.java
@@ -23,6 +23,7 @@ import static org.apache.pulsar.schema.compatibility.SchemaCompatibilityCheckTes
import static org.testng.Assert.assertEquals;
import static org.testng.Assert.assertFalse;
import static org.testng.Assert.assertNotNull;
+import static org.testng.Assert.assertNull;
import static org.testng.Assert.assertTrue;
import static org.testng.Assert.fail;
import static org.testng.internal.junit.ArrayAsserts.assertArrayEquals;
@@ -155,10 +156,6 @@ public class SchemaTest extends MockedPulsarServiceBaseTest {
producer.send(personTwo);
- Schemas.PersonTwo personConsume = consumer.receive().getValue();
- assertEquals("Tom", personConsume.getName());
- assertEquals(1, personConsume.getId());
-
Message<Schemas.PersonTwo> message = consumer.receive();
Schemas.PersonTwo personConsume = message.getValue();
assertEquals(personConsume.getName(), "Tom");
diff --git a/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/schema/GenericRecord.java b/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/schema/GenericRecord.java
index 1dda99f..2f1edb1 100644
--- a/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/schema/GenericRecord.java
+++ b/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/schema/GenericRecord.java
@@ -21,6 +21,7 @@ package org.apache.pulsar.client.api.schema;
import java.util.List;
import org.apache.pulsar.common.classification.InterfaceAudience;
import org.apache.pulsar.common.classification.InterfaceStability;
+import org.apache.pulsar.common.schema.SchemaType;
/**
* An interface represents a message with schema.
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageImpl.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageImpl.java
index 1eb7d3f..d7e262b 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageImpl.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageImpl.java
@@ -268,14 +268,6 @@ public class MessageImpl<T> implements Message<T> {
}
}
- @Override
- public int size() {
- if (msgMetadata.isNullValue()) {
- return 0;
- }
- return payload.readableBytes();
- }
-
public Schema<T> getSchemaInternal() {
return this.schema;
}
@@ -470,7 +462,8 @@ public class MessageImpl<T> implements Message<T> {
@Override
public byte[] getKeyBytes() {
- if (!msgMetadata.hasPartitionKey() || msgMetadata.isNullPartitionKey()) {
+ checkNotNull(msgMetadataBuilder);
+ if (!msgMetadataBuilder.hasPartitionKey() || msgMetadataBuilder.getPartitionKey() == null) {
return null;
} else if (hasBase64EncodedKey()) {
return Base64.getDecoder().decode(getKey());
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java
index 160a3a7..9f66875 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java
@@ -217,19 +217,6 @@ public class AutoConsumeSchema implements Schema<GenericRecord> {
}
}
- public Schema<GenericRecord> clone() {
- Schema<GenericRecord> schema = new AutoConsumeSchema();
- if (this.schema != null) {
- schema.configureSchemaInfo(topicName, componentName, this.schema.getSchemaInfo());
- } else {
- schema.configureSchemaInfo(topicName, componentName, null);
- }
- if (schemaInfoProvider != null) {
- schema.setSchemaInfoProvider(schemaInfoProvider);
- }
- return schema;
- }
-
@Override
public boolean requireFetchingSchemaInfo() {
return true;
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericAvroRecord.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericAvroRecord.java
index 767dcdb..c48c585 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericAvroRecord.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericAvroRecord.java
@@ -23,6 +23,7 @@ import java.util.stream.Collectors;
import lombok.extern.slf4j.Slf4j;
import org.apache.avro.util.Utf8;
import org.apache.pulsar.client.api.schema.Field;
+import org.apache.pulsar.common.schema.SchemaType;
/**
* A generic avro record.
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericJsonRecord.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericJsonRecord.java
index eaefaa5..4557022 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericJsonRecord.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericJsonRecord.java
@@ -28,6 +28,8 @@ import java.util.stream.Collectors;
import lombok.extern.slf4j.Slf4j;
import org.apache.pulsar.client.api.schema.Field;
import org.apache.pulsar.common.schema.SchemaInfo;
+import org.apache.pulsar.common.schema.SchemaType;
+
/**
* Generic json record.
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericProtobufNativeRecord.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericProtobufNativeRecord.java
index 1f94bb1..b061bba 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericProtobufNativeRecord.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericProtobufNativeRecord.java
@@ -21,6 +21,7 @@ package org.apache.pulsar.client.impl.schema.generic;
import com.google.protobuf.Descriptors;
import com.google.protobuf.DynamicMessage;
import org.apache.pulsar.client.api.schema.Field;
+import org.apache.pulsar.common.schema.SchemaType;
import java.util.List;
diff --git a/pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/JSONSchemaTest.java b/pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/JSONSchemaTest.java
index d6bd455..ad3da62 100644
--- a/pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/JSONSchemaTest.java
+++ b/pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/JSONSchemaTest.java
@@ -26,10 +26,11 @@ import static org.apache.pulsar.client.impl.schema.SchemaTestUtils.SCHEMA_JSON_N
import static org.testng.Assert.assertEquals;
import static org.testng.Assert.assertTrue;
import static org.testng.AssertJUnit.assertSame;
-
-import com.fasterxml.jackson.core.JsonProcessingException;
import io.netty.buffer.ByteBuf;
import io.netty.buffer.ByteBufAllocator;
+import lombok.AllArgsConstructor;
+import lombok.Data;
+import lombok.NoArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.apache.avro.Schema;
import org.apache.avro.SchemaValidationException;
@@ -372,7 +373,7 @@ public class JSONSchemaTest {
}
@Test
- public void testEncodeAndDecodeObject() throws JsonProcessingException {
+ public void testEncodeAndDecodeObject() throws Exception {
JSONSchema<PC> jsonSchema = JSONSchema.of(SchemaDefinition.<PC>builder().withPojo(PC.class).build());
PC pc = new PC("dell", "alienware", 2021, GPU.AMD,
new Seller("WA", "street", 98004));
diff --git a/pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/generic/GenericJsonRecordTest.java b/pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/generic/GenericJsonRecordTest.java
index 4a9bdae..453ecaf 100644
--- a/pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/generic/GenericJsonRecordTest.java
+++ b/pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/generic/GenericJsonRecordTest.java
@@ -21,15 +21,22 @@ package org.apache.pulsar.client.impl.schema.generic;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.testng.annotations.Test;
-
+import lombok.AllArgsConstructor;
+import lombok.Data;
+import lombok.NoArgsConstructor;
import java.util.Collections;
import static java.nio.charset.StandardCharsets.UTF_8;
+import org.apache.pulsar.common.util.ObjectMapperFactory;
+import org.apache.pulsar.common.schema.SchemaType;
import org.apache.pulsar.client.api.schema.Field;
+import org.apache.pulsar.client.api.schema.GenericSchema;
+import org.apache.pulsar.client.api.schema.SchemaDefinition;
+import org.apache.pulsar.client.impl.schema.JSONSchema;
import static org.testng.Assert.assertEquals;
import static org.testng.Assert.assertNull;
import static org.testng.Assert.assertTrue;
-
+import static org.testng.AssertJUnit.assertSame;
public class GenericJsonRecordTest {
@@ -93,7 +100,7 @@ public class GenericJsonRecordTest {
}
@Test
- public void testEncodeAndDecodeObject() throws JsonProcessingException {
+ public void testEncodeAndDecodeObject() throws Exception {
// test case from issue https://github.com/apache/pulsar/issues/9605
JSONSchema<PC> jsonSchema = JSONSchema.of(SchemaDefinition.<PC>builder().withPojo(PC.class).build());
GenericSchema genericJsonSchema = GenericJsonSchema.of(jsonSchema.getSchemaInfo());
diff --git a/pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/generic/GenericProtobufNativeReaderTest.java b/pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/generic/GenericProtobufNativeReaderTest.java
index 3a57688..84dc7fd 100644
--- a/pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/generic/GenericProtobufNativeReaderTest.java
+++ b/pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/generic/GenericProtobufNativeReaderTest.java
@@ -18,6 +18,7 @@
*/
package org.apache.pulsar.client.impl.schema.generic;
+import com.google.protobuf.DynamicMessage;
import lombok.extern.slf4j.Slf4j;
import org.apache.pulsar.client.api.schema.GenericRecord;
import org.apache.pulsar.client.api.schema.SchemaDefinition;
@@ -25,7 +26,7 @@ import org.apache.pulsar.client.impl.schema.ProtobufNativeSchema;
import org.apache.pulsar.client.schema.proto.Test.TestMessage;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;
-
+import org.apache.pulsar.common.schema.SchemaType;
import static org.testng.Assert.assertEquals;
@Slf4j
diff --git a/pulsar-functions/instance/src/test/java/org/apache/pulsar/functions/source/PulsarSourceTest.java b/pulsar-functions/instance/src/test/java/org/apache/pulsar/functions/source/PulsarSourceTest.java
index 65474ff..8e4ad26 100644
--- a/pulsar-functions/instance/src/test/java/org/apache/pulsar/functions/source/PulsarSourceTest.java
+++ b/pulsar-functions/instance/src/test/java/org/apache/pulsar/functions/source/PulsarSourceTest.java
@@ -286,7 +286,7 @@ public class PulsarSourceTest {
MessageImpl messageImpl = mock(MessageImpl.class);
Schema schema = mock(Schema.class);
when(messageImpl.getSchemaInternal()).thenReturn(schema);
- pulsarSource.received(consumer, (Message) messageImpl);
+ pulsarSource.received(consumer, messageImpl);
verify(messageImpl.getSchemaInternal(), times(1));
Record<GenericRecord> pushed = pulsarSource.read();
assertSame(pushed.getSchema(), schema);
[pulsar] 20/46: Bump Debezium version to latest in series
Posted by eo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit e275813efc7bc1af4d5f16d1e6bca27d48c71a4a
Author: Chris Bartholomew <c_...@yahoo.com>
AuthorDate: Sat Feb 6 11:26:38 2021 -0500
Bump Debezium version to latest in series
(cherry picked from commit 61805912195e64bbcdc29a7dfbdb8f90cd7aadfe)
---
pom.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/pom.xml b/pom.xml
index 52bbd0d..d415f8b 100644
--- a/pom.xml
+++ b/pom.xml
@@ -145,7 +145,7 @@
<flink.version>1.6.0</flink.version>
<scala.binary.version>2.11</scala.binary.version>
<scala-library.version>2.11.12</scala-library.version>
- <debezium.version>1.0.0.Final</debezium.version>
+ <debezium.version>1.0.3.Final</debezium.version>
<jsonwebtoken.version>0.11.1</jsonwebtoken.version>
<opencensus.version>0.18.0</opencensus.version>
<hbase.version>2.3.0</hbase.version>
[pulsar] 27/46: GenericObject: handle KeyValue with SEPARATED
encoding and add more tests (#10186)
Posted by eo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 72295b67beee0ba31297518f342ba80dfa197a61
Author: Enrico Olivelli <eo...@gmail.com>
AuthorDate: Mon Apr 12 11:12:14 2021 +0200
GenericObject: handle KeyValue with SEPARATED encoding and add more tests (#10186)
---
.../java/org/apache/pulsar/schema/SchemaTest.java | 96 ++++++++++++++++++++--
.../org/apache/pulsar/client/impl/MessageImpl.java | 27 ++++--
.../client/impl/schema/AutoConsumeSchema.java | 10 ++-
3 files changed, 116 insertions(+), 17 deletions(-)
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/schema/SchemaTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/schema/SchemaTest.java
index 56ca04c..7672cce 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/schema/SchemaTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/schema/SchemaTest.java
@@ -25,6 +25,8 @@ import static org.testng.Assert.assertFalse;
import static org.testng.Assert.assertNotNull;
import static org.testng.Assert.assertTrue;
import static org.testng.Assert.fail;
+import static org.testng.internal.junit.ArrayAsserts.assertArrayEquals;
+
import com.google.common.collect.Sets;
import java.nio.charset.StandardCharsets;
import java.util.Collections;
@@ -307,10 +309,19 @@ public class SchemaTest extends MockedPulsarServiceBaseTest {
}
@Test
- public void testKeyValueSchema() throws Exception {
+ public void testKeyValueSchemaINLINE() throws Exception {
+ testKeyValueSchema(KeyValueEncodingType.INLINE);
+ }
+
+ @Test
+ public void testKeyValueSchemaSEPARATED() throws Exception {
+ testKeyValueSchema(KeyValueEncodingType.SEPARATED);
+ }
+
+ private void testKeyValueSchema(KeyValueEncodingType keyValueEncodingType) throws Exception {
final String tenant = PUBLIC_TENANT;
final String namespace = "test-namespace-" + randomName(16);
- final String topicName = "test-string-schema";
+ final String topicName = "test-kv-schema-" + randomName(16);
final String topic = TopicName.get(
TopicDomain.persistent.value(),
@@ -325,18 +336,16 @@ public class SchemaTest extends MockedPulsarServiceBaseTest {
admin.topics().createPartitionedTopic(topic, 2);
Producer<KeyValue<String, Integer>> producer = pulsarClient
- .newProducer(Schema.KeyValue(Schema.STRING, Schema.INT32, KeyValueEncodingType.INLINE))
+ .newProducer(Schema.KeyValue(Schema.STRING, Schema.INT32, keyValueEncodingType))
.topic(topic)
.create();
- producer.send(new KeyValue<>("foo", 123));
-
- Consumer<KeyValue<String, Integer>> consumer = pulsarClient.newConsumer(Schema.KeyValue(Schema.STRING, Schema.INT32, KeyValueEncodingType.INLINE))
+ Consumer<KeyValue<String, Integer>> consumer = pulsarClient.newConsumer(Schema.KeyValue(Schema.STRING, Schema.INT32, keyValueEncodingType))
.subscriptionName("test-sub")
.topic(topic)
.subscribe();
- Consumer<GenericRecord> consumer2 = pulsarClient.newConsumer(Schema.AUTO_CONSUME())
+ Consumer<GenericRecord> consumer2 = pulsarClient.newConsumer(Schema.AUTO_CONSUME()) // keyValueEncodingType autodetected
.subscriptionName("test-sub2")
.topic(topic)
.subscribe();
@@ -347,6 +356,79 @@ public class SchemaTest extends MockedPulsarServiceBaseTest {
Message<GenericRecord> message2 = consumer2.receive();
assertEquals(message.getValue(), message2.getValue().getNativeObject());
+ if (keyValueEncodingType == KeyValueEncodingType.SEPARATED) {
+ // with "SEPARATED encoding the routing key is the key of the KeyValue
+ assertArrayEquals("foo".getBytes(StandardCharsets.UTF_8), message.getKeyBytes());
+ assertArrayEquals("foo".getBytes(StandardCharsets.UTF_8), message2.getKeyBytes());
+ } else {
+ assertNull(message.getKey());
+ assertNull(message2.getKey());
+ }
+
+ producer.close();
+ consumer.close();
+ consumer2.close();
+ }
+
+ @Test
+ public void testKeyValueSchemaWithStructsINLINE() throws Exception {
+ testKeyValueSchema(KeyValueEncodingType.INLINE);
+ }
+
+ @Test
+ public void testKeyValueSchemaWithStructsSEPARATED() throws Exception {
+ testKeyValueSchema(KeyValueEncodingType.SEPARATED);
+ }
+
+ private void testKeyValueSchemaWithStructs(KeyValueEncodingType keyValueEncodingType) throws Exception {
+ final String tenant = PUBLIC_TENANT;
+ final String namespace = "test-namespace-" + randomName(16);
+ final String topicName = "test-kv-schema-" + randomName(16);
+
+ final String topic = TopicName.get(
+ TopicDomain.persistent.value(),
+ tenant,
+ namespace,
+ topicName).toString();
+
+ admin.namespaces().createNamespace(
+ tenant + "/" + namespace,
+ Sets.newHashSet(CLUSTER_NAME));
+
+ admin.topics().createPartitionedTopic(topic, 2);
+
+ Producer<KeyValue<Schemas.PersonOne, Schemas.PersonTwo>> producer = pulsarClient
+ .newProducer(Schema.KeyValue(Schema.AVRO(Schemas.PersonOne.class), Schema.AVRO(Schemas.PersonTwo.class), keyValueEncodingType))
+ .topic(topic)
+ .create();
+
+ Consumer<KeyValue<Schemas.PersonOne, Schemas.PersonTwo>> consumer = pulsarClient.newConsumer(Schema.KeyValue(Schema.AVRO(Schemas.PersonOne.class), Schema.AVRO(Schemas.PersonTwo.class), keyValueEncodingType))
+ .subscriptionName("test-sub")
+ .topic(topic)
+ .subscribe();
+
+ Consumer<GenericRecord> consumer2 = pulsarClient.newConsumer(Schema.AUTO_CONSUME()) // keyValueEncodingType autodetected
+ .subscriptionName("test-sub2")
+ .topic(topic)
+ .subscribe();
+
+ Schemas.PersonOne key = new Schemas.PersonOne(8787);
+ Schemas.PersonTwo value = new Schemas.PersonTwo(323, "foo");
+ producer.send(new KeyValue<>(key, value));
+
+ Message<KeyValue<Schemas.PersonOne, Schemas.PersonTwo>> message = consumer.receive();
+ Message<GenericRecord> message2 = consumer2.receive();
+ assertEquals(message.getValue(), message2.getValue().getNativeObject());
+
+ if (keyValueEncodingType == KeyValueEncodingType.SEPARATED) {
+ // with "SEPARATED encoding the routing key is the key of the KeyValue
+ assertNotNull(message.getKeyBytes());
+ assertNotNull(message2.getKeyBytes());
+ } else {
+ assertNull(message.getKey());
+ assertNull(message2.getKey());
+ }
+
producer.close();
consumer.close();
consumer2.close();
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageImpl.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageImpl.java
index f7d8cf9..3cb2235 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageImpl.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageImpl.java
@@ -319,9 +319,14 @@ public class MessageImpl<T> implements Message<T> {
KeyValueSchema kvSchema = getKeyValueSchema();
byte[] schemaVersion = getSchemaVersion();
if (kvSchema.getKeyValueEncodingType() == KeyValueEncodingType.SEPARATED) {
- return (T) kvSchema.decode(
- msgMetadataBuilder.hasNullPartitionKey() ? null : getKeyBytes(),
- msgMetadataBuilder.hasNullValue() ? null : getData(), schemaVersion);
+ org.apache.pulsar.common.schema.KeyValue keyValue =
+ (org.apache.pulsar.common.schema.KeyValue) kvSchema.decode(getKeyBytes(), getData(), schemaVersion);
+ if (schema instanceof AutoConsumeSchema) {
+ return (T) AutoConsumeSchema.wrapPrimitiveObject(keyValue,
+ schema.getSchemaInfo().getType(), schemaVersion);
+ } else {
+ return (T) keyValue;
+ }
} else {
return schema.decode(getData(), schemaVersion);
}
@@ -330,9 +335,14 @@ public class MessageImpl<T> implements Message<T> {
private T getKeyValue() {
KeyValueSchema kvSchema = getKeyValueSchema();
if (kvSchema.getKeyValueEncodingType() == KeyValueEncodingType.SEPARATED) {
- return (T) kvSchema.decode(
- msgMetadataBuilder.hasNullPartitionKey() ? null : getKeyBytes(),
- msgMetadataBuilder.hasNullValue() ? null : getData(), null);
+ org.apache.pulsar.common.schema.KeyValue keyValue =
+ (org.apache.pulsar.common.schema.KeyValue) kvSchema.decode(getKeyBytes(), getData(), null);
+ if (schema instanceof AutoConsumeSchema) {
+ return (T) AutoConsumeSchema.wrapPrimitiveObject(keyValue,
+ schema.getSchemaInfo().getType(), null);
+ } else {
+ return (T) keyValue;
+ }
} else {
return schema.decode(getData());
}
@@ -420,8 +430,9 @@ public class MessageImpl<T> implements Message<T> {
@Override
public byte[] getKeyBytes() {
- checkNotNull(msgMetadataBuilder);
- if (hasBase64EncodedKey()) {
+ if (!msgMetadata.hasPartitionKey() || msgMetadata.isNullPartitionKey()) {
+ return null;
+ } else if (hasBase64EncodedKey()) {
return Base64.getDecoder().decode(getKey());
} else {
return getKey().getBytes(UTF_8);
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java
index 647a87b..41b1260 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java
@@ -21,12 +21,14 @@ package org.apache.pulsar.client.impl.schema;
import lombok.extern.slf4j.Slf4j;
import org.apache.pulsar.client.api.Schema;
import org.apache.pulsar.client.api.SchemaSerializationException;
+import org.apache.pulsar.client.api.schema.GenericObject;
import org.apache.pulsar.client.api.schema.GenericRecord;
import org.apache.pulsar.client.api.schema.SchemaInfoProvider;
import org.apache.pulsar.client.impl.schema.generic.GenericProtobufNativeSchema;
import org.apache.pulsar.client.impl.schema.generic.GenericSchemaImpl;
import org.apache.pulsar.common.schema.KeyValue;
import org.apache.pulsar.common.schema.SchemaInfo;
+import org.apache.pulsar.common.schema.SchemaType;
import java.util.concurrent.ExecutionException;
@@ -247,10 +249,14 @@ public class AutoConsumeSchema implements Schema<GenericRecord> {
if (this.schema == null) {
throw new IllegalStateException("Cannot decode a message without schema");
}
- return GenericObjectWrapper.of(value,
- this.schema.getSchemaInfo().getType(), schemaVersion);
+ return wrapPrimitiveObject(value, schema.getSchemaInfo().getType(), schemaVersion);
}
+ public static GenericRecord wrapPrimitiveObject(Object value, SchemaType type, byte[] schemaVersion) {
+ return GenericObjectWrapper.of(value, type, schemaVersion);
+ }
+
+
public Schema<?> getInternalSchema() {
return schema;
}
[pulsar] 35/46: cleaned some code in GenericJsonRecord (#10527)
Posted by eo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 56c2f6b3f95ff603de705fd4ebc65c5ac55b983c
Author: Abhilash Mandaliya <ab...@gmail.com>
AuthorDate: Wed May 12 16:50:18 2021 +0530
cleaned some code in GenericJsonRecord (#10527)
(cherry picked from commit 24b0d064e377dedda2049f6177028f4c26a870bf)
---
.../impl/schema/generic/GenericJsonRecord.java | 38 +++++++++++-----------
1 file changed, 19 insertions(+), 19 deletions(-)
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericJsonRecord.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericJsonRecord.java
index 7b4a530..eaefaa5 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericJsonRecord.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericJsonRecord.java
@@ -98,29 +98,29 @@ public class GenericJsonRecord extends VersionedGenericRecord {
}
private boolean isBinaryValue(String fieldName) {
- boolean isBinary = false;
+ if (schemaInfo == null) {
+ return false;
+ }
- do {
- if (schemaInfo == null) {
- break;
+ boolean isBinary = false;
+ try {
+ org.apache.avro.Schema schema = parseAvroSchema(schemaInfo.getSchemaDefinition());
+ org.apache.avro.Schema.Field field = schema.getField(fieldName);
+ if (field == null) {
+ return false;
}
-
- try {
- org.apache.avro.Schema schema = parseAvroSchema(schemaInfo.getSchemaDefinition());
- org.apache.avro.Schema.Field field = schema.getField(fieldName);
- ObjectMapper objectMapper = new ObjectMapper();
- JsonNode jsonNode = objectMapper.readTree(field.schema().toString());
- for (JsonNode node : jsonNode) {
- JsonNode jn = node.get("type");
- if (jn != null && ("bytes".equals(jn.asText()) || "byte".equals(jn.asText()))) {
- isBinary = true;
- }
+ ObjectMapper objectMapper = new ObjectMapper();
+ JsonNode jsonNode = objectMapper.readTree(field.schema().toString());
+ for (JsonNode node : jsonNode) {
+ JsonNode jn = node.get("type");
+ if (jn != null && ("bytes".equals(jn.asText()) || "byte".equals(jn.asText()))) {
+ isBinary = true;
+ break;
}
- } catch (Exception e) {
- log.error("parse schemaInfo failed. ", e);
}
- } while (false);
-
+ } catch (Exception e) {
+ log.error("parse schemaInfo failed. ", e);
+ }
return isBinary;
}
[pulsar] 12/46: Update "sample" tenant on standalone to stop using
old property/cluster/namespace naming convention.
Posted by eo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit e03a65fb59f441703a1f350f1ef8c80e8c39b52a
Author: Chris Bartholomew <c_...@yahoo.com>
AuthorDate: Sat Feb 15 15:55:49 2020 -0500
Update "sample" tenant on standalone to stop using old property/cluster/namespace naming convention.
(cherry picked from commit dbe1feeb3940691ece94f39f17463d4733f56995)
---
.../src/main/java/org/apache/pulsar/PulsarStandalone.java | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
index d5ec2eb..a8210bb 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
@@ -392,9 +392,9 @@ public class PulsarStandalone implements AutoCloseable {
private void createSampleNameSpace(ClusterData clusterData, String cluster) {
// Create a sample namespace
- final String property = "sample";
+ final String tenant = "sample";
final String globalCluster = "global";
- final String namespace = property + "/" + cluster + "/ns1";
+ final String namespace = tenant + "/ns1";
try {
if (!admin.clusters().getClusters().contains(cluster)) {
admin.clusters().createCluster(cluster, clusterData);
@@ -407,12 +407,12 @@ public class PulsarStandalone implements AutoCloseable {
admin.clusters().createCluster(globalCluster, new ClusterData(null, null));
}
- if (!admin.tenants().getTenants().contains(property)) {
- admin.tenants().createTenant(property,
+ if (!admin.tenants().getTenants().contains(tenant)) {
+ admin.tenants().createTenant(tenant,
new TenantInfo(Sets.newHashSet(config.getSuperUserRoles()), Sets.newHashSet(cluster)));
}
- if (!admin.namespaces().getNamespaces(property).contains(namespace)) {
+ if (!admin.namespaces().getNamespaces(tenant).contains(namespace)) {
admin.namespaces().createNamespace(namespace);
}
} catch (PulsarAdminException e) {
[pulsar] 11/46: Update command descriptions from old
'property/cluster/namespace' format to current 'tenant/namespace' format
Posted by eo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit b30ee3da90756eeae93a51c60d82fc59699d029b
Author: Chris Bartholomew <c_...@yahoo.com>
AuthorDate: Sat Feb 15 15:49:34 2020 -0500
Update command descriptions from old 'property/cluster/namespace' format to current 'tenant/namespace' format
(cherry picked from commit 81710076ff4f7171b410e1f344150458a9f518f1)
---
docker/publish.sh | 2 +
.../pulsar/admin/cli/CmdNonPersistentTopics.java | 14 ++---
.../pulsar/admin/cli/CmdPersistentTopics.java | 60 +++++++++++-----------
.../apache/pulsar/admin/cli/CmdResourceQuotas.java | 6 +--
4 files changed, 42 insertions(+), 40 deletions(-)
diff --git a/docker/publish.sh b/docker/publish.sh
index 21e9a1b..f7810fe 100755
--- a/docker/publish.sh
+++ b/docker/publish.sh
@@ -50,6 +50,8 @@ fi
MVN_VERSION=`./get-version.sh`
echo "Pulsar version: ${MVN_VERSION}"
+exit
+
if [[ -z ${DOCKER_REGISTRY} ]]; then
docker_registry_org=${DOCKER_ORG}
else
diff --git a/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdNonPersistentTopics.java b/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdNonPersistentTopics.java
index 37f6024..7d2e8dc 100644
--- a/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdNonPersistentTopics.java
+++ b/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdNonPersistentTopics.java
@@ -45,7 +45,7 @@ public class CmdNonPersistentTopics extends CmdBase {
@Parameters(commandDescription = "Lookup a topic from the current serving broker")
private class Lookup extends CliCommand {
- @Parameter(description = "non-persistent://property/cluster/namespace/topic\n", required = true)
+ @Parameter(description = "non-persistent://tenant/namespaces/topic\n", required = true)
private java.util.List<String> params;
@Override
@@ -58,7 +58,7 @@ public class CmdNonPersistentTopics extends CmdBase {
@Parameters(commandDescription = "Get the stats for the topic and its connected producers and consumers. \n"
+ "\t All the rates are computed over a 1 minute window and are relative the last completed 1 minute period.")
private class GetStats extends CliCommand {
- @Parameter(description = "non-persistent://property/cluster/namespace/topic\n", required = true)
+ @Parameter(description = "non-persistent://tenant/namespaces/topic\n", required = true)
private java.util.List<String> params;
@Override
@@ -70,7 +70,7 @@ public class CmdNonPersistentTopics extends CmdBase {
@Parameters(commandDescription = "Get the internal stats for the topic")
private class GetInternalStats extends CliCommand {
- @Parameter(description = "non-persistent://property/cluster/namespace/topic\n", required = true)
+ @Parameter(description = "non-persistent://tenant/namespaces/topic\n", required = true)
private java.util.List<String> params;
@Override
@@ -84,7 +84,7 @@ public class CmdNonPersistentTopics extends CmdBase {
+ "\t\tThe partitioned topic has to be created before creating a producer on it.")
private class CreatePartitionedCmd extends CliCommand {
- @Parameter(description = "non-persistent://property/cluster/namespace/topic\n", required = true)
+ @Parameter(description = "non-persistent://tenant/namespaces/topic\n", required = true)
private java.util.List<String> params;
@Parameter(names = { "-p",
@@ -102,7 +102,7 @@ public class CmdNonPersistentTopics extends CmdBase {
+ "\t\tIf the topic is not created or is a non-partitioned topic, it returns empty topic with 0 partitions")
private class GetPartitionedTopicMetadataCmd extends CliCommand {
- @Parameter(description = "non-persistent://property/cluster/namespace/topic\n", required = true)
+ @Parameter(description = "non-persistent://tenant/namespaces/topic\n", required = true)
private java.util.List<String> params;
@Override
@@ -114,7 +114,7 @@ public class CmdNonPersistentTopics extends CmdBase {
@Parameters(commandDescription = "Get list of non-persistent topics present under a namespace")
private class GetList extends CliCommand {
- @Parameter(description = "property/cluster/namespace\n", required = true)
+ @Parameter(description = "tenant/namespaces\n", required = true)
private java.util.List<String> params;
@Override
@@ -126,7 +126,7 @@ public class CmdNonPersistentTopics extends CmdBase {
@Parameters(commandDescription = "Get list of non-persistent topics present under a namespace bundle")
private class GetListInBundle extends CliCommand {
- @Parameter(description = "property/cluster/namespace\n", required = true)
+ @Parameter(description = "tenant/namespaces\n", required = true)
private java.util.List<String> params;
@Parameter(names = { "-b",
diff --git a/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdPersistentTopics.java b/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdPersistentTopics.java
index 4ab275d..3dfb763 100644
--- a/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdPersistentTopics.java
+++ b/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdPersistentTopics.java
@@ -89,7 +89,7 @@ public class CmdPersistentTopics extends CmdBase {
@Parameters(commandDescription = "Get the list of topics under a namespace.")
private class ListCmd extends CliCommand {
- @Parameter(description = "property/cluster/namespace\n", required = true)
+ @Parameter(description = "tenant/namespaces\n", required = true)
private java.util.List<String> params;
@Override
@@ -101,7 +101,7 @@ public class CmdPersistentTopics extends CmdBase {
@Parameters(commandDescription = "Get the list of partitioned topics under a namespace.")
private class PartitionedTopicListCmd extends CliCommand {
- @Parameter(description = "property/cluster/namespace\n", required = true)
+ @Parameter(description = "tenant/namespaces\n", required = true)
private java.util.List<String> params;
@Override
@@ -113,7 +113,7 @@ public class CmdPersistentTopics extends CmdBase {
@Parameters(commandDescription = "Grant a new permission to a client role on a single topic.")
private class GrantPermissions extends CliCommand {
- @Parameter(description = "persistent://property/cluster/namespace/topic", required = true)
+ @Parameter(description = "persistent://tenant/namespaces/topic", required = true)
private java.util.List<String> params;
@Parameter(names = "--role", description = "Client role to which grant permissions", required = true)
@@ -134,7 +134,7 @@ public class CmdPersistentTopics extends CmdBase {
+ "\t\t\t was not set at the topic level, but rather at the namespace level, this \n"
+ "\t\t\t operation will return an error (HTTP status code 412).")
private class RevokePermissions extends CliCommand {
- @Parameter(description = "persistent://property/cluster/namespace/topic", required = true)
+ @Parameter(description = "persistent://tenant/namespaces/topic", required = true)
private java.util.List<String> params;
@Parameter(names = "--role", description = "Client role to which revoke permissions", required = true)
@@ -152,7 +152,7 @@ public class CmdPersistentTopics extends CmdBase {
+ "\t\t by the permissions set at the namespace level combined (union) with any eventual \n"
+ "\t\t specific permission set on the topic.")
private class Permissions extends CliCommand {
- @Parameter(description = "persistent://property/cluster/namespace/topic\n", required = true)
+ @Parameter(description = "persistent://tenant/namespaces/topic\n", required = true)
private java.util.List<String> params;
@Override
@@ -164,7 +164,7 @@ public class CmdPersistentTopics extends CmdBase {
@Parameters(commandDescription = "Lookup a topic from the current serving broker")
private class Lookup extends CliCommand {
- @Parameter(description = "persistent://property/cluster/namespace/topic\n", required = true)
+ @Parameter(description = "persistent://tenant/namespaces/topic\n", required = true)
private java.util.List<String> params;
@Override
@@ -176,7 +176,7 @@ public class CmdPersistentTopics extends CmdBase {
@Parameters(commandDescription = "Get Namespace bundle range of a topic")
private class GetBundleRange extends CliCommand {
- @Parameter(description = "persistent://property/cluster/namespace/topic\n", required = true)
+ @Parameter(description = "persistent://tenant/namespaces/topic\n", required = true)
private java.util.List<String> params;
@Override
@@ -190,7 +190,7 @@ public class CmdPersistentTopics extends CmdBase {
+ "\t\tThe partitioned topic has to be created before creating a producer on it.")
private class CreatePartitionedCmd extends CliCommand {
- @Parameter(description = "persistent://property/cluster/namespace/topic\n", required = true)
+ @Parameter(description = "persistent://tenant/namespaces/topic\n", required = true)
private java.util.List<String> params;
@Parameter(names = { "-p",
@@ -208,7 +208,7 @@ public class CmdPersistentTopics extends CmdBase {
+ "\t\tNew updating number of partitions must be greater than existing number of partitions.")
private class UpdatePartitionedCmd extends CliCommand {
- @Parameter(description = "persistent://property/cluster/namespace/topic\n", required = true)
+ @Parameter(description = "persistent://tenant/namespaces/topic\n", required = true)
private java.util.List<String> params;
@Parameter(names = { "-p",
@@ -226,7 +226,7 @@ public class CmdPersistentTopics extends CmdBase {
+ "\t\tIf the topic is not created or is a non-partitioned topic, it returns empty topic with 0 partitions")
private class GetPartitionedTopicMetadataCmd extends CliCommand {
- @Parameter(description = "persistent://property/cluster/namespace/topic\n", required = true)
+ @Parameter(description = "persistent://tenant/namespaces/topic\n", required = true)
private java.util.List<String> params;
@Override
@@ -240,7 +240,7 @@ public class CmdPersistentTopics extends CmdBase {
+ "\t\tIt will also delete all the partitions of the topic if it exists.")
private class DeletePartitionedCmd extends CliCommand {
- @Parameter(description = "persistent://property/cluster/namespace/topic\n", required = true)
+ @Parameter(description = "persistent://tenant/namespaces/topic\n", required = true)
private java.util.List<String> params;
@Parameter(names = "--force", description = "Close all producer/consumer/replicator and delete topic forcefully")
@@ -256,7 +256,7 @@ public class CmdPersistentTopics extends CmdBase {
@Parameters(commandDescription = "Delete a topic. \n"
+ "\t\tThe topic cannot be deleted if there's any active subscription or producers connected to it.")
private class DeleteCmd extends CliCommand {
- @Parameter(description = "persistent://property/cluster/namespace/topic\n", required = true)
+ @Parameter(description = "persistent://tenant/namespaces/topic\n", required = true)
private java.util.List<String> params;
@Parameter(names = "--force", description = "Close all producer/consumer/replicator and delete topic forcefully")
@@ -271,7 +271,7 @@ public class CmdPersistentTopics extends CmdBase {
@Parameters(commandDescription = "Unload a topic. \n")
private class UnloadCmd extends CliCommand {
- @Parameter(description = "persistent://property/cluster/namespace/topic\n", required = true)
+ @Parameter(description = "persistent://tenant/namespaces/topic\n", required = true)
private java.util.List<String> params;
@Override
@@ -283,7 +283,7 @@ public class CmdPersistentTopics extends CmdBase {
@Parameters(commandDescription = "Get the list of subscriptions on the topic")
private class ListSubscriptions extends CliCommand {
- @Parameter(description = "persistent://property/cluster/namespace/topic\n", required = true)
+ @Parameter(description = "persistent://tenant/namespaces/topic\n", required = true)
private java.util.List<String> params;
@Override
@@ -296,7 +296,7 @@ public class CmdPersistentTopics extends CmdBase {
@Parameters(commandDescription = "Delete a durable subscriber from a topic. \n"
+ "\t\tThe subscription cannot be deleted if there are any active consumers attached to it \n")
private class DeleteSubscription extends CliCommand {
- @Parameter(description = "persistent://property/cluster/namespace/topic", required = true)
+ @Parameter(description = "persistent://tenant/namespaces/topic", required = true)
private java.util.List<String> params;
@Parameter(names = { "-f",
@@ -316,7 +316,7 @@ public class CmdPersistentTopics extends CmdBase {
@Parameters(commandDescription = "Get the stats for the topic and its connected producers and consumers. \n"
+ "\t All the rates are computed over a 1 minute window and are relative the last completed 1 minute period.")
private class GetStats extends CliCommand {
- @Parameter(description = "persistent://property/cluster/namespace/topic\n", required = true)
+ @Parameter(description = "persistent://tenant/namespaces/topic\n", required = true)
private java.util.List<String> params;
@Override
@@ -328,7 +328,7 @@ public class CmdPersistentTopics extends CmdBase {
@Parameters(commandDescription = "Get the internal stats for the topic")
private class GetInternalStats extends CliCommand {
- @Parameter(description = "persistent://property/cluster/namespace/topic\n", required = true)
+ @Parameter(description = "persistent://tenant/namespaces/topic\n", required = true)
private java.util.List<String> params;
@Parameter(names = { "-m",
@@ -344,7 +344,7 @@ public class CmdPersistentTopics extends CmdBase {
@Parameters(commandDescription = "Get the internal metadata info for the topic")
private class GetInternalInfo extends CliCommand {
- @Parameter(description = "persistent://property/cluster/namespace/topic\n", required = true)
+ @Parameter(description = "persistent://tenant/namespaces/topic\n", required = true)
private java.util.List<String> params;
@Override
@@ -359,7 +359,7 @@ public class CmdPersistentTopics extends CmdBase {
@Parameters(commandDescription = "Get the stats for the partitioned topic and its connected producers and consumers. \n"
+ "\t All the rates are computed over a 1 minute window and are relative the last completed 1 minute period.")
private class GetPartitionedStats extends CliCommand {
- @Parameter(description = "persistent://property/cluster/namespace/topic\n", required = true)
+ @Parameter(description = "persistent://tenant/namespaces/topic\n", required = true)
private java.util.List<String> params;
@Parameter(names = "--per-partition", description = "Get per partition stats")
@@ -375,7 +375,7 @@ public class CmdPersistentTopics extends CmdBase {
@Parameters(commandDescription = "Get the stats-internal for the partitioned topic and its connected producers and consumers. \n"
+ "\t All the rates are computed over a 1 minute window and are relative the last completed 1 minute period.")
private class GetPartitionedStatsInternal extends CliCommand {
- @Parameter(description = "persistent://property/cluster/namespace/topic\n", required = true)
+ @Parameter(description = "persistent://tenant/namespaces/topic\n", required = true)
private java.util.List<String> params;
@Override
@@ -387,7 +387,7 @@ public class CmdPersistentTopics extends CmdBase {
@Parameters(commandDescription = "Skip all the messages for the subscription")
private class SkipAll extends CliCommand {
- @Parameter(description = "persistent://property/cluster/namespace/topic", required = true)
+ @Parameter(description = "persistent://tenant/namespaces/topic", required = true)
private java.util.List<String> params;
@Parameter(names = { "-s", "--subscription" }, description = "Subscription to be cleared", required = true)
@@ -402,7 +402,7 @@ public class CmdPersistentTopics extends CmdBase {
@Parameters(commandDescription = "Skip some messages for the subscription")
private class Skip extends CliCommand {
- @Parameter(description = "persistent://property/cluster/namespace/topic", required = true)
+ @Parameter(description = "persistent://tenant/namespaces/topic", required = true)
private java.util.List<String> params;
@Parameter(names = { "-s",
@@ -421,7 +421,7 @@ public class CmdPersistentTopics extends CmdBase {
@Parameters(commandDescription = "Expire messages that older than given expiry time (in seconds) for the subscription")
private class ExpireMessages extends CliCommand {
- @Parameter(description = "persistent://property/cluster/namespace/topic", required = true)
+ @Parameter(description = "persistent://tenant/namespaces/topic", required = true)
private java.util.List<String> params;
@Parameter(names = { "-s",
@@ -440,7 +440,7 @@ public class CmdPersistentTopics extends CmdBase {
@Parameters(commandDescription = "Expire messages that older than given expiry time (in seconds) for all subscriptions")
private class ExpireMessagesForAllSubscriptions extends CliCommand {
- @Parameter(description = "persistent://property/cluster/namespace/topic", required = true)
+ @Parameter(description = "persistent://tenant/namespaces/topic", required = true)
private java.util.List<String> params;
@Parameter(names = { "-t", "--expireTime" }, description = "Expire messages older than time in seconds", required = true)
@@ -455,7 +455,7 @@ public class CmdPersistentTopics extends CmdBase {
@Parameters(commandDescription = "Create a new subscription on a topic")
private class CreateSubscription extends CliCommand {
- @Parameter(description = "persistent://property/cluster/namespace/topic", required = true)
+ @Parameter(description = "persistent://tenant/namespaces/topic", required = true)
private java.util.List<String> params;
@Parameter(names = { "-s",
@@ -484,7 +484,7 @@ public class CmdPersistentTopics extends CmdBase {
@Parameters(commandDescription = "Reset position for subscription to position closest to timestamp or messageId")
private class ResetCursor extends CliCommand {
- @Parameter(description = "persistent://property/cluster/namespace/topic", required = true)
+ @Parameter(description = "persistent://tenant/namespaces/topic", required = true)
private java.util.List<String> params;
@Parameter(names = { "-s",
@@ -520,7 +520,7 @@ public class CmdPersistentTopics extends CmdBase {
@Parameters(commandDescription = "Terminate a topic and don't allow any more messages to be published")
private class Terminate extends CliCommand {
- @Parameter(description = "persistent://property/cluster/namespace/topic", required = true)
+ @Parameter(description = "persistent://tenant/namespaces/topic", required = true)
private java.util.List<String> params;
@Override
@@ -538,7 +538,7 @@ public class CmdPersistentTopics extends CmdBase {
@Parameters(commandDescription = "Peek some messages for the subscription")
private class PeekMessages extends CliCommand {
- @Parameter(description = "persistent://property/cluster/namespace/topic", required = true)
+ @Parameter(description = "persistent://tenant/namespaces/topic", required = true)
private java.util.List<String> params;
@Parameter(names = { "-s",
@@ -616,7 +616,7 @@ public class CmdPersistentTopics extends CmdBase {
@Parameters(commandDescription = "Compact a topic")
private class Compact extends CliCommand {
- @Parameter(description = "persistent://property/cluster/namespace/topic", required = true)
+ @Parameter(description = "persistent://tenant/namespaces/topic", required = true)
private java.util.List<String> params;
@Override
@@ -630,7 +630,7 @@ public class CmdPersistentTopics extends CmdBase {
@Parameters(commandDescription = "Status of compaction on a topic")
private class CompactionStatusCmd extends CliCommand {
- @Parameter(description = "persistent://property/cluster/namespace/topic", required = true)
+ @Parameter(description = "persistent://tenant/namespaces/topic", required = true)
private java.util.List<String> params;
@Parameter(names = { "-w", "--wait-complete" },
diff --git a/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdResourceQuotas.java b/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdResourceQuotas.java
index 2e0e45c..410f3f8 100644
--- a/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdResourceQuotas.java
+++ b/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdResourceQuotas.java
@@ -33,7 +33,7 @@ public class CmdResourceQuotas extends CmdBase {
private class GetResourceQuota extends CliCommand {
@Parameter(names = { "--namespace",
- "-n" }, description = "property/cluster/namespace, must be specified together with '--bundle'\n")
+ "-n" }, description = "tenant/namespaces, must be specified together with '--bundle'\n")
private java.util.List<String> names;
@Parameter(names = { "--bundle",
@@ -57,7 +57,7 @@ public class CmdResourceQuotas extends CmdBase {
private class SetResourceQuota extends CliCommand {
@Parameter(names = { "--namespace",
- "-n" }, description = "property/cluster/namespace, must be specified together with '--bundle'\n")
+ "-n" }, description = "tenant/namespaces, must be specified together with '--bundle'\n")
private java.util.List<String> names;
@Parameter(names = { "--bundle",
@@ -111,7 +111,7 @@ public class CmdResourceQuotas extends CmdBase {
@Parameters(commandDescription = "Reset the specified namespace bundle's resource quota to default value.")
private class ResetNamespaceBundleResourceQuota extends CliCommand {
- @Parameter(names = { "--namespace", "-n" }, description = "property/cluster/namespace\n", required = true)
+ @Parameter(names = { "--namespace", "-n" }, description = "tenant/namespaces\n", required = true)
private java.util.List<String> names;
@Parameter(names = { "--bundle", "-b" }, description = "{start-boundary}_{end-boundary}\n", required = true)
[pulsar] 30/46: [Security] Upgrade vertx to 3.9.7,
addresses CVE-2018-12541 (#10261)
Posted by eo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit ea5b676372f62ed2d2d456f6c29aee57823cbf88
Author: Lari Hotari <lh...@users.noreply.github.com>
AuthorDate: Mon Apr 19 16:25:07 2021 +0300
[Security] Upgrade vertx to 3.9.7, addresses CVE-2018-12541 (#10261)
---
distribution/server/src/assemble/LICENSE.bin.txt | 9 +++++----
pom.xml | 2 +-
2 files changed, 6 insertions(+), 5 deletions(-)
diff --git a/distribution/server/src/assemble/LICENSE.bin.txt b/distribution/server/src/assemble/LICENSE.bin.txt
index 6808141..6cb9273 100644
--- a/distribution/server/src/assemble/LICENSE.bin.txt
+++ b/distribution/server/src/assemble/LICENSE.bin.txt
@@ -509,10 +509,11 @@ The Apache Software License, Version 2.0
* JCTools - Java Concurrency Tools for the JVM
- org.jctools-jctools-core-2.1.2.jar
* Vertx
- - io.vertx-vertx-auth-common-3.5.3.jar
- - io.vertx-vertx-bridge-common-3.5.3.jar
- - io.vertx-vertx-core-3.5.3.jar
- - io.vertx-vertx-web-3.5.3.jar
+ - io.vertx-vertx-auth-common-3.9.7.jar
+ - io.vertx-vertx-bridge-common-3.9.7.jar
+ - io.vertx-vertx-core-3.9.7.jar
+ - io.vertx-vertx-web-3.9.7.jar
+ - io.vertx-vertx-web-common-3.9.7.jar
* Apache ZooKeeper
- org.apache.zookeeper-zookeeper-jute-3.5.7.jar
diff --git a/pom.xml b/pom.xml
index 348ef7a..d590f20 100644
--- a/pom.xml
+++ b/pom.xml
@@ -105,7 +105,7 @@ flexible messaging model and an intuitive client API.</description>
<athenz.version>1.10.9</athenz.version>
<prometheus.version>0.5.0</prometheus.version>
<aspectj.version>1.9.2</aspectj.version>
- <vertx.version>3.5.3</vertx.version>
+ <vertx.version>3.9.7</vertx.version>
<rocksdb.version>6.10.2</rocksdb.version>
<slf4j.version>1.7.25</slf4j.version>
<commons.collections.version>3.2.2</commons.collections.version>
[pulsar] 39/46: fix build
Posted by eo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 53e8c3e5eeebfea9d43dd81c44e14d9638fd63e7
Author: Enrico Olivelli <eo...@apache.org>
AuthorDate: Wed May 12 18:32:18 2021 +0200
fix build
---
.../pulsar/client/api/SimpleProducerConsumerTest.java | 2 ++
tests/docker-images/java-test-functions/pom.xml | 5 +++++
.../tests/integration/docker/ContainerExecResult.java | 16 ++++++++++++++++
3 files changed, 23 insertions(+)
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleProducerConsumerTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleProducerConsumerTest.java
index 4613cb3..f612c93 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleProducerConsumerTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleProducerConsumerTest.java
@@ -39,6 +39,8 @@ import com.google.common.collect.Sets;
import io.netty.buffer.ByteBuf;
import io.netty.buffer.Unpooled;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.node.JsonNodeType;
import java.io.IOException;
import java.lang.reflect.Field;
import java.lang.reflect.Modifier;
diff --git a/tests/docker-images/java-test-functions/pom.xml b/tests/docker-images/java-test-functions/pom.xml
index 8c38d3b..90e394c 100644
--- a/tests/docker-images/java-test-functions/pom.xml
+++ b/tests/docker-images/java-test-functions/pom.xml
@@ -34,6 +34,11 @@
<artifactId>pulsar-io-core</artifactId>
<version>${project.version}</version>
</dependency>
+ <dependency>
+ <groupId>org.apache.pulsar</groupId>
+ <artifactId>pulsar-client-original</artifactId>
+ <version>${project.version}</version>
+ </dependency>
</dependencies>
<packaging>jar</packaging>
diff --git a/tests/integration/src/test/java/org/apache/pulsar/tests/integration/docker/ContainerExecResult.java b/tests/integration/src/test/java/org/apache/pulsar/tests/integration/docker/ContainerExecResult.java
index fe040ad..b28bce5 100644
--- a/tests/integration/src/test/java/org/apache/pulsar/tests/integration/docker/ContainerExecResult.java
+++ b/tests/integration/src/test/java/org/apache/pulsar/tests/integration/docker/ContainerExecResult.java
@@ -18,6 +18,7 @@
*/
package org.apache.pulsar.tests.integration.docker;
+import static org.testng.Assert.assertTrue;
import lombok.Data;
/**
@@ -30,4 +31,19 @@ public class ContainerExecResult {
private final String stdout;
private final String stderr;
+
+ public void assertNoOutput() {
+ assertNoStdout();
+ assertNoStderr();
+ }
+
+ public void assertNoStdout() {
+ assertTrue(stdout.isEmpty(),
+ "stdout should be empty, but was '" + stdout + "'");
+ }
+
+ public void assertNoStderr() {
+ assertTrue(stderr.isEmpty(),
+ "stderr should be empty, but was '" + stderr + "'");
+ }
}
[pulsar] 09/46: Update path for TTL config in Java 11
Posted by eo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit c9e071cca7291ccfdcb72df6e644c6174c33c470
Author: Chris Bartholomew <c_...@yahoo.com>
AuthorDate: Sun Sep 13 11:20:59 2020 -0400
Update path for TTL config in Java 11
(cherry picked from commit 341f7a9897aa03dd446432c0b8a9d1a9fabb6fb2)
---
docker/pulsar/Dockerfile | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docker/pulsar/Dockerfile b/docker/pulsar/Dockerfile
index 54609e8..a224c33 100644
--- a/docker/pulsar/Dockerfile
+++ b/docker/pulsar/Dockerfile
@@ -56,7 +56,7 @@ RUN update-alternatives --install /usr/bin/python python /usr/bin/python3 10
ADD target/python-client/ /pulsar/pulsar-client
ADD target/cpp-client/ /pulsar/cpp-client
-RUN echo networkaddress.cache.ttl=1 >> $JAVA_HOME/jre/lib/security/java.security
+RUN echo networkaddress.cache.ttl=1 >> $JAVA_HOME/conf/security/java.security
RUN apt-get update \
&& apt install -y /pulsar/cpp-client/*.deb \
&& apt-get clean \
[pulsar] 18/46: More fixes about running tests on JDK11 (#9893)
Posted by eo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 0550df69efcb0524387f403a5869ca74a7d30a6d
Author: Enrico Olivelli <eo...@gmail.com>
AuthorDate: Sun Mar 14 03:13:57 2021 +0100
More fixes about running tests on JDK11 (#9893)
When you are running tests on JDK11 you encounter a lot of issues.
This patch includes a list of minor fixes that can be grouped.
Master issue #9578
- Upgrade Mockito to latest version
- Add "--add-opens java.base/jdk.internal.loader=ALL-UNNAMED" in order to allow PowerMock to work
- add JAXB into jcloud provider (we already have it on the classpath in production, it is only in order to let tests run)
- Upgrade HDFS minicluster to 3.3.0
- Pin netty-codec-http dependency to the same version of netty (inherited from HDFS client)
- Use the same version of Jetty for hdfs-offload (old version does not work with JDK11)
---
pom.xml | 13 ++++++++++-
pulsar-client/pom.xml | 6 +++++
.../pulsar/client/impl/PulsarClientImplTest.java | 7 +++---
tiered-storage/file-system/pom.xml | 12 +++++++---
tiered-storage/jcloud/pom.xml | 26 ++++++++++++++++++++++
5 files changed, 56 insertions(+), 8 deletions(-)
diff --git a/pom.xml b/pom.xml
index 44ec252..52bbd0d 100644
--- a/pom.xml
+++ b/pom.xml
@@ -184,7 +184,7 @@
<testcontainers.version>1.14.3</testcontainers.version>
<kerby.version>1.1.1</kerby.version>
<testng.version>7.3.0</testng.version>
- <mockito.version>3.0.0</mockito.version>
+ <mockito.version>3.8.0</mockito.version>
<powermock.version>2.0.9</powermock.version>
<javassist.version>3.25.0-GA</javassist.version>
<failsafe.version>2.3.1</failsafe.version>
@@ -1096,6 +1096,7 @@
-Dpulsar.allocator.leak_detection=Advanced
-Dpulsar.allocator.exit_on_oom=false
-Dlog4j.configurationFile=log4j2.xml
+ ${test.additional.args}
</argLine>
<reuseForks>${testReuseFork}</reuseForks>
<forkCount>${testForkCount}</forkCount>
@@ -1490,6 +1491,15 @@
<profiles>
<profile>
+ <id>jdk11-tests</id>
+ <activation>
+ <jdk>[11,)</jdk>
+ </activation>
+ <properties>
+ <test.additional.args> --add-opens java.base/jdk.internal.loader=ALL-UNNAMED </test.additional.args>
+ </properties>
+ </profile>
+ <profile>
<id>coverage</id>
<build>
<plugins>
@@ -1614,6 +1624,7 @@
<id>main</id>
<activation>
<activeByDefault>true</activeByDefault>
+ <jdk>[8,)</jdk>
</activation>
<modules>
<module>buildtools</module>
diff --git a/pulsar-client/pom.xml b/pulsar-client/pom.xml
index 24920fc..cefece2 100644
--- a/pulsar-client/pom.xml
+++ b/pulsar-client/pom.xml
@@ -162,6 +162,12 @@
<version>${skyscreamer.version}</version>
<scope>test</scope>
</dependency>
+
+ <dependency>
+ <groupId>org.mockito</groupId>
+ <artifactId>mockito-core</artifactId>
+ </dependency>
+
</dependencies>
<build>
diff --git a/pulsar-client/src/test/java/org/apache/pulsar/client/impl/PulsarClientImplTest.java b/pulsar-client/src/test/java/org/apache/pulsar/client/impl/PulsarClientImplTest.java
index d712c26..1dac09b 100644
--- a/pulsar-client/src/test/java/org/apache/pulsar/client/impl/PulsarClientImplTest.java
+++ b/pulsar-client/src/test/java/org/apache/pulsar/client/impl/PulsarClientImplTest.java
@@ -27,7 +27,6 @@ import static org.testng.Assert.assertFalse;
import static org.testng.Assert.assertNotSame;
import static org.testng.Assert.assertSame;
import static org.testng.Assert.assertTrue;
-
import io.netty.buffer.ByteBuf;
import io.netty.channel.Channel;
import io.netty.channel.ChannelFuture;
@@ -56,8 +55,8 @@ import org.apache.pulsar.common.naming.NamespaceName;
import org.apache.pulsar.common.naming.TopicName;
import org.apache.pulsar.common.partition.PartitionedTopicMetadata;
import org.apache.pulsar.common.util.netty.EventLoopUtil;
-import org.mockito.internal.util.reflection.FieldSetter;
import org.mockito.Mockito;
+import org.powermock.reflect.Whitebox;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;
@@ -118,8 +117,8 @@ public class PulsarClientImplTest {
.thenReturn(CompletableFuture.completedFuture(mock(ProducerResponse.class)));
when(pool.getConnection(any(InetSocketAddress.class), any(InetSocketAddress.class)))
.thenReturn(CompletableFuture.completedFuture(cnx));
- FieldSetter.setField(clientImpl, clientImpl.getClass().getDeclaredField("cnxPool"), pool);
- FieldSetter.setField(clientImpl, clientImpl.getClass().getDeclaredField("lookup"), lookup);
+ Whitebox.setInternalState(clientImpl, "cnxPool", pool);
+ Whitebox.setInternalState(clientImpl, "lookup", lookup);
List<ConsumerBase<byte[]>> consumers = new ArrayList<>();
/**
diff --git a/tiered-storage/file-system/pom.xml b/tiered-storage/file-system/pom.xml
index df50d1d..2e201d9 100644
--- a/tiered-storage/file-system/pom.xml
+++ b/tiered-storage/file-system/pom.xml
@@ -71,22 +71,28 @@
</exclusion>
</exclusions>
</dependency>
+
+ <dependency>
+ <groupId>io.netty</groupId>
+ <artifactId>netty-codec-http</artifactId>
+ </dependency>
+
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-server</artifactId>
- <version>${org.eclipse.jetty-hdfs-offload}</version>
+ <version>${jetty.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-servlet</artifactId>
- <version>${org.eclipse.jetty-hdfs-offload}</version>
+ <version>${jetty.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-util</artifactId>
- <version>${org.eclipse.jetty-hdfs-offload}</version>
+ <version>${jetty.version}</version>
<scope>test</scope>
</dependency>
</dependencies>
diff --git a/tiered-storage/jcloud/pom.xml b/tiered-storage/jcloud/pom.xml
index 35fcc54..852cb61 100644
--- a/tiered-storage/jcloud/pom.xml
+++ b/tiered-storage/jcloud/pom.xml
@@ -87,6 +87,32 @@
<version>${project.version}</version>
<scope>test</scope>
</dependency>
+
+ <dependency>
+ <groupId>org.apache.jclouds</groupId>
+ <artifactId>jclouds-blobstore</artifactId>
+ <version>${jclouds.version}</version>
+ <scope>provided</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>javax.xml.bind</groupId>
+ <artifactId>jaxb-api</artifactId>
+ <exclusions>
+ <exclusion>
+ <groupId>javax.activation</groupId>
+ <artifactId>javax.activation-api</artifactId>
+ </exclusion>
+ </exclusions>
+ <scope>runtime</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>com.sun.activation</groupId>
+ <artifactId>javax.activation</artifactId>
+ <scope>runtime</scope>
+ </dependency>
+
</dependencies>
<build>
<plugins>
[pulsar] 03/46: upgrade presto version to 334 and JDK11
Posted by eo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit d6f953893fada414a864102db77f5091629e8bbd
Author: ming luo <it...@gmail.com>
AuthorDate: Thu Jan 7 18:58:13 2021 +0000
upgrade presto version to 334 and JDK11
---
conf/presto/config.properties | 4 +-
pom.xml | 305 +++++++++++----------
pulsar-sql/presto-distribution/pom.xml | 9 +-
.../pulsar/sql/presto/PulsarConnectorFactory.java | 4 +-
4 files changed, 165 insertions(+), 157 deletions(-)
diff --git a/conf/presto/config.properties b/conf/presto/config.properties
index f98b29a..506c5ac 100644
--- a/conf/presto/config.properties
+++ b/conf/presto/config.properties
@@ -37,6 +37,6 @@ scheduler.http-client.idle-timeout=1m
query.client.timeout=5m
query.min-expire-age=30m
-presto.version=testversion
-
node-scheduler.include-coordinator=true
+
+catalog.config-dir=/pulsar/conf/presto/catalog
diff --git a/pom.xml b/pom.xml
index c0f3a24..d122c8a 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1,151 +1,152 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<!--
-
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
-
--->
-<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
- xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
- <modelVersion>4.0.0</modelVersion>
- <packaging>pom</packaging>
- <parent>
- <groupId>org.apache</groupId>
- <artifactId>apache</artifactId>
- <version>18</version>
- </parent>
-
- <groupId>org.apache.pulsar</groupId>
- <artifactId>pulsar</artifactId>
-
- <version>2.7.2_1.0.0</version>
-
- <name>Pulsar</name>
- <description>Pulsar is a distributed pub-sub messaging platform with a very
-flexible messaging model and an intuitive client API.</description>
- <url>https://github.com/apache/pulsar</url>
-
- <organization>
- <name>Apache Software Foundation</name>
- <url>http://www.apache.org/</url>
- </organization>
- <inceptionYear>2017</inceptionYear>
-
- <developers>
- <developer>
- <organization>Apache Pulsar developers</organization>
- <organizationUrl>http://pulsar.apache.org/</organizationUrl>
- </developer>
- </developers>
-
- <licenses>
- <license>
- <name>Apache License, Version 2.0</name>
- <url>https://www.apache.org/licenses/LICENSE-2.0.txt</url>
- <distribution>repo</distribution>
- </license>
- </licenses>
-
- <scm>
- <url>https://github.com/apache/pulsar</url>
- <connection>scm:git:https://github.com/apache/pulsar.git</connection>
- <developerConnection>scm:git:ssh://git@github.com:apache/pulsar.git</developerConnection>
- </scm>
-
- <ciManagement>
- <system>Travis</system>
- <url>https://travis-ci.org/apache/pulsar</url>
- </ciManagement>
-
- <issueManagement>
- <system>Github</system>
- <url>https://github.com/apache/pulsar/issues</url>
- </issueManagement>
-
- <properties>
- <!--config keys to congiure test selection -->
- <include>*</include>
- <exclude/>
- <groups/>
- <excludedGroups/>
-
- <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
- <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
- <redirectTestOutputToFile>true</redirectTestOutputToFile>
- <testReuseFork>true</testReuseFork>
- <testForkCount>4</testForkCount>
- <testRealAWS>false</testRealAWS>
- <testRetryCount>1</testRetryCount>
- <docker.organization>apachepulsar</docker.organization>
-
- <!-- pin the protobuf-shaded version to make the pulsar build friendly to intellij -->
- <pulsar.protobuf.shaded.version>2.1.0-incubating</pulsar.protobuf.shaded.version>
-
- <!-- apache commons -->
- <commons-compress.version>1.19</commons-compress.version>
-
- <bookkeeper.version>4.12.0</bookkeeper.version>
- <zookeeper.version>3.5.7</zookeeper.version>
- <netty.version>4.1.60.Final</netty.version>
- <netty-tc-native.version>2.0.36.Final</netty-tc-native.version>
- <jetty.version>9.4.39.v20210325</jetty.version>
- <jersey.version>2.31</jersey.version>
- <athenz.version>1.10.9</athenz.version>
- <prometheus.version>0.5.0</prometheus.version>
- <aspectj.version>1.9.2</aspectj.version>
- <vertx.version>3.5.3</vertx.version>
- <rocksdb.version>6.10.2</rocksdb.version>
- <slf4j.version>1.7.25</slf4j.version>
- <commons.collections.version>3.2.2</commons.collections.version>
- <log4j2.version>2.10.0</log4j2.version>
- <bouncycastle.version>1.68</bouncycastle.version>
- <bouncycastlefips.version>1.0.2</bouncycastlefips.version>
- <jackson.version>2.11.1</jackson.version>
- <jackson.databind.version>2.11.1</jackson.databind.version>
- <reflections.version>0.9.11</reflections.version>
- <swagger.version>1.5.21</swagger.version>
- <puppycrawl.checkstyle.version>8.37</puppycrawl.checkstyle.version>
- <dockerfile-maven.version>1.4.13</dockerfile-maven.version>
- <typetools.version>0.5.0</typetools.version>
- <protobuf2.version>2.4.1</protobuf2.version>
- <protobuf3.version>3.11.4</protobuf3.version>
- <protoc3.version>${protobuf3.version}</protoc3.version>
- <grpc.version>1.18.0</grpc.version>
- <protoc-gen-grpc-java.version>${grpc.version}</protoc-gen-grpc-java.version>
- <gson.version>2.8.6</gson.version>
- <sketches.version>0.8.3</sketches.version>
- <hbc-core.version>2.2.0</hbc-core.version>
- <cassandra-driver-core.version>3.6.0</cassandra-driver-core.version>
- <aerospike-client.version>4.4.8</aerospike-client.version>
- <kafka-client.version>2.3.0</kafka-client.version>
- <rabbitmq-client.version>5.1.1</rabbitmq-client.version>
- <aws-sdk.version>1.11.774</aws-sdk.version>
- <avro.version>1.9.1</avro.version>
- <joda.version>2.10.1</joda.version>
- <jclouds.version>2.2.1</jclouds.version>
- <sqlite-jdbc.version>3.8.11.2</sqlite-jdbc.version>
- <mysql-jdbc.version>8.0.11</mysql-jdbc.version>
- <postgresql-jdbc.version>42.2.12</postgresql-jdbc.version>
- <clickhouse-jdbc.version>0.2.4</clickhouse-jdbc.version>
- <mariadb-jdbc.version>2.6.0</mariadb-jdbc.version>
- <hdfs-offload-version3>3.2.0</hdfs-offload-version3>
- <org.eclipse.jetty-hdfs-offload>9.3.24.v20180605</org.eclipse.jetty-hdfs-offload>
- <elasticsearch.version>7.9.1</elasticsearch.version>
- <presto.version>332</presto.version>
+ <?xml version="1.0" encoding="UTF-8"?>
+ <!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+ -->
+ <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <packaging>pom</packaging>
+ <parent>
+ <groupId>org.apache</groupId>
+ <artifactId>apache</artifactId>
+ <version>18</version>
+ </parent>
+
+ <groupId>org.apache.pulsar</groupId>
+ <artifactId>pulsar</artifactId>
+
+ <version>2.7.2_1.0.0</version>
+
+ <name>Pulsar</name>
+ <description>Pulsar is a distributed pub-sub messaging platform with a very
+ flexible messaging model and an intuitive client API.</description>
+ <url>https://github.com/apache/pulsar</url>
+
+ <organization>
+ <name>Apache Software Foundation</name>
+ <url>http://www.apache.org/</url>
+ </organization>
+ <inceptionYear>2017</inceptionYear>
+
+ <developers>
+ <developer>
+ <organization>Apache Pulsar developers</organization>
+ <organizationUrl>http://pulsar.apache.org/</organizationUrl>
+ </developer>
+ </developers>
+
+ <licenses>
+ <license>
+ <name>Apache License, Version 2.0</name>
+ <url>https://www.apache.org/licenses/LICENSE-2.0.txt</url>
+ <distribution>repo</distribution>
+ </license>
+ </licenses>
+
+ <scm>
+ <url>https://github.com/apache/pulsar</url>
+ <connection>scm:git:https://github.com/apache/pulsar.git</connection>
+ <developerConnection>scm:git:ssh://git@github.com:apache/pulsar.git</developerConnection>
+ </scm>
+
+ <ciManagement>
+ <system>Travis</system>
+ <url>https://travis-ci.org/apache/pulsar</url>
+ </ciManagement>
+
+ <issueManagement>
+ <system>Github</system>
+ <url>https://github.com/apache/pulsar/issues</url>
+ </issueManagement>
+
+ <properties>
+ <!--config keys to congiure test selection -->
+ <include>*</include>
+ <exclude/>
+ <groups/>
+ <excludedGroups/>
+
+ <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
+ <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
+ <redirectTestOutputToFile>true</redirectTestOutputToFile>
+ <testReuseFork>true</testReuseFork>
+ <testForkCount>4</testForkCount>
+ <testRealAWS>false</testRealAWS>
+ <testRetryCount>1</testRetryCount>
+ <docker.organization>apachepulsar</docker.organization>
+
+ <!-- pin the protobuf-shaded version to make the pulsar build friendly to intellij -->
+ <pulsar.protobuf.shaded.version>2.1.0-incubating</pulsar.protobuf.shaded.version>
+
+ <!-- apache commons -->
+ <commons-compress.version>1.19</commons-compress.version>
+
+ <bookkeeper.version>4.12.0</bookkeeper.version>
+ <zookeeper.version>3.5.7</zookeeper.version>
+ <netty.version>4.1.60.Final</netty.version>
+ <netty-tc-native.version>2.0.36.Final</netty-tc-native.version>
+ <jetty.version>9.4.39.v20210325</jetty.version>
+ <jersey.version>2.31</jersey.version>
+ <athenz.version>1.10.9</athenz.version>
+ <prometheus.version>0.5.0</prometheus.version>
+ <aspectj.version>1.9.2</aspectj.version>
+ <vertx.version>3.5.3</vertx.version>
+ <rocksdb.version>6.10.2</rocksdb.version>
+ <slf4j.version>1.7.25</slf4j.version>
+ <commons.collections.version>3.2.2</commons.collections.version>
+ <log4j2.version>2.10.0</log4j2.version>
+ <bouncycastle.version>1.68</bouncycastle.version>
+ <bouncycastlefips.version>1.0.2</bouncycastlefips.version>
+ <jackson.version>2.11.1</jackson.version>
+ <jackson.databind.version>2.11.1</jackson.databind.version>
+ <reflections.version>0.9.11</reflections.version>
+ <swagger.version>1.5.21</swagger.version>
+ <puppycrawl.checkstyle.version>8.37</puppycrawl.checkstyle.version>
+ <dockerfile-maven.version>1.4.13</dockerfile-maven.version>
+ <typetools.version>0.5.0</typetools.version>
+ <protobuf2.version>2.4.1</protobuf2.version>
+ <protobuf3.version>3.11.4</protobuf3.version>
+ <protoc3.version>${protobuf3.version}</protoc3.version>
+ <grpc.version>1.18.0</grpc.version>
+ <protoc-gen-grpc-java.version>${grpc.version}</protoc-gen-grpc-java.version>
+ <gson.version>2.8.6</gson.version>
+ <sketches.version>0.8.3</sketches.version>
+ <hbc-core.version>2.2.0</hbc-core.version>
+ <cassandra-driver-core.version>3.6.0</cassandra-driver-core.version>
+ <aerospike-client.version>4.4.8</aerospike-client.version>
+ <kafka-client.version>2.3.0</kafka-client.version>
+ <rabbitmq-client.version>5.1.1</rabbitmq-client.version>
+ <aws-sdk.version>1.11.774</aws-sdk.version>
+ <avro.version>1.9.1</avro.version>
+ <joda.version>2.10.1</joda.version>
+ <jclouds.version>2.2.1</jclouds.version>
+ <sqlite-jdbc.version>3.8.11.2</sqlite-jdbc.version>
+ <mysql-jdbc.version>8.0.11</mysql-jdbc.version>
+ <postgresql-jdbc.version>42.2.12</postgresql-jdbc.version>
+ <clickhouse-jdbc.version>0.2.4</clickhouse-jdbc.version>
+ <mariadb-jdbc.version>2.6.0</mariadb-jdbc.version>
+ <hdfs-offload-version3>3.2.0</hdfs-offload-version3>
+ <org.eclipse.jetty-hdfs-offload>9.3.24.v20180605</org.eclipse.jetty-hdfs-offload>
+ <elasticsearch.version>7.9.1</elasticsearch.version>
+ <presto.version>334</presto.version>
+ <flink.version>1.6.0</flink.version>
<scala.binary.version>2.11</scala.binary.version>
<scala-library.version>2.11.12</scala-library.version>
<debezium.version>1.0.0.Final</debezium.version>
@@ -482,7 +483,11 @@ flexible messaging model and an intuitive client API.</description>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
+<<<<<<< HEAD
<version>${commons-lang3.version}</version>
+=======
+ <version>3.9</version>
+>>>>>>> da22685e074... upgrade presto version to 334 and JDK11
</dependency>
<dependency>
@@ -1082,8 +1087,8 @@ flexible messaging model and an intuitive client API.</description>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
- <source>1.8</source>
- <target>1.8</target>
+ <source>8</source>
+ <target>11</target>
<encoding>UTF-8</encoding>
<showDeprecation>true</showDeprecation>
<showWarnings>true</showWarnings>
diff --git a/pulsar-sql/presto-distribution/pom.xml b/pulsar-sql/presto-distribution/pom.xml
index f02eba1..502f3c8 100644
--- a/pulsar-sql/presto-distribution/pom.xml
+++ b/pulsar-sql/presto-distribution/pom.xml
@@ -35,8 +35,7 @@
<properties>
<jersey.version>2.31</jersey.version>
- <presto.version>332</presto.version>
- <jersey.version>2.31</jersey.version>
+ <presto.version>334</presto.version>
<airlift.version>0.170</airlift.version>
<objenesis.version>2.6</objenesis.version>
<objectsize.version>0.0.12</objectsize.version>
@@ -132,6 +131,12 @@
<dependency>
<groupId>io.prestosql</groupId>
+ <artifactId>presto-server-main</artifactId>
+ <version>${presto.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>io.prestosql</groupId>
<artifactId>presto-cli</artifactId>
<version>${presto.version}</version>
</dependency>
diff --git a/pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto/PulsarConnectorFactory.java b/pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto/PulsarConnectorFactory.java
index f495cb4..a7d4198 100644
--- a/pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto/PulsarConnectorFactory.java
+++ b/pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto/PulsarConnectorFactory.java
@@ -50,9 +50,7 @@ public class PulsarConnectorFactory implements ConnectorFactory {
@Override
public Connector create(String connectorId, Map<String, String> config, ConnectorContext context) {
requireNonNull(config, "requiredConfig is null");
- if (log.isDebugEnabled()) {
- log.debug("Creating Pulsar connector with configs: %s", config);
- }
+ log.info("Creating Pulsar connector with configs: %s", config);
try {
// A plugin is not required to use Guice; it is just very convenient
Bootstrap app = new Bootstrap(
[pulsar] 46/46: Upgrade Lombok to 1.18.20 (#10259)
Posted by eo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit e7670be4055ee9f9ee58723fddaf98727f83649a
Author: Lari Hotari <lh...@users.noreply.github.com>
AuthorDate: Mon Apr 19 14:40:45 2021 +0300
Upgrade Lombok to 1.18.20 (#10259)
- see https://projectlombok.org/changelog
- Fixes compilation on JDK 16
(cherry picked from commit 82c5b81055c20cdb2bbd8c64873a9c4c5049593f)
---
pom.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/pom.xml b/pom.xml
index d590f20..258fabb 100644
--- a/pom.xml
+++ b/pom.xml
@@ -172,7 +172,7 @@ flexible messaging model and an intuitive client API.</description>
<hppc.version>0.7.3</hppc.version>
<spark-streaming_2.10.version>2.1.0</spark-streaming_2.10.version>
<assertj-core.version>3.18.1</assertj-core.version>
- <lombok.version>1.18.16</lombok.version>
+ <lombok.version>1.18.20</lombok.version>
<javax.annotation-api.version>1.2</javax.annotation-api.version>
<jaxb-api>2.3.1</jaxb-api>
<javax.activation.version>1.2.0</javax.activation.version>
[pulsar] 08/46: Switch Docker image to Java 11
Posted by eo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 830d79803e32cc8b246ff0702b7c92a5a261c137
Author: Chris Bartholomew <c_...@yahoo.com>
AuthorDate: Sun Sep 13 10:35:54 2020 -0400
Switch Docker image to Java 11
(cherry picked from commit 13c35538fa1db38e16bf0aecb191ade93be88d80)
---
docker/pulsar-standalone/Dockerfile | 2 +-
docker/pulsar/Dockerfile | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/docker/pulsar-standalone/Dockerfile b/docker/pulsar-standalone/Dockerfile
index 0035de5..777541b 100644
--- a/docker/pulsar-standalone/Dockerfile
+++ b/docker/pulsar-standalone/Dockerfile
@@ -24,7 +24,7 @@ FROM apachepulsar/pulsar-all:latest as pulsar
FROM apachepulsar/pulsar-dashboard:latest as dashboard
# Restart from
-FROM openjdk:8-jdk
+FROM openjdk:11-jdk
# Note that the libpq-dev package is needed here in order to install
# the required python psycopg2 package (for postgresql) later
diff --git a/docker/pulsar/Dockerfile b/docker/pulsar/Dockerfile
index 23820fc..54609e8 100644
--- a/docker/pulsar/Dockerfile
+++ b/docker/pulsar/Dockerfile
@@ -38,7 +38,7 @@ COPY scripts/install-pulsar-client-37.sh /pulsar/bin
### Create 2nd stage from OpenJDK image
### and add Python dependencies (for Pulsar functions)
-FROM openjdk:8-jdk-slim
+FROM openjdk:11-jdk-slim
# Install some utilities
RUN apt-get update \
[pulsar] 43/46: [pulsar-broker] Dispatch messaages to consumer with
permits (#10417)
Posted by eo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit a26de54f1bfe68fdd3d2084dc4505fceb768f96c
Author: Rajan Dhabalia <rd...@apache.org>
AuthorDate: Tue May 11 15:31:50 2021 -0700
[pulsar-broker] Dispatch messaages to consumer with permits (#10417)
* [pulsar-broker] Dispatch messaages to consumer with permits
* move test
(cherry picked from commit 3550f2e7c1bf41ff548d2cf5d16ae50ebf4c0556)
---
.../PersistentDispatcherMultipleConsumers.java | 25 ++++++---
.../pulsar/client/api/ConsumerRedeliveryTest.java | 59 ++++++++++++++++++++++
.../apache/pulsar/client/impl/ConsumerBase.java | 11 ++++
3 files changed, 89 insertions(+), 6 deletions(-)
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherMultipleConsumers.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherMultipleConsumers.java
index f3c6d94..bba7c98 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherMultipleConsumers.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherMultipleConsumers.java
@@ -249,8 +249,9 @@ public class PersistentDispatcherMultipleConsumers extends AbstractDispatcherMul
public synchronized void readMoreEntries() {
// totalAvailablePermits may be updated by other threads
- int currentTotalAvailablePermits = totalAvailablePermits;
- if (currentTotalAvailablePermits > 0 && isAtleastOneConsumerAvailable()) {
+ int firstAvailableConsumerPermits = getFirstAvailableConsumerPermits();
+ int currentTotalAvailablePermits = Math.max(totalAvailablePermits, firstAvailableConsumerPermits);
+ if (currentTotalAvailablePermits > 0 && firstAvailableConsumerPermits > 0) {
int messagesToRead = calculateNumOfMessageToRead(currentTotalAvailablePermits);
if (-1 == messagesToRead) {
@@ -510,7 +511,15 @@ public class PersistentDispatcherMultipleConsumers extends AbstractDispatcherMul
long totalMessagesSent = 0;
long totalBytesSent = 0;
- while (entriesToDispatch > 0 && totalAvailablePermits > 0 && isAtleastOneConsumerAvailable()) {
+ int firstAvailableConsumerPermits, currentTotalAvailablePermits;
+ boolean dispatchMessage;
+ while (entriesToDispatch > 0) {
+ firstAvailableConsumerPermits = getFirstAvailableConsumerPermits();
+ currentTotalAvailablePermits = Math.max(totalAvailablePermits, firstAvailableConsumerPermits);
+ dispatchMessage = currentTotalAvailablePermits > 0 && firstAvailableConsumerPermits > 0;
+ if (!dispatchMessage) {
+ break;
+ }
Consumer c = getNextConsumer();
if (c == null) {
// Do nothing, cursor will be rewind at reconnection
@@ -668,16 +677,20 @@ public class PersistentDispatcherMultipleConsumers extends AbstractDispatcherMul
* @return
*/
protected boolean isAtleastOneConsumerAvailable() {
+ return getFirstAvailableConsumerPermits() > 0;
+ }
+
+ protected int getFirstAvailableConsumerPermits() {
if (consumerList.isEmpty() || IS_CLOSED_UPDATER.get(this) == TRUE) {
// abort read if no consumers are connected or if disconnect is initiated
- return false;
+ return 0;
}
for(Consumer consumer : consumerList) {
if (isConsumerAvailable(consumer)) {
- return true;
+ return consumer.getAvailablePermits();
}
}
- return false;
+ return 0;
}
private boolean isConsumerWritable() {
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/ConsumerRedeliveryTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/ConsumerRedeliveryTest.java
index e828598..95cf90a 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/ConsumerRedeliveryTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/ConsumerRedeliveryTest.java
@@ -21,6 +21,7 @@ package org.apache.pulsar.client.api;
import java.util.ArrayList;
import java.util.List;
import java.util.Set;
+import java.util.UUID;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.TimeUnit;
@@ -29,6 +30,8 @@ import lombok.Cleanup;
import org.apache.pulsar.client.impl.ConsumerImpl;
import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
import org.testng.annotations.AfterClass;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Test;
@@ -40,6 +43,9 @@ import static org.testng.Assert.assertTrue;
import static org.testng.Assert.assertEquals;
public class ConsumerRedeliveryTest extends ProducerConsumerBase {
+
+ private static final Logger log = LoggerFactory.getLogger(ConsumerRedeliveryTest.class);
+
@BeforeClass
@Override
protected void setup() throws Exception {
@@ -179,4 +185,57 @@ public class ConsumerRedeliveryTest extends ProducerConsumerBase {
consumer.close();
}
+ /**
+ * Validates broker should dispatch messages to consumer which still has the permit to consume more messages.
+ *
+ * @throws Exception
+ */
+ @Test
+ public void testConsumerWithPermitReceiveBatchMessages() throws Exception {
+
+ log.info("-- Starting {} test --", methodName);
+
+ final int queueSize = 10;
+ int batchSize = 100;
+ String subName = "my-subscriber-name";
+ String topicName = "permitReceiveBatchMessages"+(UUID.randomUUID().toString());
+ ConsumerImpl<byte[]> consumer1 = (ConsumerImpl<byte[]>) pulsarClient.newConsumer().topic(topicName)
+ .receiverQueueSize(queueSize).subscriptionType(SubscriptionType.Shared).subscriptionName(subName)
+ .subscribe();
+
+ ProducerBuilder<byte[]> producerBuilder = pulsarClient.newProducer().topic(topicName);
+
+ producerBuilder.enableBatching(true);
+ producerBuilder.batchingMaxPublishDelay(2000, TimeUnit.MILLISECONDS);
+ producerBuilder.batchingMaxMessages(100);
+
+ Producer<byte[]> producer = producerBuilder.create();
+ for (int i = 0; i < batchSize; i++) {
+ String message = "my-message-" + i;
+ producer.sendAsync(message.getBytes());
+ }
+ producer.flush();
+
+ for (int i = 0; i < queueSize; i++) {
+ String message = "my-message-" + i;
+ producer.sendAsync(message.getBytes());
+ }
+ producer.flush();
+
+ retryStrategically((test) -> {
+ return consumer1.getTotalIncomingMessages() == batchSize;
+ }, 5, 2000);
+
+ assertEquals(consumer1.getTotalIncomingMessages(), batchSize);
+
+ ConsumerImpl<byte[]> consumer2 = (ConsumerImpl<byte[]>) pulsarClient.newConsumer().topic(topicName)
+ .receiverQueueSize(queueSize).subscriptionType(SubscriptionType.Shared).subscriptionName(subName)
+ .subscribe();
+
+ retryStrategically((test) -> {
+ return consumer2.getTotalIncomingMessages() == queueSize;
+ }, 5, 2000);
+ assertEquals(consumer2.getTotalIncomingMessages(), queueSize);
+ log.info("-- Exiting {} test --", methodName);
+ }
}
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerBase.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerBase.java
index 30c3d01..b296a21 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerBase.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerBase.java
@@ -925,6 +925,17 @@ public abstract class ConsumerBase<T> extends HandlerState implements Consumer<T
return INCOMING_MESSAGES_SIZE_UPDATER.get(this);
}
+ public int getTotalIncomingMessages() {
+ return incomingMessages.size();
+ }
+
+ protected void clearIncomingMessages() {
+ // release messages if they are pooled messages
+ incomingMessages.forEach(Message::release);
+ incomingMessages.clear();
+ resetIncomingMessageSize();
+ }
+
protected abstract void completeOpBatchReceive(OpBatchReceive<T> op);
private static final Logger log = LoggerFactory.getLogger(ConsumerBase.class);
[pulsar] 25/46: GenericObject - support KeyValue in
Message#getValue() (#10107)
Posted by eo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit ab3caca169c35b2ce189d84c6b3834023bfb2e90
Author: Enrico Olivelli <eo...@gmail.com>
AuthorDate: Fri Apr 9 19:29:00 2021 +0200
GenericObject - support KeyValue in Message#getValue() (#10107)
---
.../java/org/apache/pulsar/schema/SchemaTest.java | 48 ++++++++++++++++++++++
.../org/apache/pulsar/client/impl/MessageImpl.java | 13 +++++-
.../client/impl/schema/AutoConsumeSchema.java | 4 ++
3 files changed, 63 insertions(+), 2 deletions(-)
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/schema/SchemaTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/schema/SchemaTest.java
index fba8108..56ca04c 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/schema/SchemaTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/schema/SchemaTest.java
@@ -49,6 +49,8 @@ import org.apache.pulsar.common.naming.TopicDomain;
import org.apache.pulsar.common.naming.TopicName;
import org.apache.pulsar.common.policies.data.ClusterData;
import org.apache.pulsar.common.policies.data.TenantInfo;
+import org.apache.pulsar.common.schema.KeyValue;
+import org.apache.pulsar.common.schema.KeyValueEncodingType;
import org.apache.pulsar.common.schema.SchemaType;
import org.apache.pulsar.common.util.FutureUtil;
import org.testng.annotations.AfterMethod;
@@ -305,6 +307,52 @@ public class SchemaTest extends MockedPulsarServiceBaseTest {
}
@Test
+ public void testKeyValueSchema() throws Exception {
+ final String tenant = PUBLIC_TENANT;
+ final String namespace = "test-namespace-" + randomName(16);
+ final String topicName = "test-string-schema";
+
+ final String topic = TopicName.get(
+ TopicDomain.persistent.value(),
+ tenant,
+ namespace,
+ topicName).toString();
+
+ admin.namespaces().createNamespace(
+ tenant + "/" + namespace,
+ Sets.newHashSet(CLUSTER_NAME));
+
+ admin.topics().createPartitionedTopic(topic, 2);
+
+ Producer<KeyValue<String, Integer>> producer = pulsarClient
+ .newProducer(Schema.KeyValue(Schema.STRING, Schema.INT32, KeyValueEncodingType.INLINE))
+ .topic(topic)
+ .create();
+
+ producer.send(new KeyValue<>("foo", 123));
+
+ Consumer<KeyValue<String, Integer>> consumer = pulsarClient.newConsumer(Schema.KeyValue(Schema.STRING, Schema.INT32, KeyValueEncodingType.INLINE))
+ .subscriptionName("test-sub")
+ .topic(topic)
+ .subscribe();
+
+ Consumer<GenericRecord> consumer2 = pulsarClient.newConsumer(Schema.AUTO_CONSUME())
+ .subscriptionName("test-sub2")
+ .topic(topic)
+ .subscribe();
+
+ producer.send(new KeyValue<>("foo", 123));
+
+ Message<KeyValue<String, Integer>> message = consumer.receive();
+ Message<GenericRecord> message2 = consumer2.receive();
+ assertEquals(message.getValue(), message2.getValue().getNativeObject());
+
+ producer.close();
+ consumer.close();
+ consumer2.close();
+ }
+
+ @Test
public void testIsUsingAvroSchemaParser() {
for (SchemaType value : SchemaType.values()) {
if (value == SchemaType.AVRO || value == SchemaType.JSON || value == SchemaType.PROTOBUF) {
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageImpl.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageImpl.java
index ac387a4..f7d8cf9 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageImpl.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageImpl.java
@@ -41,6 +41,7 @@ import java.util.stream.Collectors;
import org.apache.pulsar.client.api.Message;
import org.apache.pulsar.client.api.MessageId;
import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.impl.schema.AutoConsumeSchema;
import org.apache.pulsar.client.impl.schema.KeyValueSchema;
import org.apache.pulsar.common.protocol.Commands;
import org.apache.pulsar.common.api.EncryptionContext;
@@ -306,8 +307,16 @@ public class MessageImpl<T> implements Message<T> {
}
}
+ private KeyValueSchema getKeyValueSchema() {
+ if (schema instanceof AutoConsumeSchema) {
+ return (KeyValueSchema) ((AutoConsumeSchema) schema).getInternalSchema();
+ } else {
+ return (KeyValueSchema) schema;
+ }
+ }
+
private T getKeyValueBySchemaVersion() {
- KeyValueSchema kvSchema = (KeyValueSchema) schema;
+ KeyValueSchema kvSchema = getKeyValueSchema();
byte[] schemaVersion = getSchemaVersion();
if (kvSchema.getKeyValueEncodingType() == KeyValueEncodingType.SEPARATED) {
return (T) kvSchema.decode(
@@ -319,7 +328,7 @@ public class MessageImpl<T> implements Message<T> {
}
private T getKeyValue() {
- KeyValueSchema kvSchema = (KeyValueSchema) schema;
+ KeyValueSchema kvSchema = getKeyValueSchema();
if (kvSchema.getKeyValueEncodingType() == KeyValueEncodingType.SEPARATED) {
return (T) kvSchema.decode(
msgMetadataBuilder.hasNullPartitionKey() ? null : getKeyBytes(),
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java
index 6ab92ad..647a87b 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java
@@ -250,4 +250,8 @@ public class AutoConsumeSchema implements Schema<GenericRecord> {
return GenericObjectWrapper.of(value,
this.schema.getSchemaInfo().getType(), schemaVersion);
}
+
+ public Schema<?> getInternalSchema() {
+ return schema;
+ }
}
[pulsar] 16/46: remove underscore from version,
it is not valid for deb package
Posted by eo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 98d4e75d09aa9714fb8f34b9f6cc51ca37e6d938
Author: Enrico Olivelli <eo...@apache.org>
AuthorDate: Mon May 10 08:41:00 2021 +0200
remove underscore from version, it is not valid for deb package
---
bouncy-castle/bc/pom.xml | 2 +-
bouncy-castle/bcfips-include-test/pom.xml | 2 +-
bouncy-castle/bcfips/pom.xml | 2 +-
bouncy-castle/pom.xml | 2 +-
buildtools/pom.xml | 2 +-
dashboard/pom.xml | 2 +-
distribution/io/pom.xml | 2 +-
distribution/offloaders/pom.xml | 2 +-
distribution/pom.xml | 2 +-
distribution/server/pom.xml | 2 +-
docker/grafana/pom.xml | 2 +-
docker/pom.xml | 2 +-
docker/pulsar-all/pom.xml | 2 +-
docker/pulsar-standalone/pom.xml | 2 +-
docker/pulsar/pom.xml | 2 +-
jclouds-shaded/pom.xml | 2 +-
kafka-connect-avro-converter-shaded/pom.xml | 2 +-
managed-ledger/pom.xml | 2 +-
pom.xml | 2 +-
protobuf-shaded/pom.xml | 2 +-
pulsar-broker-auth-athenz/pom.xml | 2 +-
pulsar-broker-auth-sasl/pom.xml | 2 +-
pulsar-broker-common/pom.xml | 2 +-
pulsar-broker-shaded/pom.xml | 2 +-
pulsar-broker/pom.xml | 2 +-
pulsar-client-1x-base/pom.xml | 2 +-
pulsar-client-1x-base/pulsar-client-1x/pom.xml | 2 +-
pulsar-client-1x-base/pulsar-client-2x-shaded/pom.xml | 2 +-
pulsar-client-admin-shaded/pom.xml | 2 +-
pulsar-client-admin/pom.xml | 2 +-
pulsar-client-all/pom.xml | 2 +-
pulsar-client-api/pom.xml | 2 +-
pulsar-client-auth-athenz/pom.xml | 2 +-
pulsar-client-auth-sasl/pom.xml | 2 +-
pulsar-client-messagecrypto-bc/pom.xml | 2 +-
pulsar-client-shaded/pom.xml | 2 +-
pulsar-client-tools-test/pom.xml | 2 +-
pulsar-client-tools/pom.xml | 2 +-
pulsar-client/pom.xml | 2 +-
pulsar-common/pom.xml | 2 +-
pulsar-config-validation/pom.xml | 2 +-
pulsar-discovery-service/pom.xml | 2 +-
pulsar-functions/api-java/pom.xml | 2 +-
pulsar-functions/instance/pom.xml | 2 +-
pulsar-functions/java-examples/pom.xml | 2 +-
pulsar-functions/localrun-shaded/pom.xml | 2 +-
pulsar-functions/localrun/pom.xml | 2 +-
pulsar-functions/pom.xml | 2 +-
pulsar-functions/proto/pom.xml | 2 +-
pulsar-functions/runtime-all/pom.xml | 2 +-
pulsar-functions/runtime/pom.xml | 2 +-
pulsar-functions/secrets/pom.xml | 2 +-
pulsar-functions/utils/pom.xml | 2 +-
pulsar-functions/worker/pom.xml | 2 +-
pulsar-io/aerospike/pom.xml | 2 +-
pulsar-io/aws/pom.xml | 2 +-
pulsar-io/batch-data-generator/pom.xml | 2 +-
pulsar-io/batch-discovery-triggerers/pom.xml | 2 +-
pulsar-io/canal/pom.xml | 2 +-
pulsar-io/cassandra/pom.xml | 2 +-
pulsar-io/common/pom.xml | 2 +-
pulsar-io/core/pom.xml | 2 +-
pulsar-io/data-generator/pom.xml | 2 +-
pulsar-io/debezium/core/pom.xml | 2 +-
pulsar-io/debezium/mongodb/pom.xml | 2 +-
pulsar-io/debezium/mysql/pom.xml | 2 +-
pulsar-io/debezium/pom.xml | 2 +-
pulsar-io/debezium/postgres/pom.xml | 2 +-
pulsar-io/docs/pom.xml | 2 +-
pulsar-io/dynamodb/pom.xml | 2 +-
pulsar-io/elastic-search/pom.xml | 2 +-
pulsar-io/file/pom.xml | 2 +-
pulsar-io/flume/pom.xml | 2 +-
pulsar-io/hbase/pom.xml | 2 +-
pulsar-io/hdfs2/pom.xml | 2 +-
pulsar-io/hdfs3/pom.xml | 2 +-
pulsar-io/influxdb/pom.xml | 2 +-
pulsar-io/jdbc/clickhouse/pom.xml | 2 +-
pulsar-io/jdbc/core/pom.xml | 2 +-
pulsar-io/jdbc/mariadb/pom.xml | 2 +-
pulsar-io/jdbc/pom.xml | 2 +-
pulsar-io/jdbc/postgres/pom.xml | 2 +-
pulsar-io/jdbc/sqlite/pom.xml | 2 +-
pulsar-io/kafka-connect-adaptor/pom.xml | 2 +-
pulsar-io/kafka/pom.xml | 2 +-
pulsar-io/kinesis/pom.xml | 2 +-
pulsar-io/mongo/pom.xml | 2 +-
pulsar-io/netty/pom.xml | 2 +-
pulsar-io/nsq/pom.xml | 2 +-
pulsar-io/pom.xml | 2 +-
pulsar-io/rabbitmq/pom.xml | 2 +-
pulsar-io/redis/pom.xml | 2 +-
pulsar-io/solr/pom.xml | 2 +-
pulsar-io/twitter/pom.xml | 2 +-
pulsar-metadata/pom.xml | 2 +-
pulsar-proxy/pom.xml | 2 +-
pulsar-sql/pom.xml | 2 +-
pulsar-sql/presto-distribution/pom.xml | 2 +-
pulsar-sql/presto-pulsar-plugin/pom.xml | 2 +-
pulsar-sql/presto-pulsar/pom.xml | 2 +-
pulsar-testclient/pom.xml | 2 +-
pulsar-transaction/common/pom.xml | 2 +-
pulsar-transaction/coordinator/pom.xml | 2 +-
pulsar-transaction/pom.xml | 2 +-
pulsar-websocket/pom.xml | 2 +-
pulsar-zookeeper-utils/pom.xml | 2 +-
pulsar-zookeeper/pom.xml | 2 +-
testmocks/pom.xml | 2 +-
tests/bc_2_0_0/pom.xml | 2 +-
tests/bc_2_0_1/pom.xml | 2 +-
tests/docker-images/java-test-functions/pom.xml | 2 +-
tests/docker-images/latest-version-image/pom.xml | 2 +-
tests/docker-images/pom.xml | 2 +-
tests/integration/pom.xml | 2 +-
tests/pom.xml | 2 +-
tests/pulsar-client-admin-shade-test/pom.xml | 2 +-
tests/pulsar-client-all-shade-test/pom.xml | 2 +-
tests/pulsar-client-shade-test/pom.xml | 2 +-
tiered-storage/file-system/pom.xml | 2 +-
tiered-storage/jcloud/pom.xml | 2 +-
tiered-storage/pom.xml | 2 +-
121 files changed, 121 insertions(+), 121 deletions(-)
diff --git a/bouncy-castle/bc/pom.xml b/bouncy-castle/bc/pom.xml
index 77fac2c..8a6a06e 100644
--- a/bouncy-castle/bc/pom.xml
+++ b/bouncy-castle/bc/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>bouncy-castle-parent</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/bouncy-castle/bcfips-include-test/pom.xml b/bouncy-castle/bcfips-include-test/pom.xml
index 0e51db6..ead5a7b 100644
--- a/bouncy-castle/bcfips-include-test/pom.xml
+++ b/bouncy-castle/bcfips-include-test/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>bouncy-castle-parent</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/bouncy-castle/bcfips/pom.xml b/bouncy-castle/bcfips/pom.xml
index 45c2ebf..db46b42 100644
--- a/bouncy-castle/bcfips/pom.xml
+++ b/bouncy-castle/bcfips/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>bouncy-castle-parent</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/bouncy-castle/pom.xml b/bouncy-castle/pom.xml
index 14061a4..9c6f553 100644
--- a/bouncy-castle/pom.xml
+++ b/bouncy-castle/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/buildtools/pom.xml b/buildtools/pom.xml
index 6656abe..31808cd 100644
--- a/buildtools/pom.xml
+++ b/buildtools/pom.xml
@@ -30,7 +30,7 @@
<groupId>org.apache.pulsar</groupId>
<artifactId>buildtools</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<packaging>jar</packaging>
<name>Pulsar Build Tools</name>
diff --git a/dashboard/pom.xml b/dashboard/pom.xml
index 5a23db5..e20c19f 100644
--- a/dashboard/pom.xml
+++ b/dashboard/pom.xml
@@ -23,7 +23,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>docker-images</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<relativePath>../docker</relativePath>
</parent>
<modelVersion>4.0.0</modelVersion>
diff --git a/distribution/io/pom.xml b/distribution/io/pom.xml
index cbbe128..8523d9c 100644
--- a/distribution/io/pom.xml
+++ b/distribution/io/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>distribution</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/distribution/offloaders/pom.xml b/distribution/offloaders/pom.xml
index 2b4ef1a..f802fa0 100644
--- a/distribution/offloaders/pom.xml
+++ b/distribution/offloaders/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>distribution</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/distribution/pom.xml b/distribution/pom.xml
index 9e90d26..eb05d6b 100644
--- a/distribution/pom.xml
+++ b/distribution/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/distribution/server/pom.xml b/distribution/server/pom.xml
index 0ae9d04..fb8633e 100644
--- a/distribution/server/pom.xml
+++ b/distribution/server/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>distribution</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/docker/grafana/pom.xml b/docker/grafana/pom.xml
index 3899e3b..e1fe8d6 100644
--- a/docker/grafana/pom.xml
+++ b/docker/grafana/pom.xml
@@ -23,7 +23,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>docker-images</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>grafana-docker-image</artifactId>
diff --git a/docker/pom.xml b/docker/pom.xml
index 7ff30bf..b07058f 100644
--- a/docker/pom.xml
+++ b/docker/pom.xml
@@ -26,7 +26,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>docker-images</artifactId>
<name>Apache Pulsar :: Docker Images</name>
diff --git a/docker/pulsar-all/pom.xml b/docker/pulsar-all/pom.xml
index 272fa8a..4cedb3c 100644
--- a/docker/pulsar-all/pom.xml
+++ b/docker/pulsar-all/pom.xml
@@ -23,7 +23,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>docker-images</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>pulsar-all-docker-image</artifactId>
diff --git a/docker/pulsar-standalone/pom.xml b/docker/pulsar-standalone/pom.xml
index b5e5ba1..e3a60d4 100644
--- a/docker/pulsar-standalone/pom.xml
+++ b/docker/pulsar-standalone/pom.xml
@@ -23,7 +23,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>docker-images</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>pulsar-standalone-docker-image</artifactId>
diff --git a/docker/pulsar/pom.xml b/docker/pulsar/pom.xml
index 9055441..4b5de19 100644
--- a/docker/pulsar/pom.xml
+++ b/docker/pulsar/pom.xml
@@ -23,7 +23,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>docker-images</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>pulsar-docker-image</artifactId>
diff --git a/jclouds-shaded/pom.xml b/jclouds-shaded/pom.xml
index 43e85e5..7896c4e 100644
--- a/jclouds-shaded/pom.xml
+++ b/jclouds-shaded/pom.xml
@@ -26,7 +26,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/kafka-connect-avro-converter-shaded/pom.xml b/kafka-connect-avro-converter-shaded/pom.xml
index 9f26f7c..38ec729 100644
--- a/kafka-connect-avro-converter-shaded/pom.xml
+++ b/kafka-connect-avro-converter-shaded/pom.xml
@@ -26,7 +26,7 @@
<parent>
<artifactId>pulsar</artifactId>
<groupId>org.apache.pulsar</groupId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/managed-ledger/pom.xml b/managed-ledger/pom.xml
index 8f7efea..c26940d 100644
--- a/managed-ledger/pom.xml
+++ b/managed-ledger/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pom.xml b/pom.xml
index e9cc7cc..44ec252 100644
--- a/pom.xml
+++ b/pom.xml
@@ -29,7 +29,7 @@
</parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<name>Pulsar</name>
<description>Pulsar is a distributed pub-sub messaging platform with a very
flexible messaging model and an intuitive client API.</description>
diff --git a/protobuf-shaded/pom.xml b/protobuf-shaded/pom.xml
index 49a1201..535d6c5 100644
--- a/protobuf-shaded/pom.xml
+++ b/protobuf-shaded/pom.xml
@@ -26,7 +26,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-broker-auth-athenz/pom.xml b/pulsar-broker-auth-athenz/pom.xml
index 4c74389..f197e29 100644
--- a/pulsar-broker-auth-athenz/pom.xml
+++ b/pulsar-broker-auth-athenz/pom.xml
@@ -26,7 +26,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-broker-auth-athenz</artifactId>
diff --git a/pulsar-broker-auth-sasl/pom.xml b/pulsar-broker-auth-sasl/pom.xml
index 3ced729..4d719ce 100644
--- a/pulsar-broker-auth-sasl/pom.xml
+++ b/pulsar-broker-auth-sasl/pom.xml
@@ -26,7 +26,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-broker-auth-sasl</artifactId>
diff --git a/pulsar-broker-common/pom.xml b/pulsar-broker-common/pom.xml
index ed8d978..f67595e 100644
--- a/pulsar-broker-common/pom.xml
+++ b/pulsar-broker-common/pom.xml
@@ -26,7 +26,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-broker-common</artifactId>
diff --git a/pulsar-broker-shaded/pom.xml b/pulsar-broker-shaded/pom.xml
index 22add27..35c67a7 100644
--- a/pulsar-broker-shaded/pom.xml
+++ b/pulsar-broker-shaded/pom.xml
@@ -26,7 +26,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-broker/pom.xml b/pulsar-broker/pom.xml
index c811db6..c3c3dc5 100644
--- a/pulsar-broker/pom.xml
+++ b/pulsar-broker/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-client-1x-base/pom.xml b/pulsar-client-1x-base/pom.xml
index c4baae7..5d9ff19 100644
--- a/pulsar-client-1x-base/pom.xml
+++ b/pulsar-client-1x-base/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-client-1x-base/pulsar-client-1x/pom.xml b/pulsar-client-1x-base/pulsar-client-1x/pom.xml
index 8754d7e..9e160c8 100644
--- a/pulsar-client-1x-base/pulsar-client-1x/pom.xml
+++ b/pulsar-client-1x-base/pulsar-client-1x/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-client-1x-base</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-client-1x-base/pulsar-client-2x-shaded/pom.xml b/pulsar-client-1x-base/pulsar-client-2x-shaded/pom.xml
index 108b367..3daac79 100644
--- a/pulsar-client-1x-base/pulsar-client-2x-shaded/pom.xml
+++ b/pulsar-client-1x-base/pulsar-client-2x-shaded/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-client-1x-base</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-client-admin-shaded/pom.xml b/pulsar-client-admin-shaded/pom.xml
index d0e0cc5..7e3eadb 100644
--- a/pulsar-client-admin-shaded/pom.xml
+++ b/pulsar-client-admin-shaded/pom.xml
@@ -26,7 +26,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-client-admin/pom.xml b/pulsar-client-admin/pom.xml
index 432c33b..c66e8fe 100644
--- a/pulsar-client-admin/pom.xml
+++ b/pulsar-client-admin/pom.xml
@@ -26,7 +26,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-client-all/pom.xml b/pulsar-client-all/pom.xml
index 4623f57..20ce941 100644
--- a/pulsar-client-all/pom.xml
+++ b/pulsar-client-all/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-client-api/pom.xml b/pulsar-client-api/pom.xml
index ffb69a4..3b165a6 100644
--- a/pulsar-client-api/pom.xml
+++ b/pulsar-client-api/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-client-auth-athenz/pom.xml b/pulsar-client-auth-athenz/pom.xml
index b71fcca..c28d67b 100644
--- a/pulsar-client-auth-athenz/pom.xml
+++ b/pulsar-client-auth-athenz/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-client-auth-sasl/pom.xml b/pulsar-client-auth-sasl/pom.xml
index d42172b..6d6de31 100644
--- a/pulsar-client-auth-sasl/pom.xml
+++ b/pulsar-client-auth-sasl/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-client-messagecrypto-bc/pom.xml b/pulsar-client-messagecrypto-bc/pom.xml
index bf0882b..d679efd 100644
--- a/pulsar-client-messagecrypto-bc/pom.xml
+++ b/pulsar-client-messagecrypto-bc/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-client-shaded/pom.xml b/pulsar-client-shaded/pom.xml
index 66b65b1..2b875a6 100644
--- a/pulsar-client-shaded/pom.xml
+++ b/pulsar-client-shaded/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-client-tools-test/pom.xml b/pulsar-client-tools-test/pom.xml
index 6e66a8e..7e95255 100644
--- a/pulsar-client-tools-test/pom.xml
+++ b/pulsar-client-tools-test/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-client-tools/pom.xml b/pulsar-client-tools/pom.xml
index 89b103c..ee2b3a6 100644
--- a/pulsar-client-tools/pom.xml
+++ b/pulsar-client-tools/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-client/pom.xml b/pulsar-client/pom.xml
index f2d6370..24920fc 100644
--- a/pulsar-client/pom.xml
+++ b/pulsar-client/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-common/pom.xml b/pulsar-common/pom.xml
index 2fcf9fd..bc72cf2 100644
--- a/pulsar-common/pom.xml
+++ b/pulsar-common/pom.xml
@@ -26,7 +26,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-config-validation/pom.xml b/pulsar-config-validation/pom.xml
index 5dd64c4..e683cb9 100644
--- a/pulsar-config-validation/pom.xml
+++ b/pulsar-config-validation/pom.xml
@@ -26,7 +26,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-discovery-service/pom.xml b/pulsar-discovery-service/pom.xml
index 65bd2b2..065a177 100644
--- a/pulsar-discovery-service/pom.xml
+++ b/pulsar-discovery-service/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-functions/api-java/pom.xml b/pulsar-functions/api-java/pom.xml
index 0bb9733..7f9d6eb 100644
--- a/pulsar-functions/api-java/pom.xml
+++ b/pulsar-functions/api-java/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-functions</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-functions-api</artifactId>
diff --git a/pulsar-functions/instance/pom.xml b/pulsar-functions/instance/pom.xml
index a6d05d5..f735bc3 100644
--- a/pulsar-functions/instance/pom.xml
+++ b/pulsar-functions/instance/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-functions</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-functions-instance</artifactId>
diff --git a/pulsar-functions/java-examples/pom.xml b/pulsar-functions/java-examples/pom.xml
index fa8f400..dfe1693 100644
--- a/pulsar-functions/java-examples/pom.xml
+++ b/pulsar-functions/java-examples/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-functions</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-functions-api-examples</artifactId>
diff --git a/pulsar-functions/localrun-shaded/pom.xml b/pulsar-functions/localrun-shaded/pom.xml
index 858f724..66c44f6 100644
--- a/pulsar-functions/localrun-shaded/pom.xml
+++ b/pulsar-functions/localrun-shaded/pom.xml
@@ -26,7 +26,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-functions</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-functions/localrun/pom.xml b/pulsar-functions/localrun/pom.xml
index 6d13bcb..f0d948e 100644
--- a/pulsar-functions/localrun/pom.xml
+++ b/pulsar-functions/localrun/pom.xml
@@ -26,7 +26,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-functions</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-functions/pom.xml b/pulsar-functions/pom.xml
index 0147e7e..21eed8b 100644
--- a/pulsar-functions/pom.xml
+++ b/pulsar-functions/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-functions</artifactId>
diff --git a/pulsar-functions/proto/pom.xml b/pulsar-functions/proto/pom.xml
index 5d35a6f..2a13b99 100644
--- a/pulsar-functions/proto/pom.xml
+++ b/pulsar-functions/proto/pom.xml
@@ -27,7 +27,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-functions</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-functions-proto</artifactId>
diff --git a/pulsar-functions/runtime-all/pom.xml b/pulsar-functions/runtime-all/pom.xml
index 0d25d27..122f902 100644
--- a/pulsar-functions/runtime-all/pom.xml
+++ b/pulsar-functions/runtime-all/pom.xml
@@ -26,7 +26,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-functions</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-functions/runtime/pom.xml b/pulsar-functions/runtime/pom.xml
index 2a5e77f..bae18f4 100644
--- a/pulsar-functions/runtime/pom.xml
+++ b/pulsar-functions/runtime/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-functions</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-functions-runtime</artifactId>
diff --git a/pulsar-functions/secrets/pom.xml b/pulsar-functions/secrets/pom.xml
index 553f5a4..3c6c78f 100644
--- a/pulsar-functions/secrets/pom.xml
+++ b/pulsar-functions/secrets/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-functions</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-functions-secrets</artifactId>
diff --git a/pulsar-functions/utils/pom.xml b/pulsar-functions/utils/pom.xml
index 293cdac..2dfcf2f 100644
--- a/pulsar-functions/utils/pom.xml
+++ b/pulsar-functions/utils/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-functions</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-functions-utils</artifactId>
diff --git a/pulsar-functions/worker/pom.xml b/pulsar-functions/worker/pom.xml
index c833431..fd3a43d 100644
--- a/pulsar-functions/worker/pom.xml
+++ b/pulsar-functions/worker/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-functions</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-io/aerospike/pom.xml b/pulsar-io/aerospike/pom.xml
index 40032b3..f8ba372 100644
--- a/pulsar-io/aerospike/pom.xml
+++ b/pulsar-io/aerospike/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-io-aerospike</artifactId>
diff --git a/pulsar-io/aws/pom.xml b/pulsar-io/aws/pom.xml
index 10a4071..5ee8c1b 100644
--- a/pulsar-io/aws/pom.xml
+++ b/pulsar-io/aws/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-io-aws</artifactId>
diff --git a/pulsar-io/batch-data-generator/pom.xml b/pulsar-io/batch-data-generator/pom.xml
index 993463b..9ed1e92 100644
--- a/pulsar-io/batch-data-generator/pom.xml
+++ b/pulsar-io/batch-data-generator/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-io-batch-data-generator</artifactId>
diff --git a/pulsar-io/batch-discovery-triggerers/pom.xml b/pulsar-io/batch-discovery-triggerers/pom.xml
index f1aa8ac..686cfb7 100644
--- a/pulsar-io/batch-discovery-triggerers/pom.xml
+++ b/pulsar-io/batch-discovery-triggerers/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-io-batch-discovery-triggerers</artifactId>
diff --git a/pulsar-io/canal/pom.xml b/pulsar-io/canal/pom.xml
index cbf3e7e..352fb92 100644
--- a/pulsar-io/canal/pom.xml
+++ b/pulsar-io/canal/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<modelVersion>4.0.0</modelVersion>
diff --git a/pulsar-io/cassandra/pom.xml b/pulsar-io/cassandra/pom.xml
index 070b2e2..ff418d4 100644
--- a/pulsar-io/cassandra/pom.xml
+++ b/pulsar-io/cassandra/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-io-cassandra</artifactId>
diff --git a/pulsar-io/common/pom.xml b/pulsar-io/common/pom.xml
index 6e28b03..8588cdc 100644
--- a/pulsar-io/common/pom.xml
+++ b/pulsar-io/common/pom.xml
@@ -27,7 +27,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-io-common</artifactId>
diff --git a/pulsar-io/core/pom.xml b/pulsar-io/core/pom.xml
index 3619208..076e06c 100644
--- a/pulsar-io/core/pom.xml
+++ b/pulsar-io/core/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-io-core</artifactId>
diff --git a/pulsar-io/data-generator/pom.xml b/pulsar-io/data-generator/pom.xml
index ef62a08..b098520 100644
--- a/pulsar-io/data-generator/pom.xml
+++ b/pulsar-io/data-generator/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-io-data-generator</artifactId>
diff --git a/pulsar-io/debezium/core/pom.xml b/pulsar-io/debezium/core/pom.xml
index 6213542..05c15b7 100644
--- a/pulsar-io/debezium/core/pom.xml
+++ b/pulsar-io/debezium/core/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io-debezium</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-io-debezium-core</artifactId>
diff --git a/pulsar-io/debezium/mongodb/pom.xml b/pulsar-io/debezium/mongodb/pom.xml
index 535b8d5..01e4eee 100644
--- a/pulsar-io/debezium/mongodb/pom.xml
+++ b/pulsar-io/debezium/mongodb/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io-debezium</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-io-debezium-mongodb</artifactId>
diff --git a/pulsar-io/debezium/mysql/pom.xml b/pulsar-io/debezium/mysql/pom.xml
index edb98f5..cb6322c 100644
--- a/pulsar-io/debezium/mysql/pom.xml
+++ b/pulsar-io/debezium/mysql/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io-debezium</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-io-debezium-mysql</artifactId>
diff --git a/pulsar-io/debezium/pom.xml b/pulsar-io/debezium/pom.xml
index fcdcfb3..188a033 100644
--- a/pulsar-io/debezium/pom.xml
+++ b/pulsar-io/debezium/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-io-debezium</artifactId>
diff --git a/pulsar-io/debezium/postgres/pom.xml b/pulsar-io/debezium/postgres/pom.xml
index 1662601..71df130 100644
--- a/pulsar-io/debezium/postgres/pom.xml
+++ b/pulsar-io/debezium/postgres/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io-debezium</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-io-debezium-postgres</artifactId>
diff --git a/pulsar-io/docs/pom.xml b/pulsar-io/docs/pom.xml
index 4ba8b77..46c57c5 100644
--- a/pulsar-io/docs/pom.xml
+++ b/pulsar-io/docs/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-io-docs</artifactId>
diff --git a/pulsar-io/dynamodb/pom.xml b/pulsar-io/dynamodb/pom.xml
index 6808d65..8c5a2ba 100644
--- a/pulsar-io/dynamodb/pom.xml
+++ b/pulsar-io/dynamodb/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-io-dynamodb</artifactId>
diff --git a/pulsar-io/elastic-search/pom.xml b/pulsar-io/elastic-search/pom.xml
index fca5fe0..ec85837 100644
--- a/pulsar-io/elastic-search/pom.xml
+++ b/pulsar-io/elastic-search/pom.xml
@@ -23,7 +23,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-io-elastic-search</artifactId>
<name>Pulsar IO :: ElasticSearch</name>
diff --git a/pulsar-io/file/pom.xml b/pulsar-io/file/pom.xml
index 1dd0847..ea4fbe1 100644
--- a/pulsar-io/file/pom.xml
+++ b/pulsar-io/file/pom.xml
@@ -23,7 +23,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-io-file</artifactId>
diff --git a/pulsar-io/flume/pom.xml b/pulsar-io/flume/pom.xml
index 47f093b..4e3ef37 100644
--- a/pulsar-io/flume/pom.xml
+++ b/pulsar-io/flume/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-io-flume</artifactId>
diff --git a/pulsar-io/hbase/pom.xml b/pulsar-io/hbase/pom.xml
index 6dd5088..780aa71 100644
--- a/pulsar-io/hbase/pom.xml
+++ b/pulsar-io/hbase/pom.xml
@@ -25,7 +25,7 @@
<parent>
<artifactId>pulsar-io</artifactId>
<groupId>org.apache.pulsar</groupId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-io-hbase</artifactId>
<name>Pulsar IO :: Hbase</name>
diff --git a/pulsar-io/hdfs2/pom.xml b/pulsar-io/hdfs2/pom.xml
index 7c01777..5e34ee4 100644
--- a/pulsar-io/hdfs2/pom.xml
+++ b/pulsar-io/hdfs2/pom.xml
@@ -23,7 +23,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-io-hdfs2</artifactId>
<name>Pulsar IO :: Hdfs2</name>
diff --git a/pulsar-io/hdfs3/pom.xml b/pulsar-io/hdfs3/pom.xml
index af77bc1..2095b8c 100644
--- a/pulsar-io/hdfs3/pom.xml
+++ b/pulsar-io/hdfs3/pom.xml
@@ -23,7 +23,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-io-hdfs3</artifactId>
<name>Pulsar IO :: Hdfs3</name>
diff --git a/pulsar-io/influxdb/pom.xml b/pulsar-io/influxdb/pom.xml
index 0141024..12c8b37 100644
--- a/pulsar-io/influxdb/pom.xml
+++ b/pulsar-io/influxdb/pom.xml
@@ -25,7 +25,7 @@
<parent>
<artifactId>pulsar-io</artifactId>
<groupId>org.apache.pulsar</groupId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-io-influxdb</artifactId>
diff --git a/pulsar-io/jdbc/clickhouse/pom.xml b/pulsar-io/jdbc/clickhouse/pom.xml
index cd8b345..483106f 100644
--- a/pulsar-io/jdbc/clickhouse/pom.xml
+++ b/pulsar-io/jdbc/clickhouse/pom.xml
@@ -24,7 +24,7 @@
<parent>
<artifactId>pulsar-io-jdbc</artifactId>
<groupId>org.apache.pulsar</groupId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<modelVersion>4.0.0</modelVersion>
diff --git a/pulsar-io/jdbc/core/pom.xml b/pulsar-io/jdbc/core/pom.xml
index 2c39e76..49142fb 100644
--- a/pulsar-io/jdbc/core/pom.xml
+++ b/pulsar-io/jdbc/core/pom.xml
@@ -24,7 +24,7 @@
<parent>
<artifactId>pulsar-io-jdbc</artifactId>
<groupId>org.apache.pulsar</groupId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<modelVersion>4.0.0</modelVersion>
diff --git a/pulsar-io/jdbc/mariadb/pom.xml b/pulsar-io/jdbc/mariadb/pom.xml
index f377bfc..dc88d2a 100644
--- a/pulsar-io/jdbc/mariadb/pom.xml
+++ b/pulsar-io/jdbc/mariadb/pom.xml
@@ -24,7 +24,7 @@
<parent>
<artifactId>pulsar-io-jdbc</artifactId>
<groupId>org.apache.pulsar</groupId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<modelVersion>4.0.0</modelVersion>
diff --git a/pulsar-io/jdbc/pom.xml b/pulsar-io/jdbc/pom.xml
index 5988e5f..e327ebf 100644
--- a/pulsar-io/jdbc/pom.xml
+++ b/pulsar-io/jdbc/pom.xml
@@ -32,7 +32,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-io-jdbc</artifactId>
diff --git a/pulsar-io/jdbc/postgres/pom.xml b/pulsar-io/jdbc/postgres/pom.xml
index 3caaeb1..e5d689d 100644
--- a/pulsar-io/jdbc/postgres/pom.xml
+++ b/pulsar-io/jdbc/postgres/pom.xml
@@ -24,7 +24,7 @@
<parent>
<artifactId>pulsar-io-jdbc</artifactId>
<groupId>org.apache.pulsar</groupId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<modelVersion>4.0.0</modelVersion>
diff --git a/pulsar-io/jdbc/sqlite/pom.xml b/pulsar-io/jdbc/sqlite/pom.xml
index 75766d5..c5216bb 100644
--- a/pulsar-io/jdbc/sqlite/pom.xml
+++ b/pulsar-io/jdbc/sqlite/pom.xml
@@ -24,7 +24,7 @@
<parent>
<artifactId>pulsar-io-jdbc</artifactId>
<groupId>org.apache.pulsar</groupId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>pulsar-io-jdbc-sqlite</artifactId>
diff --git a/pulsar-io/kafka-connect-adaptor/pom.xml b/pulsar-io/kafka-connect-adaptor/pom.xml
index 6ca2c14..7e82346 100644
--- a/pulsar-io/kafka-connect-adaptor/pom.xml
+++ b/pulsar-io/kafka-connect-adaptor/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-io-kafka-connect-adaptor</artifactId>
diff --git a/pulsar-io/kafka/pom.xml b/pulsar-io/kafka/pom.xml
index 08ce625..4ded489 100644
--- a/pulsar-io/kafka/pom.xml
+++ b/pulsar-io/kafka/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-io-kafka</artifactId>
diff --git a/pulsar-io/kinesis/pom.xml b/pulsar-io/kinesis/pom.xml
index 996b3d0..50ccfce 100644
--- a/pulsar-io/kinesis/pom.xml
+++ b/pulsar-io/kinesis/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-io-kinesis</artifactId>
diff --git a/pulsar-io/mongo/pom.xml b/pulsar-io/mongo/pom.xml
index 6cd9383..faa3686 100644
--- a/pulsar-io/mongo/pom.xml
+++ b/pulsar-io/mongo/pom.xml
@@ -26,7 +26,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-io-mongo</artifactId>
diff --git a/pulsar-io/netty/pom.xml b/pulsar-io/netty/pom.xml
index 09b2b9f..81dfdc5 100644
--- a/pulsar-io/netty/pom.xml
+++ b/pulsar-io/netty/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-io-netty</artifactId>
diff --git a/pulsar-io/nsq/pom.xml b/pulsar-io/nsq/pom.xml
index 8b1eab8..4358712 100644
--- a/pulsar-io/nsq/pom.xml
+++ b/pulsar-io/nsq/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-io-nsq</artifactId>
diff --git a/pulsar-io/pom.xml b/pulsar-io/pom.xml
index 2f90671..40b476a 100644
--- a/pulsar-io/pom.xml
+++ b/pulsar-io/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-io</artifactId>
diff --git a/pulsar-io/rabbitmq/pom.xml b/pulsar-io/rabbitmq/pom.xml
index 5338b12..bea2e49 100644
--- a/pulsar-io/rabbitmq/pom.xml
+++ b/pulsar-io/rabbitmq/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-io-rabbitmq</artifactId>
diff --git a/pulsar-io/redis/pom.xml b/pulsar-io/redis/pom.xml
index f12fdf5..1bc88f4 100644
--- a/pulsar-io/redis/pom.xml
+++ b/pulsar-io/redis/pom.xml
@@ -25,7 +25,7 @@
<parent>
<artifactId>pulsar-io</artifactId>
<groupId>org.apache.pulsar</groupId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-io-redis</artifactId>
diff --git a/pulsar-io/solr/pom.xml b/pulsar-io/solr/pom.xml
index 318775f..a8d6103 100644
--- a/pulsar-io/solr/pom.xml
+++ b/pulsar-io/solr/pom.xml
@@ -25,7 +25,7 @@
<parent>
<artifactId>pulsar-io</artifactId>
<groupId>org.apache.pulsar</groupId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<properties>
diff --git a/pulsar-io/twitter/pom.xml b/pulsar-io/twitter/pom.xml
index 831aeaa..88fecc6 100644
--- a/pulsar-io/twitter/pom.xml
+++ b/pulsar-io/twitter/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-io-twitter</artifactId>
diff --git a/pulsar-metadata/pom.xml b/pulsar-metadata/pom.xml
index 19872a5..48c93e2 100644
--- a/pulsar-metadata/pom.xml
+++ b/pulsar-metadata/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-proxy/pom.xml b/pulsar-proxy/pom.xml
index 887425c..3e8fab9 100644
--- a/pulsar-proxy/pom.xml
+++ b/pulsar-proxy/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-proxy</artifactId>
diff --git a/pulsar-sql/pom.xml b/pulsar-sql/pom.xml
index cc04d70..22276b7 100644
--- a/pulsar-sql/pom.xml
+++ b/pulsar-sql/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-sql</artifactId>
diff --git a/pulsar-sql/presto-distribution/pom.xml b/pulsar-sql/presto-distribution/pom.xml
index 502f3c8..710eda8 100644
--- a/pulsar-sql/presto-distribution/pom.xml
+++ b/pulsar-sql/presto-distribution/pom.xml
@@ -31,7 +31,7 @@
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-presto-distribution</artifactId>
<name>Pulsar SQL :: Pulsar Presto Distribution</name>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<properties>
<jersey.version>2.31</jersey.version>
diff --git a/pulsar-sql/presto-pulsar-plugin/pom.xml b/pulsar-sql/presto-pulsar-plugin/pom.xml
index 1d8ad55..f00bc66 100644
--- a/pulsar-sql/presto-pulsar-plugin/pom.xml
+++ b/pulsar-sql/presto-pulsar-plugin/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-sql</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-presto-connector</artifactId>
diff --git a/pulsar-sql/presto-pulsar/pom.xml b/pulsar-sql/presto-pulsar/pom.xml
index 67ddea0..69bebbe 100644
--- a/pulsar-sql/presto-pulsar/pom.xml
+++ b/pulsar-sql/presto-pulsar/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-sql</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-presto-connector-original</artifactId>
diff --git a/pulsar-testclient/pom.xml b/pulsar-testclient/pom.xml
index 354304f..131bda7 100644
--- a/pulsar-testclient/pom.xml
+++ b/pulsar-testclient/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-transaction/common/pom.xml b/pulsar-transaction/common/pom.xml
index 82479f7..e99243f 100644
--- a/pulsar-transaction/common/pom.xml
+++ b/pulsar-transaction/common/pom.xml
@@ -27,7 +27,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-transaction-parent</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-transaction-common</artifactId>
diff --git a/pulsar-transaction/coordinator/pom.xml b/pulsar-transaction/coordinator/pom.xml
index 494fc80..6a6e482 100644
--- a/pulsar-transaction/coordinator/pom.xml
+++ b/pulsar-transaction/coordinator/pom.xml
@@ -27,7 +27,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-transaction-parent</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-transaction-coordinator</artifactId>
diff --git a/pulsar-transaction/pom.xml b/pulsar-transaction/pom.xml
index 748e96e..5e16583 100644
--- a/pulsar-transaction/pom.xml
+++ b/pulsar-transaction/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-transaction-parent</artifactId>
diff --git a/pulsar-websocket/pom.xml b/pulsar-websocket/pom.xml
index adf68c7..3df5a89 100644
--- a/pulsar-websocket/pom.xml
+++ b/pulsar-websocket/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-zookeeper-utils/pom.xml b/pulsar-zookeeper-utils/pom.xml
index 4b593d5..d611d7d 100644
--- a/pulsar-zookeeper-utils/pom.xml
+++ b/pulsar-zookeeper-utils/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-zookeeper/pom.xml b/pulsar-zookeeper/pom.xml
index a6f6529..524f7b1 100644
--- a/pulsar-zookeeper/pom.xml
+++ b/pulsar-zookeeper/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/testmocks/pom.xml b/testmocks/pom.xml
index 9c78d16..8454863 100644
--- a/testmocks/pom.xml
+++ b/testmocks/pom.xml
@@ -25,7 +25,7 @@
<parent>
<artifactId>pulsar</artifactId>
<groupId>org.apache.pulsar</groupId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>testmocks</artifactId>
diff --git a/tests/bc_2_0_0/pom.xml b/tests/bc_2_0_0/pom.xml
index 4924abc..20b49d2 100644
--- a/tests/bc_2_0_0/pom.xml
+++ b/tests/bc_2_0_0/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar.tests</groupId>
<artifactId>tests-parent</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>bc_2_0_0</artifactId>
diff --git a/tests/bc_2_0_1/pom.xml b/tests/bc_2_0_1/pom.xml
index b770a29..3fbf924 100644
--- a/tests/bc_2_0_1/pom.xml
+++ b/tests/bc_2_0_1/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar.tests</groupId>
<artifactId>tests-parent</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>bc_2_0_1</artifactId>
diff --git a/tests/docker-images/java-test-functions/pom.xml b/tests/docker-images/java-test-functions/pom.xml
index 748d53b..8c38d3b 100644
--- a/tests/docker-images/java-test-functions/pom.xml
+++ b/tests/docker-images/java-test-functions/pom.xml
@@ -23,7 +23,7 @@
<parent>
<groupId>org.apache.pulsar.tests</groupId>
<artifactId>docker-images</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>java-test-functions</artifactId>
diff --git a/tests/docker-images/latest-version-image/pom.xml b/tests/docker-images/latest-version-image/pom.xml
index bfb6e6b..ba89675 100644
--- a/tests/docker-images/latest-version-image/pom.xml
+++ b/tests/docker-images/latest-version-image/pom.xml
@@ -23,7 +23,7 @@
<parent>
<groupId>org.apache.pulsar.tests</groupId>
<artifactId>docker-images</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>latest-version-image</artifactId>
diff --git a/tests/docker-images/pom.xml b/tests/docker-images/pom.xml
index 5ee8f20..0532fcb 100644
--- a/tests/docker-images/pom.xml
+++ b/tests/docker-images/pom.xml
@@ -26,7 +26,7 @@
<parent>
<groupId>org.apache.pulsar.tests</groupId>
<artifactId>tests-parent</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>docker-images</artifactId>
<name>Apache Pulsar :: Tests :: Docker Images</name>
diff --git a/tests/integration/pom.xml b/tests/integration/pom.xml
index b147b46..667ea67 100644
--- a/tests/integration/pom.xml
+++ b/tests/integration/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar.tests</groupId>
<artifactId>tests-parent</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>integration</artifactId>
diff --git a/tests/pom.xml b/tests/pom.xml
index 59fc9c2..7046c46 100644
--- a/tests/pom.xml
+++ b/tests/pom.xml
@@ -26,7 +26,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<groupId>org.apache.pulsar.tests</groupId>
<artifactId>tests-parent</artifactId>
diff --git a/tests/pulsar-client-admin-shade-test/pom.xml b/tests/pulsar-client-admin-shade-test/pom.xml
index 9c05735..8da4693 100644
--- a/tests/pulsar-client-admin-shade-test/pom.xml
+++ b/tests/pulsar-client-admin-shade-test/pom.xml
@@ -26,7 +26,7 @@
<parent>
<groupId>org.apache.pulsar.tests</groupId>
<artifactId>tests-parent</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-client-admin-shade-test</artifactId>
diff --git a/tests/pulsar-client-all-shade-test/pom.xml b/tests/pulsar-client-all-shade-test/pom.xml
index b2d045e..5d8d3af 100644
--- a/tests/pulsar-client-all-shade-test/pom.xml
+++ b/tests/pulsar-client-all-shade-test/pom.xml
@@ -26,7 +26,7 @@
<parent>
<groupId>org.apache.pulsar.tests</groupId>
<artifactId>tests-parent</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-client-all-shade-test</artifactId>
diff --git a/tests/pulsar-client-shade-test/pom.xml b/tests/pulsar-client-shade-test/pom.xml
index b7d04b6..d483bad 100644
--- a/tests/pulsar-client-shade-test/pom.xml
+++ b/tests/pulsar-client-shade-test/pom.xml
@@ -27,7 +27,7 @@
<parent>
<groupId>org.apache.pulsar.tests</groupId>
<artifactId>tests-parent</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
</parent>
<artifactId>pulsar-client-shade-test</artifactId>
diff --git a/tiered-storage/file-system/pom.xml b/tiered-storage/file-system/pom.xml
index fc5657b..df50d1d 100644
--- a/tiered-storage/file-system/pom.xml
+++ b/tiered-storage/file-system/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>tiered-storage-parent</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/tiered-storage/jcloud/pom.xml b/tiered-storage/jcloud/pom.xml
index 462e706..35fcc54 100644
--- a/tiered-storage/jcloud/pom.xml
+++ b/tiered-storage/jcloud/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>tiered-storage-parent</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/tiered-storage/pom.xml b/tiered-storage/pom.xml
index 72d426b..ec10e48 100644
--- a/tiered-storage/pom.xml
+++ b/tiered-storage/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2_1.0.0</version>
+ <version>2.7.2.1.0.0</version>
<relativePath>..</relativePath>
</parent>
[pulsar] 29/46: Java Client: MessageImpl - ensure that
AutoConsumeSchema downloaded the schema before decoding the payload
(#10248)
Posted by eo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit ff5924492d784fb69d795d694e3e58183dd989c9
Author: Enrico Olivelli <eo...@gmail.com>
AuthorDate: Mon Apr 19 14:18:29 2021 +0200
Java Client: MessageImpl - ensure that AutoConsumeSchema downloaded the schema before decoding the payload (#10248)
(cherry picked from commit 54523bbff9b2b2b861409a93cfb3712bfdb73f27)
---
.../org/apache/pulsar/client/impl/MessageImpl.java | 11 +-
.../client/impl/schema/AutoConsumeSchema.java | 72 ++++++++----
.../integration/io/TestGenericObjectSink.java | 4 +-
.../io/PulsarGenericObjectSinkTest.java | 123 ++++++++++-----------
.../suites/PulsarStandaloneTestSuite.java | 1 +
.../topologies/PulsarStandaloneTestBase.java | 15 +++
6 files changed, 137 insertions(+), 89 deletions(-)
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageImpl.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageImpl.java
index 3cb2235..a148677 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageImpl.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageImpl.java
@@ -49,6 +49,7 @@ import org.apache.pulsar.common.api.proto.PulsarApi;
import org.apache.pulsar.common.api.proto.PulsarApi.KeyValue;
import org.apache.pulsar.common.api.proto.PulsarApi.MessageMetadata;
import org.apache.pulsar.common.schema.KeyValueEncodingType;
+import org.apache.pulsar.common.schema.SchemaInfo;
import org.apache.pulsar.common.schema.SchemaType;
public class MessageImpl<T> implements Message<T> {
@@ -279,10 +280,18 @@ public class MessageImpl<T> implements Message<T> {
}
}
+ private SchemaInfo getSchemaInfo() {
+ if (schema instanceof AutoConsumeSchema) {
+ ((AutoConsumeSchema) schema).fetchSchemaIfNeeded();
+ }
+ return schema.getSchemaInfo();
+ }
+
@Override
public T getValue() {
checkNotNull(msgMetadataBuilder);
- if (schema.getSchemaInfo() != null && SchemaType.KEY_VALUE == schema.getSchemaInfo().getType()) {
+ SchemaInfo schemaInfo = getSchemaInfo();
+ if (schemaInfo != null && SchemaType.KEY_VALUE == schemaInfo.getType()) {
if (schema.supportSchemaVersioning()) {
return getKeyValueBySchemaVersion();
} else {
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java
index 41b1260..07c807c 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java
@@ -21,7 +21,6 @@ package org.apache.pulsar.client.impl.schema;
import lombok.extern.slf4j.Slf4j;
import org.apache.pulsar.client.api.Schema;
import org.apache.pulsar.client.api.SchemaSerializationException;
-import org.apache.pulsar.client.api.schema.GenericObject;
import org.apache.pulsar.client.api.schema.GenericRecord;
import org.apache.pulsar.client.api.schema.SchemaInfoProvider;
import org.apache.pulsar.client.impl.schema.generic.GenericProtobufNativeSchema;
@@ -30,7 +29,11 @@ import org.apache.pulsar.common.schema.KeyValue;
import org.apache.pulsar.common.schema.SchemaInfo;
import org.apache.pulsar.common.schema.SchemaType;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
import static com.google.common.base.Preconditions.checkState;
@@ -77,27 +80,7 @@ public class AutoConsumeSchema implements Schema<GenericRecord> {
@Override
public GenericRecord decode(byte[] bytes, byte[] schemaVersion) {
- if (schema == null) {
- SchemaInfo schemaInfo = null;
- try {
- schemaInfo = schemaInfoProvider.getLatestSchema().get();
- if (schemaInfo == null) {
- // schemaless topic
- schemaInfo = BytesSchema.of().getSchemaInfo();
- }
- } catch (InterruptedException | ExecutionException e ) {
- if (e instanceof InterruptedException) {
- Thread.currentThread().interrupt();
- }
- log.error("Can't get last schema for topic {} use AutoConsumeSchema", topicName);
- throw new SchemaSerializationException(e.getCause());
- }
- // schemaInfo null means that there is no schema attached to the topic.
- schema = generateSchema(schemaInfo);
- schema.setSchemaInfoProvider(schemaInfoProvider);
- log.info("Configure {} schema for topic {} : {}",
- componentName, topicName, schemaInfo.getSchemaDefinition());
- }
+ fetchSchemaIfNeeded();
ensureSchemaInitialized();
return adapt(schema.decode(bytes, schemaVersion), schemaVersion);
}
@@ -157,7 +140,7 @@ public class AutoConsumeSchema implements Schema<GenericRecord> {
}
}
- private Schema<?> generateSchema(SchemaInfo schemaInfo) {
+ private static Schema<?> generateSchema(SchemaInfo schemaInfo) {
// when using `AutoConsumeSchema`, we use the schema associated with the messages as schema reader
// to decode the messages.
final boolean useProvidedSchemaAsReaderSchema = false;
@@ -260,4 +243,47 @@ public class AutoConsumeSchema implements Schema<GenericRecord> {
public Schema<?> getInternalSchema() {
return schema;
}
+
+ /**
+ * It may happen that the schema is not loaded but we need it, for instance in order to call getSchemaInfo()
+ * We cannot call this method in getSchemaInfo, because getSchemaInfo is called in many
+ * places and we will introduce lots of deadlocks.
+ */
+ public void fetchSchemaIfNeeded() throws SchemaSerializationException {
+ if (schema == null) {
+ if (schemaInfoProvider == null) {
+ throw new SchemaSerializationException("Can't get accurate schema information for topic " + topicName +
+ "using AutoConsumeSchema because SchemaInfoProvider is not set yet");
+ } else {
+ SchemaInfo schemaInfo = null;
+ try {
+ schemaInfo = schemaInfoProvider.getLatestSchema().get();
+ if (schemaInfo == null) {
+ // schemaless topic
+ schemaInfo = BytesSchema.of().getSchemaInfo();
+ }
+ } catch (InterruptedException | ExecutionException e) {
+ if (e instanceof InterruptedException) {
+ Thread.currentThread().interrupt();
+ }
+ log.error("Can't get last schema for topic {} using AutoConsumeSchema", topicName);
+ throw new SchemaSerializationException(e.getCause());
+ }
+ // schemaInfo null means that there is no schema attached to the topic.
+ schema = generateSchema(schemaInfo);
+ schema.setSchemaInfoProvider(schemaInfoProvider);
+ log.info("Configure {} schema for topic {} : {}",
+ componentName, topicName, schemaInfo.getSchemaDefinition());
+ }
+ }
+ }
+
+ @Override
+ public String toString() {
+ if (schema != null && schema.getSchemaInfo() != null) {
+ return "AUTO_CONSUME(schematype=" + schema.getSchemaInfo().getType() + ")";
+ } else {
+ return "AUTO_CONSUME(uninitialized)";
+ }
+ }
}
diff --git a/tests/docker-images/java-test-functions/src/main/java/org/apache/pulsar/tests/integration/io/TestGenericObjectSink.java b/tests/docker-images/java-test-functions/src/main/java/org/apache/pulsar/tests/integration/io/TestGenericObjectSink.java
index c0c4ac2..b7645ba 100644
--- a/tests/docker-images/java-test-functions/src/main/java/org/apache/pulsar/tests/integration/io/TestGenericObjectSink.java
+++ b/tests/docker-images/java-test-functions/src/main/java/org/apache/pulsar/tests/integration/io/TestGenericObjectSink.java
@@ -36,7 +36,7 @@ public class TestGenericObjectSink implements Sink<GenericObject> {
}
public void write(Record<GenericObject> record) {
-
+ log.info("topic {}", record.getTopicName().orElse(null));
log.info("properties {}", record.getProperties());
log.info("received record {} {}", record, record.getClass());
log.info("schema {}", record.getSchema());
@@ -65,6 +65,8 @@ public class TestGenericObjectSink implements Sink<GenericObject> {
log.info("value {}", record.getValue());
log.info("value schema type {}", record.getValue().getSchemaType());
log.info("value native object {}", record.getValue().getNativeObject());
+
+ record.ack();
}
@Override
diff --git a/tests/integration/src/test/java/org/apache/pulsar/tests/integration/io/PulsarGenericObjectSinkTest.java b/tests/integration/src/test/java/org/apache/pulsar/tests/integration/io/PulsarGenericObjectSinkTest.java
index 1e5fb74..c810b1b 100644
--- a/tests/integration/src/test/java/org/apache/pulsar/tests/integration/io/PulsarGenericObjectSinkTest.java
+++ b/tests/integration/src/test/java/org/apache/pulsar/tests/integration/io/PulsarGenericObjectSinkTest.java
@@ -21,6 +21,7 @@ package org.apache.pulsar.tests.integration.io;
import lombok.Builder;
import lombok.Cleanup;
import lombok.Data;
+import lombok.Getter;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.io.IOUtils;
import org.apache.commons.lang3.StringUtils;
@@ -31,6 +32,7 @@ import org.apache.pulsar.client.api.PulsarClient;
import org.apache.pulsar.client.api.Schema;
import org.apache.pulsar.common.policies.data.SinkStatus;
import org.apache.pulsar.common.schema.KeyValue;
+import org.apache.pulsar.common.schema.KeyValueEncodingType;
import org.apache.pulsar.tests.integration.docker.ContainerExecException;
import org.apache.pulsar.tests.integration.docker.ContainerExecResult;
import org.apache.pulsar.tests.integration.suites.PulsarStandaloneTestSuite;
@@ -43,6 +45,8 @@ import java.nio.charset.StandardCharsets;
import java.util.Arrays;
import java.util.List;
import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
import static org.apache.pulsar.tests.integration.functions.utils.CommandGenerator.JAVAJAR;
import static org.testng.Assert.assertEquals;
@@ -55,15 +59,14 @@ import static org.testng.Assert.fail;
@Slf4j
public class PulsarGenericObjectSinkTest extends PulsarStandaloneTestSuite {
+ @Getter
private static final class SinkSpec<T> {
final String outputTopicName;
- final String sinkName;
final Schema<T> schema;
final T testValue;
- public SinkSpec(String outputTopicName, String sinkName, Schema<T> schema, T testValue) {
+ public SinkSpec(String outputTopicName, Schema<T> schema, T testValue) {
this.outputTopicName = outputTopicName;
- this.sinkName = sinkName;
this.schema = schema;
this.testValue = testValue;
}
@@ -89,90 +92,82 @@ public class PulsarGenericObjectSinkTest extends PulsarStandaloneTestSuite {
.serviceUrl(container.getPlainTextServiceUrl())
.build();
+ @Cleanup
+ PulsarAdmin admin = PulsarAdmin.builder().serviceHttpUrl(container.getHttpServiceUrl()).build();
+
// we are not using a parametrized test in order to save resources
- // we create N sinks, send the records and verify each sink
- // sinks execution happens in parallel
+ // we create one sink that listens on multiple topics, send the records and verify the sink
List<SinkSpec> specs = Arrays.asList(
- new SinkSpec("test-kv-sink-input-string-" + randomName(8), "test-kv-sink-string-" + randomName(8), Schema.STRING, "foo"),
- new SinkSpec("test-kv-sink-input-avro-" + randomName(8), "test-kv-sink-avro-" + randomName(8), Schema.AVRO(Pojo.class), Pojo.builder().field1("a").field2(2).build()),
- new SinkSpec("test-kv-sink-input-kv-string-int-" + randomName(8), "test-kv-sink-input-kv-string-int-" + randomName(8),
+ new SinkSpec("test-kv-sink-input-string-" + randomName(8), Schema.STRING, "foo"),
+ new SinkSpec("test-kv-sink-input-avro-" + randomName(8), Schema.AVRO(Pojo.class), Pojo.builder().field1("a").field2(2).build()),
+ new SinkSpec("test-kv-sink-input-kv-string-int-" + randomName(8),
Schema.KeyValue(Schema.STRING, Schema.INT32), new KeyValue<>("foo", 123)),
- new SinkSpec("test-kv-sink-input-kv-avro-json-" + randomName(8), "test-kv-sink-input-kv-string-int-" + randomName(8),
- Schema.KeyValue(Schema.AVRO(PojoKey.class), Schema.JSON(Pojo.class)), new KeyValue<>(PojoKey.builder().field1("a").build(), Pojo.builder().field1("a").field2(2).build()))
+ new SinkSpec("test-kv-sink-input-kv-avro-json-inl-" + randomName(8),
+ Schema.KeyValue(Schema.AVRO(PojoKey.class), Schema.JSON(Pojo.class), KeyValueEncodingType.INLINE), new KeyValue<>(PojoKey.builder().field1("a").build(), Pojo.builder().field1("a").field2(2).build())),
+ new SinkSpec("test-kv-sink-input-kv-avro-json-sep-" + randomName(8),
+ Schema.KeyValue(Schema.AVRO(PojoKey.class), Schema.JSON(Pojo.class), KeyValueEncodingType.SEPARATED), new KeyValue<>(PojoKey.builder().field1("a").build(), Pojo.builder().field1("a").field2(2).build()))
);
- // submit all sinks
- for (SinkSpec spec : specs) {
- submitSinkConnector(spec.sinkName, spec.outputTopicName, "org.apache.pulsar.tests.integration.io.TestGenericObjectSink", JAVAJAR);
- }
- // check all sinks
- for (SinkSpec spec : specs) {
- // get sink info
- getSinkInfoSuccess(spec.sinkName);
- getSinkStatus(spec.sinkName);
- }
- final int numRecords = 10;
+ final int numRecordsPerTopic = 2;
+
+ String sinkName = "genericobject-sink";
+ String topicNames = specs
+ .stream()
+ .map(SinkSpec::getOutputTopicName)
+ .collect(Collectors.joining(","));
+ submitSinkConnector(sinkName, topicNames, "org.apache.pulsar.tests.integration.io.TestGenericObjectSink", JAVAJAR);
+ // get sink info
+ getSinkInfoSuccess(sinkName);
+ getSinkStatus(sinkName);
+
for (SinkSpec spec : specs) {
+
@Cleanup Producer<Object> producer = client.newProducer(spec.schema)
.topic(spec.outputTopicName)
.create();
- for (int i = 0; i < numRecords; i++) {
+ for (int i = 0; i < numRecordsPerTopic; i++) {
MessageId messageId = producer.newMessage()
.value(spec.testValue)
.property("expectedType", spec.schema.getSchemaInfo().getType().toString())
+ .property("recordNumber", i + "")
.send();
log.info("sent message {} {} with ID {}", spec.testValue, spec.schema.getSchemaInfo().getType().toString(), messageId);
}
}
- // wait that all sinks processed all records without errors
- try (PulsarAdmin admin = PulsarAdmin.builder().serviceHttpUrl(container.getHttpServiceUrl()).build()) {
- for (SinkSpec spec : specs) {
- try {
- log.info("waiting for sink {}", spec.sinkName);
- for (int i = 0; i < 120; i++) {
- SinkStatus status = admin.sinks().getSinkStatus("public", "default", spec.sinkName);
- log.info("sink {} status {}", spec.sinkName, status);
- assertEquals(status.getInstances().size(), 1);
- SinkStatus.SinkInstanceStatus instance = status.getInstances().get(0);
- if (instance.getStatus().numWrittenToSink >= numRecords) {
- break;
- }
- assertTrue(instance.getStatus().numRestarts > 1, "Sink was restarted, probably an error occurred");
- Thread.sleep(1000);
- }
-
- SinkStatus status = admin.sinks().getSinkStatus("public", "default", spec.sinkName);
- log.info("sink {} status {}", spec.sinkName, status);
- assertEquals(status.getInstances().size(), 1);
- assertTrue(status.getInstances().get(0).getStatus().numWrittenToSink >= numRecords);
- assertTrue(status.getInstances().get(0).getStatus().numSinkExceptions == 0);
- assertTrue(status.getInstances().get(0).getStatus().numSystemExceptions == 0);
- log.info("sink {} is okay", spec.sinkName);
- } finally {
- dumpSinkLogs(spec);
+ // wait that sink processed all records without errors
+
+ try {
+ log.info("waiting for sink {}", sinkName);
+
+ for (int i = 0; i < 120; i++) {
+ SinkStatus status = admin.sinks().getSinkStatus("public", "default", sinkName);
+ log.info("sink {} status {}", sinkName, status);
+ assertEquals(status.getInstances().size(), 1);
+ SinkStatus.SinkInstanceStatus instance = status.getInstances().get(0);
+ if (instance.getStatus().numWrittenToSink >= numRecordsPerTopic * specs.size()
+ || instance.getStatus().numSinkExceptions > 0
+ || instance.getStatus().numSystemExceptions > 0
+ || instance.getStatus().numRestarts > 0) {
+ break;
}
+ Thread.sleep(1000);
}
- }
-
- for (SinkSpec spec : specs) {
- deleteSink(spec.sinkName);
- getSinkInfoNotFound(spec.sinkName);
+ SinkStatus status = admin.sinks().getSinkStatus("public", "default", sinkName);
+ log.info("sink {} status {}", sinkName, status);
+ assertEquals(status.getInstances().size(), 1);
+ assertTrue(status.getInstances().get(0).getStatus().numWrittenToSink >= numRecordsPerTopic * specs.size());
+ assertTrue(status.getInstances().get(0).getStatus().numSinkExceptions == 0);
+ assertTrue(status.getInstances().get(0).getStatus().numSystemExceptions == 0);
+ log.info("sink {} is okay", sinkName);
+ } finally {
+ dumpFunctionLogs(sinkName);
}
- }
- private void dumpSinkLogs(SinkSpec spec) {
- try {
- String logFile = "/pulsar/logs/functions/public/default/" + spec.sinkName + "/" + spec.sinkName + "-0.log";
- String logs = container.<String>copyFileFromContainer(logFile, (inputStream) -> {
- return IOUtils.toString(inputStream, "utf-8");
- });
- log.info("Sink {} logs {}", spec.sinkName, logs);
- } catch (Throwable err) {
- log.info("Cannot download sink {} logs", spec.sinkName, err);
- }
+ deleteSink(sinkName);
+ getSinkInfoNotFound(sinkName);
}
private void submitSinkConnector(String sinkName,
diff --git a/tests/integration/src/test/java/org/apache/pulsar/tests/integration/suites/PulsarStandaloneTestSuite.java b/tests/integration/src/test/java/org/apache/pulsar/tests/integration/suites/PulsarStandaloneTestSuite.java
index 4e2e601..93e2e07 100644
--- a/tests/integration/src/test/java/org/apache/pulsar/tests/integration/suites/PulsarStandaloneTestSuite.java
+++ b/tests/integration/src/test/java/org/apache/pulsar/tests/integration/suites/PulsarStandaloneTestSuite.java
@@ -40,4 +40,5 @@ public class PulsarStandaloneTestSuite extends PulsarStandaloneTestBase implemen
public String getTestName() {
return "pulsar-standalone-suite";
}
+
}
diff --git a/tests/integration/src/test/java/org/apache/pulsar/tests/integration/topologies/PulsarStandaloneTestBase.java b/tests/integration/src/test/java/org/apache/pulsar/tests/integration/topologies/PulsarStandaloneTestBase.java
index 03132f5..459837b 100644
--- a/tests/integration/src/test/java/org/apache/pulsar/tests/integration/topologies/PulsarStandaloneTestBase.java
+++ b/tests/integration/src/test/java/org/apache/pulsar/tests/integration/topologies/PulsarStandaloneTestBase.java
@@ -21,6 +21,7 @@ package org.apache.pulsar.tests.integration.topologies;
import static org.testng.Assert.assertEquals;
import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.io.IOUtils;
import org.apache.pulsar.tests.integration.containers.StandaloneContainer;
import org.apache.pulsar.tests.integration.docker.ContainerExecResult;
import org.testcontainers.containers.Network;
@@ -90,4 +91,18 @@ public abstract class PulsarStandaloneTestBase extends PulsarTestBase {
network.close();
}
+
+
+ protected void dumpFunctionLogs(String name) {
+ try {
+ String logFile = "/pulsar/logs/functions/public/default/" + name + "/" + name + "-0.log";
+ String logs = container.<String>copyFileFromContainer(logFile, (inputStream) -> {
+ return IOUtils.toString(inputStream, "utf-8");
+ });
+ log.info("Function {} logs {}", name, logs);
+ } catch (Throwable err) {
+ log.info("Cannot download {} logs", name, err);
+ }
+ }
+
}
[pulsar] 42/46: [Broker] Make
Persistent*DispatcherMultipleConsumers.readMoreEntries synchronized
(#10413)
Posted by eo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 83d10a140f220626eaa5360a62d4b78bacb63e1c
Author: Lari Hotari <lh...@users.noreply.github.com>
AuthorDate: Wed Apr 28 19:57:54 2021 +0300
[Broker] Make Persistent*DispatcherMultipleConsumers.readMoreEntries synchronized (#10413)
(cherry picked from commit 55867f1ea3e3288b8f43c8c5b35418a545b16b29)
---
.../persistent/PersistentStreamingDispatcherMultipleConsumers.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStreamingDispatcherMultipleConsumers.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStreamingDispatcherMultipleConsumers.java
index 9340e17..a080e62 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStreamingDispatcherMultipleConsumers.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStreamingDispatcherMultipleConsumers.java
@@ -135,7 +135,7 @@ public class PersistentStreamingDispatcherMultipleConsumers extends PersistentDi
}
@Override
- public void readMoreEntries() {
+ public synchronized void readMoreEntries() {
// totalAvailablePermits may be updated by other threads
int currentTotalAvailablePermits = totalAvailablePermits;
if (currentTotalAvailablePermits > 0 && isAtleastOneConsumerAvailable()) {
[pulsar] 38/46: AutoConsumeSchema: handle schema NONE as BYTES
(#10277)
Posted by eo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit c73cdae681ff1326ae7717bb6d27c01106ac188f
Author: Enrico Olivelli <eo...@gmail.com>
AuthorDate: Tue Apr 20 12:35:37 2021 +0200
AutoConsumeSchema: handle schema NONE as BYTES (#10277)
---
.../java/org/apache/pulsar/schema/SchemaTest.java | 23 ++++++++++++++++++++--
.../client/impl/schema/AutoConsumeSchema.java | 1 +
2 files changed, 22 insertions(+), 2 deletions(-)
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/schema/SchemaTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/schema/SchemaTest.java
index 0897e67..f167e0d 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/schema/SchemaTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/schema/SchemaTest.java
@@ -55,6 +55,7 @@ import org.apache.pulsar.common.policies.data.ClusterData;
import org.apache.pulsar.common.policies.data.TenantInfo;
import org.apache.pulsar.common.schema.KeyValue;
import org.apache.pulsar.common.schema.KeyValueEncodingType;
+import org.apache.pulsar.common.schema.SchemaInfo;
import org.apache.pulsar.common.schema.SchemaType;
import org.apache.pulsar.common.util.FutureUtil;
import org.testng.annotations.AfterMethod;
@@ -392,7 +393,16 @@ public class SchemaTest extends MockedPulsarServiceBaseTest {
}
@Test
- public void testUseAutoConsumeWithSchemalessTopic() throws Exception {
+ public void testUseAutoConsumeWithBytesSchemaTopic() throws Exception {
+ testUseAutoConsumeWithSchemalessTopic(SchemaType.BYTES);
+ }
+
+ @Test
+ public void testUseAutoConsumeWithNoneSchemaTopic() throws Exception {
+ testUseAutoConsumeWithSchemalessTopic(SchemaType.NONE);
+ }
+
+ private void testUseAutoConsumeWithSchemalessTopic(SchemaType schema) throws Exception {
final String tenant = PUBLIC_TENANT;
final String namespace = "test-namespace-" + randomName(16);
final String topicName = "test-schemaless";
@@ -409,6 +419,15 @@ public class SchemaTest extends MockedPulsarServiceBaseTest {
admin.topics().createPartitionedTopic(topic, 2);
+ // set schema
+ SchemaInfo schemaInfo = SchemaInfo
+ .builder()
+ .schema(new byte[0])
+ .name("dummySchema")
+ .type(schema)
+ .build();
+ admin.schemas().createSchema(topic, schemaInfo);
+
Producer<byte[]> producer = pulsarClient
.newProducer()
.topic(topic)
@@ -420,7 +439,7 @@ public class SchemaTest extends MockedPulsarServiceBaseTest {
.subscribe();
// use GenericRecord even for primitive types
- // it will be a PrimitiveRecord
+ // it will be a GenericObjectWrapper
Consumer<GenericRecord> consumer2 = pulsarClient.newConsumer(Schema.AUTO_CONSUME())
.subscriptionName("test-sub3")
.topic(topic)
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java
index 9f66875..af0ec7b 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java
@@ -184,6 +184,7 @@ public class AutoConsumeSchema implements Schema<GenericRecord> {
case BOOLEAN:
return BooleanSchema.of();
case BYTES:
+ case NONE:
return BytesSchema.of();
case DATE:
return DateSchema.of();
[pulsar] 32/46: [Java client] Fix behaviour of
Schema.AUTO_CONSUME() with KeyValueSchema and multi versions (#10492)
Posted by eo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 1264c47622366c4a9fe86e5999f511f1c8816473
Author: Enrico Olivelli <eo...@gmail.com>
AuthorDate: Fri May 7 22:56:37 2021 +0200
[Java client] Fix behaviour of Schema.AUTO_CONSUME() with KeyValueSchema and multi versions (#10492)
Co-authored-by: Enrico Olivelli <eo...@apache.org>
(cherry picked from commit 9b92dc26a6b7fad9e1406ecfb010069119fab7bc)
---
.../apache/pulsar/client/api/SimpleSchemaTest.java | 381 ++++++++++++++++-----
.../client/impl/schema/AutoConsumeSchema.java | 2 +-
2 files changed, 293 insertions(+), 90 deletions(-)
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleSchemaTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleSchemaTest.java
index f670ef1..ae44311 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleSchemaTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleSchemaTest.java
@@ -29,8 +29,11 @@ import static org.testng.Assert.assertNotNull;
import static org.testng.Assert.assertTrue;
import static org.testng.Assert.fail;
+import lombok.extern.slf4j.Slf4j;
import org.apache.avro.reflect.ReflectData;
import org.apache.avro.Schema.Parser;
+import org.apache.pulsar.client.impl.MessageImpl;
+import org.apache.pulsar.common.protocol.schema.BytesSchemaVersion;
import org.apache.pulsar.common.schema.LongSchemaVersion;
import org.apache.pulsar.client.admin.PulsarAdminException;
import org.apache.pulsar.client.api.PulsarClientException.IncompatibleSchemaException;
@@ -58,16 +61,19 @@ import java.io.ByteArrayInputStream;
import java.nio.ByteBuffer;
import java.util.Arrays;
import java.util.List;
+import java.util.UUID;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.TimeUnit;
+@Test(groups = "broker-api")
+@Slf4j
public class SimpleSchemaTest extends ProducerConsumerBase {
@DataProvider(name = "batchingModes")
public static Object[][] batchingModes() {
return new Object[][] {
- { true },
- { false }
+ { true },
+ { false }
};
}
@@ -110,7 +116,7 @@ public class SimpleSchemaTest extends ProducerConsumerBase {
.topic("persistent://my-property/my-ns/my-topic1")
.subscriptionName("my-subscriber-name").subscribe();
Producer<String> producer = pulsarClient.newProducer(Schema.STRING)
- .topic("persistent://my-property/my-ns/my-topic1").create()) {
+ .topic("persistent://my-property/my-ns/my-topic1").create()) {
int N = 10;
for (int i = 0; i < N; i++) {
@@ -202,14 +208,14 @@ public class SimpleSchemaTest extends ProducerConsumerBase {
p.send("junkdata".getBytes(UTF_8));
} else {
Assert.fail("Shouldn't be able to connect to a schema'd topic with no schema"
- + " if SchemaValidationEnabled is enabled");
+ + " if SchemaValidationEnabled is enabled");
}
} catch (PulsarClientException e) {
if (schemaValidationEnforced) {
Assert.assertTrue(e instanceof IncompatibleSchemaException);
} else {
Assert.fail("Shouldn't throw IncompatibleSchemaException"
- + " if SchemaValidationEnforced is disabled");
+ + " if SchemaValidationEnforced is disabled");
}
}
@@ -233,10 +239,10 @@ public class SimpleSchemaTest extends ProducerConsumerBase {
AvroWriter<V2Data> v2Writer = new AvroWriter<>(
new Parser().parse(new ByteArrayInputStream(v2SchemaBytes)));
try (Producer<V1Data> ignored = pulsarClient.newProducer(v1Schema)
- .topic(topic).create()) {
+ .topic(topic).create()) {
}
try (Producer<V2Data> p = pulsarClient.newProducer(Schema.AVRO(V2Data.class))
- .topic(topic).create()) {
+ .topic(topic).create()) {
p.send(new V2Data(-1, -1));
}
V1Data dataV1 = new V1Data(2);
@@ -244,14 +250,14 @@ public class SimpleSchemaTest extends ProducerConsumerBase {
byte[] contentV1 = v1Writer.write(dataV1);
byte[] contentV2 = v2Writer.write(dataV2);
try (Producer<byte[]> p = pulsarClient.newProducer(Schema.AUTO_PRODUCE_BYTES())
- .topic(topic).create();
- Consumer<V2Data> c = pulsarClient.newConsumer(v2Schema)
- .topic(topic)
- .subscriptionName("sub1").subscribe()) {
+ .topic(topic).create();
+ Consumer<V2Data> c = pulsarClient.newConsumer(v2Schema)
+ .topic(topic)
+ .subscriptionName("sub1").subscribe()) {
Assert.expectThrows(SchemaSerializationException.class, () -> p.send(contentV1));
p.newMessage(Schema.AUTO_PRODUCE_BYTES(Schema.AVRO(V1Data.class)))
- .value(contentV1).send();
+ .value(contentV1).send();
p.send(contentV2);
Message<V2Data> msg1 = c.receive();
V2Data msg1Value = msg1.getValue();
@@ -267,7 +273,7 @@ public class SimpleSchemaTest extends ProducerConsumerBase {
p.newMessage(Schema.BYTES).value(contentV1).send();
if (schemaValidationEnforced) {
Assert.fail("Shouldn't be able to send to a schema'd topic with no schema"
- + " if SchemaValidationEnabled is enabled");
+ + " if SchemaValidationEnabled is enabled");
}
Message<V2Data> msg3 = c.receive();
Assert.assertEquals(msg3.getSchemaVersion(), SchemaVersion.Empty.bytes());
@@ -276,7 +282,7 @@ public class SimpleSchemaTest extends ProducerConsumerBase {
Assert.assertTrue(e instanceof IncompatibleSchemaException);
} else {
Assert.fail("Shouldn't throw IncompatibleSchemaException"
- + " if SchemaValidationEnforced is disabled");
+ + " if SchemaValidationEnforced is disabled");
}
}
}
@@ -289,10 +295,10 @@ public class SimpleSchemaTest extends ProducerConsumerBase {
V2Data data2 = new V2Data(3, 5);
V1Data data3 = new V1Data(8);
try (Producer<V1Data> p = pulsarClient.newProducer(Schema.AVRO(V1Data.class))
- .topic(topic).create();
+ .topic(topic).create();
Consumer<V2Data> c = pulsarClient.newConsumer(Schema.AVRO(V2Data.class))
- .topic(topic)
- .subscriptionName("sub1").subscribe()) {
+ .topic(topic)
+ .subscriptionName("sub1").subscribe()) {
p.newMessage().value(data1).send();
p.newMessage(Schema.AVRO(V2Data.class)).value(data2).send();
p.newMessage(Schema.AVRO(V1Data.class)).value(data3).send();
@@ -326,10 +332,10 @@ public class SimpleSchemaTest extends ProducerConsumerBase {
AvroWriter<V2Data> v2Writer = new AvroWriter<>(
new Parser().parse(new ByteArrayInputStream(v2SchemaBytes)));
try (Producer<byte[]> p = pulsarClient.newProducer()
- .topic(topic).create();
+ .topic(topic).create();
Consumer<byte[]> c = pulsarClient.newConsumer()
- .topic(topic)
- .subscriptionName("sub1").subscribe()) {
+ .topic(topic)
+ .subscriptionName("sub1").subscribe()) {
for (int i = 0; i < 2; ++i) {
V1Data dataV1 = new V1Data(i);
V2Data dataV2 = new V2Data(i, -i);
@@ -348,19 +354,19 @@ public class SimpleSchemaTest extends ProducerConsumerBase {
List<SchemaInfo> allSchemas = admin.schemas().getAllSchemas(topic);
Assert.assertEquals(allSchemas, Arrays.asList(v1Schema.getSchemaInfo(),
- v2Schema.getSchemaInfo()));
+ v2Schema.getSchemaInfo()));
}
@Test
public void newProducerForMessageSchemaWithBatch() throws Exception {
String topic = "my-property/my-ns/schema-test";
Consumer<V2Data> c = pulsarClient.newConsumer(Schema.AVRO(V2Data.class))
- .topic(topic)
- .subscriptionName("sub1").subscribe();
+ .topic(topic)
+ .subscriptionName("sub1").subscribe();
Producer<byte[]> p = pulsarClient.newProducer(Schema.AUTO_PRODUCE_BYTES())
- .topic(topic)
- .enableBatching(true)
- .batchingMaxPublishDelay(10, TimeUnit.SECONDS).create();
+ .topic(topic)
+ .enableBatching(true)
+ .batchingMaxPublishDelay(10, TimeUnit.SECONDS).create();
AvroWriter<V1Data> v1DataAvroWriter = new AvroWriter<>(
ReflectData.AllowNull.get().getSchema(V1Data.class));
AvroWriter<V2Data> v2DataAvroWriter = new AvroWriter<>(
@@ -374,17 +380,17 @@ public class SimpleSchemaTest extends ProducerConsumerBase {
if (i / batch % 2 == 0) {
byte[] content = v1DataAvroWriter.write(new V1Data(i));
p.newMessage(Schema.AUTO_PRODUCE_BYTES(Schema.AVRO(V1Data.class)))
- .value(content).sendAsync();
+ .value(content).sendAsync();
} else {
byte[] content = v2DataAvroWriter.write(new V2Data(i, i + total));
p.newMessage(Schema.AUTO_PRODUCE_BYTES(Schema.AVRO(V2Data.class)))
- .value(content).sendAsync();
+ .value(content).sendAsync();
}
if ((i + 1) % incompatible == 0) {
byte[] content = incompatibleDataAvroWriter.write(new IncompatibleData(-i, -i));
try {
p.newMessage(Schema.AUTO_PRODUCE_BYTES(Schema.AVRO(IncompatibleData.class)))
- .value(content).send();
+ .value(content).send();
} catch (Exception e) {
Assert.assertTrue(e instanceof IncompatibleSchemaException, e.getMessage());
}
@@ -409,11 +415,11 @@ public class SimpleSchemaTest extends ProducerConsumerBase {
AvroWriter<V1Data> v1DataAvroWriter = new AvroWriter<>(
ReflectData.AllowNull.get().getSchema(V1Data.class));
try (Producer<byte[]> p = pulsarClient.newProducer()
- .topic(topic)
- .enableMultiSchema(false).create()) {
+ .topic(topic)
+ .enableMultiSchema(false).create()) {
Assert.assertThrows(InvalidMessageException.class,
() -> p.newMessage(Schema.AUTO_PRODUCE_BYTES(Schema.AVRO(V1Data.class)))
- .value(v1DataAvroWriter.write(new V1Data(0))).send());
+ .value(v1DataAvroWriter.write(new V1Data(0))).send());
}
}
@@ -449,7 +455,7 @@ public class SimpleSchemaTest extends ProducerConsumerBase {
try (Producer<V1Data> p = pulsarClient.newProducer(Schema.AVRO(V1Data.class)).topic(topic).create();
Consumer<V1Data> c = pulsarClient.newConsumer(Schema.AVRO(V1Data.class))
- .topic(topic).subscriptionName("sub1").subscribe()) {
+ .topic(topic).subscriptionName("sub1").subscribe()) {
V1Data toSend = new V1Data(1);
p.send(toSend);
Assert.assertEquals(toSend, c.receive().getValue());
@@ -481,14 +487,14 @@ public class SimpleSchemaTest extends ProducerConsumerBase {
String topic = "my-property/my-ns/schema-test";
try (Producer<V1Data> p = pulsarClient.newProducer(Schema.AVRO(V1Data.class))
- .topic(topic)
- .enableBatching(batching)
- .create();
+ .topic(topic)
+ .enableBatching(batching)
+ .create();
Consumer<V1Data> c = pulsarClient.newConsumer(Schema.AVRO(V1Data.class))
- .topic(topic)
- .subscriptionName("sub1")
- .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
- .subscribe()) {
+ .topic(topic)
+ .subscriptionName("sub1")
+ .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
+ .subscribe()) {
p.send(new V1Data(1));
Message<V1Data> data = c.receive();
@@ -502,14 +508,14 @@ public class SimpleSchemaTest extends ProducerConsumerBase {
String topic = "my-property/my-ns/schema-test-auto-consume-" + batching;
try (Producer<V1Data> p = pulsarClient.newProducer(Schema.AVRO(V1Data.class))
- .topic(topic)
- .enableBatching(batching)
- .create();
+ .topic(topic)
+ .enableBatching(batching)
+ .create();
Consumer<GenericRecord> c = pulsarClient.newConsumer(Schema.AUTO_CONSUME())
- .topic(topic)
- .subscriptionName("sub1")
- .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
- .subscribe()) {
+ .topic(topic)
+ .subscriptionName("sub1")
+ .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
+ .subscribe()) {
int numMessages = 10;
@@ -531,50 +537,55 @@ public class SimpleSchemaTest extends ProducerConsumerBase {
String topic = "my-property/my-ns/schema-test-auto-keyvalue-consume-" + batching;
Schema<KeyValue<V1Data, V1Data>> pojoSchema = Schema.KeyValue(
- Schema.AVRO(V1Data.class),
- Schema.AVRO(V1Data.class),
- KeyValueEncodingType.SEPARATED);
+ Schema.AVRO(V1Data.class),
+ Schema.AVRO(V1Data.class),
+ KeyValueEncodingType.SEPARATED);
try (Producer<KeyValue<V1Data, V1Data>> p = pulsarClient.newProducer(pojoSchema)
- .topic(topic)
- .enableBatching(batching)
- .create();
+ .topic(topic)
+ .enableBatching(batching)
+ .create();
+ Consumer<GenericRecord> c0 = pulsarClient.newConsumer(Schema.AUTO_CONSUME())
+ .topic(topic)
+ .subscriptionName("sub0")
+ .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
+ .subscribe();
Consumer<KeyValue<GenericRecord, GenericRecord>> c1 = pulsarClient.newConsumer(
- Schema.KeyValue(
- Schema.AUTO_CONSUME(),
- Schema.AUTO_CONSUME(),
- KeyValueEncodingType.SEPARATED))
- .topic(topic)
- .subscriptionName("sub1")
- .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
- .subscribe();
+ Schema.KeyValue(
+ Schema.AUTO_CONSUME(),
+ Schema.AUTO_CONSUME(),
+ KeyValueEncodingType.SEPARATED))
+ .topic(topic)
+ .subscriptionName("sub1")
+ .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
+ .subscribe();
Consumer<KeyValue<V1Data, V1Data>> c2 = pulsarClient.newConsumer(
- Schema.KeyValue(
- Schema.AVRO(V1Data.class),
- Schema.AVRO(V1Data.class),
- KeyValueEncodingType.SEPARATED))
- .topic(topic)
- .subscriptionName("sub2")
- .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
- .subscribe();
+ Schema.KeyValue(
+ Schema.AVRO(V1Data.class),
+ Schema.AVRO(V1Data.class),
+ KeyValueEncodingType.SEPARATED))
+ .topic(topic)
+ .subscriptionName("sub2")
+ .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
+ .subscribe();
Consumer<KeyValue<GenericRecord, V1Data>> c3 = pulsarClient.newConsumer(
- Schema.KeyValue(
- Schema.AUTO_CONSUME(),
- Schema.AVRO(V1Data.class),
- KeyValueEncodingType.SEPARATED))
- .topic(topic)
- .subscriptionName("sub3")
- .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
- .subscribe();
+ Schema.KeyValue(
+ Schema.AUTO_CONSUME(),
+ Schema.AVRO(V1Data.class),
+ KeyValueEncodingType.SEPARATED))
+ .topic(topic)
+ .subscriptionName("sub3")
+ .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
+ .subscribe();
Consumer<KeyValue<V1Data, GenericRecord>> c4 = pulsarClient.newConsumer(
- Schema.KeyValue(
- Schema.AVRO(V1Data.class),
- Schema.AUTO_CONSUME(),
- KeyValueEncodingType.SEPARATED))
- .topic(topic)
- .subscriptionName("sub4")
- .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
- .subscribe()
+ Schema.KeyValue(
+ Schema.AVRO(V1Data.class),
+ Schema.AUTO_CONSUME(),
+ KeyValueEncodingType.SEPARATED))
+ .topic(topic)
+ .subscriptionName("sub4")
+ .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
+ .subscribe()
) {
int numMessages = 10;
@@ -584,12 +595,22 @@ public class SimpleSchemaTest extends ProducerConsumerBase {
}
p.flush();
+ // verify c0
+ for (int i = 0; i < numMessages; i++) {
+ Message<GenericRecord> wrapper = c0.receive();
+ KeyValue<GenericRecord, GenericRecord> data = (KeyValue<GenericRecord, GenericRecord>) wrapper.getValue().getNativeObject();
+ assertNotNull(wrapper.getSchemaVersion());
+ assertEquals(data.getKey().getField("i"), i * 100);
+ assertEquals(data.getValue().getField("i"), i * 1000);
+ c0.acknowledge(wrapper);
+ }
// verify c1
for (int i = 0; i < numMessages; i++) {
Message<KeyValue<GenericRecord, GenericRecord>> data = c1.receive();
assertNotNull(data.getSchemaVersion());
assertEquals(data.getValue().getKey().getField("i"), i * 100);
assertEquals(data.getValue().getValue().getField("i"), i * 1000);
+ c1.acknowledge(data);
}
// verify c2
@@ -598,6 +619,7 @@ public class SimpleSchemaTest extends ProducerConsumerBase {
assertNotNull(data.getSchemaVersion());
assertEquals(data.getValue().getKey().i, i * 100);
assertEquals(data.getValue().getValue().i, i * 1000);
+ c2.acknowledge(data);
}
// verify c3
@@ -606,6 +628,7 @@ public class SimpleSchemaTest extends ProducerConsumerBase {
assertNotNull(data.getSchemaVersion());
assertEquals(data.getValue().getKey().getField("i"), i * 100);
assertEquals(data.getValue().getValue().i, i * 1000);
+ c3.acknowledge(data);
}
// verify c4
@@ -614,6 +637,186 @@ public class SimpleSchemaTest extends ProducerConsumerBase {
assertNotNull(data.getSchemaVersion());
assertEquals(data.getValue().getKey().i, i * 100);
assertEquals(data.getValue().getValue().getField("i"), i * 1000);
+ c4.acknowledge(data);
+ }
+ }
+
+ // new schema version
+
+ Schema<KeyValue<V2Data, V2Data>> pojoSchemaV2 = Schema.KeyValue(
+ Schema.AVRO(V2Data.class),
+ Schema.AVRO(V2Data.class),
+ KeyValueEncodingType.SEPARATED);
+
+ try (Producer<KeyValue<V2Data, V2Data>> p = pulsarClient.newProducer(pojoSchemaV2)
+ .topic(topic)
+ .enableBatching(batching)
+ .create();
+ Consumer<GenericRecord> c0 = pulsarClient.newConsumer(Schema.AUTO_CONSUME())
+ .topic(topic)
+ .subscriptionName("sub0")
+ .subscribe();
+ Consumer<KeyValue<GenericRecord, GenericRecord>> c1 = pulsarClient.newConsumer(
+ Schema.KeyValue(
+ Schema.AUTO_CONSUME(),
+ Schema.AUTO_CONSUME(),
+ KeyValueEncodingType.SEPARATED))
+ .topic(topic)
+ .subscriptionName("sub1")
+ .subscribe();
+ Consumer<KeyValue<V2Data, V2Data>> c2 = pulsarClient.newConsumer(
+ Schema.KeyValue(
+ Schema.AVRO(V2Data.class),
+ Schema.AVRO(V2Data.class),
+ KeyValueEncodingType.SEPARATED))
+ .topic(topic)
+ .subscriptionName("sub2")
+ .subscribe();
+ Consumer<KeyValue<GenericRecord, V2Data>> c3 = pulsarClient.newConsumer(
+ Schema.KeyValue(
+ Schema.AUTO_CONSUME(),
+ Schema.AVRO(V2Data.class),
+ KeyValueEncodingType.SEPARATED))
+ .topic(topic)
+ .subscriptionName("sub3")
+ .subscribe();
+ Consumer<KeyValue<V2Data, GenericRecord>> c4 = pulsarClient.newConsumer(
+ Schema.KeyValue(
+ Schema.AVRO(V2Data.class),
+ Schema.AUTO_CONSUME(),
+ KeyValueEncodingType.SEPARATED))
+ .topic(topic)
+ .subscriptionName("sub4")
+ .subscribe()
+ ) {
+
+ int numMessages = 10;
+
+ for (int i = 0; i < numMessages; i++) {
+ p.sendAsync(new KeyValue<>(new V2Data(i * 100, i), new V2Data(i * 1000, i * 20)));
+ }
+ p.flush();
+
+ // verify c0
+ for (int i = 0; i < numMessages; i++) {
+ Message<GenericRecord> wrapper = c0.receive();
+ KeyValue<GenericRecord, GenericRecord> data = (KeyValue<GenericRecord, GenericRecord>) wrapper.getValue().getNativeObject();
+ assertNotNull(wrapper.getSchemaVersion());
+ assertEquals(data.getKey().getField("i"), i * 100);
+ assertEquals(data.getValue().getField("i"), i * 1000);
+ assertEquals(data.getKey().getField("j"), i);
+ assertEquals(data.getValue().getField("j"), i * 20);
+ }
+ // verify c1
+ for (int i = 0; i < numMessages; i++) {
+ Message<KeyValue<GenericRecord, GenericRecord>> data = c1.receive();
+ assertNotNull(data.getSchemaVersion());
+ assertEquals(data.getValue().getKey().getField("i"), i * 100);
+ assertEquals(data.getValue().getValue().getField("i"), i * 1000);
+ assertEquals(data.getValue().getKey().getField("j"), i);
+ assertEquals(data.getValue().getValue().getField("j"), i * 20);
+ }
+
+ // verify c2
+ for (int i = 0; i < numMessages; i++) {
+ Message<KeyValue<V2Data, V2Data>> data = c2.receive();
+ assertNotNull(data.getSchemaVersion());
+ assertEquals(data.getValue().getKey().i, i * 100);
+ assertEquals(data.getValue().getValue().i, i * 1000);
+ assertEquals(data.getValue().getKey().j, (Integer) i);
+ assertEquals(data.getValue().getValue().j, (Integer) (i * 20));
+ }
+
+ // verify c3
+ for (int i = 0; i < numMessages; i++) {
+ Message<KeyValue<GenericRecord, V2Data>> data = c3.receive();
+ assertNotNull(data.getSchemaVersion());
+ assertEquals(data.getValue().getKey().getField("i"), i * 100);
+ assertEquals(data.getValue().getValue().i, i * 1000);
+ assertEquals(data.getValue().getKey().getField("j"), (Integer) i);
+ assertEquals(data.getValue().getValue().j, (Integer) (i * 20));
+ }
+
+ // verify c4
+ for (int i = 0; i < numMessages; i++) {
+ Message<KeyValue<V2Data, GenericRecord>> data = c4.receive();
+ assertNotNull(data.getSchemaVersion());
+ assertEquals(data.getValue().getKey().i, i * 100);
+ assertEquals(data.getValue().getValue().getField("i"), i * 1000);
+ assertEquals(data.getValue().getKey().j, (Integer) i);
+ assertEquals(data.getValue().getValue().getField("j"), i * 20);
+ }
+ }
+
+ }
+
+ @Test
+ public void testAutoKeyValueConsumeGenericObject() throws Exception {
+ String topic = "my-property/my-ns/schema-test-auto-keyvalue-consume-"+ UUID.randomUUID();
+
+ Schema<KeyValue<V1Data, V1Data>> pojoSchema = Schema.KeyValue(
+ Schema.AVRO(V1Data.class),
+ Schema.AVRO(V1Data.class),
+ KeyValueEncodingType.SEPARATED);
+
+ try (Producer<KeyValue<V1Data, V1Data>> p = pulsarClient.newProducer(pojoSchema)
+ .topic(topic)
+ .create();
+ Consumer<GenericRecord> c0 = pulsarClient.newConsumer(Schema.AUTO_CONSUME())
+ .topic(topic)
+ .subscriptionName("sub0")
+ .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
+ .subscribe();
+ ) {
+
+ int numMessages = 10;
+
+ for (int i = 0; i < numMessages; i++) {
+ p.sendAsync(new KeyValue<>(new V1Data(i * 100), new V1Data(i * 1000)));
+ }
+ p.flush();
+
+ // verify c0
+ for (int i = 0; i < numMessages; i++) {
+ Message<GenericRecord> wrapper = c0.receive();
+ log.info("schema version {}", BytesSchemaVersion.of(wrapper.getSchemaVersion()));
+ KeyValue<GenericRecord, GenericRecord> data = (KeyValue<GenericRecord, GenericRecord>) wrapper.getValue().getNativeObject();
+ assertNotNull(wrapper.getSchemaVersion());
+ assertEquals(data.getKey().getField("i"), i * 100);
+ assertEquals(data.getValue().getField("i"), i * 1000);
+ c0.acknowledge(wrapper);
+ }
+
+
+ // new schema version
+
+ Schema<KeyValue<V2Data, V2Data>> pojoSchemaV2 = Schema.KeyValue(
+ Schema.AVRO(V2Data.class),
+ Schema.AVRO(V2Data.class),
+ KeyValueEncodingType.SEPARATED);
+
+ try (Producer<KeyValue<V2Data, V2Data>> p2 = pulsarClient.newProducer(pojoSchemaV2)
+ .topic(topic)
+ .create();
+ ) {
+
+ for (int i = 0; i < numMessages; i++) {
+ p2.sendAsync(new KeyValue<>(new V2Data(i * 100, i), new V2Data(i * 1000, i * 20)));
+ }
+ p2.flush();
+
+ // verify c0
+ for (int i = 0; i < numMessages; i++) {
+ Message<GenericRecord> wrapper = c0.receive();
+ log.info("schema version {}", BytesSchemaVersion.of(wrapper.getSchemaVersion()));
+ KeyValue<GenericRecord, GenericRecord> data = (KeyValue<GenericRecord, GenericRecord>) wrapper.getValue().getNativeObject();
+ assertNotNull(wrapper.getSchemaVersion());
+ assertEquals(data.getKey().getField("i"), i * 100);
+ assertEquals(data.getValue().getField("i"), i * 1000);
+ assertEquals(data.getKey().getField("j"), i);
+ assertEquals(data.getValue().getField("j"), i * 20);
+ }
+
}
}
}
@@ -626,12 +829,12 @@ public class SimpleSchemaTest extends ProducerConsumerBase {
PulsarClientImpl binaryProtocolClient = (PulsarClientImpl) pulsarClient;
pulsarClient.newProducer(Schema.AVRO(V1Data.class))
- .topic(topic)
- .create();
+ .topic(topic)
+ .create();
pulsarClient.newProducer(Schema.AVRO(V2Data.class))
- .topic(topic)
- .create();
+ .topic(topic)
+ .create();
LookupService httpLookupService = httpProtocolClient.getLookup();
LookupService binaryLookupService = binaryProtocolClient.getLookup();
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java
index 07c807c..1e43fc4 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java
@@ -191,7 +191,7 @@ public class AutoConsumeSchema implements Schema<GenericRecord> {
return LocalDateTimeSchema.of();
case JSON:
case AVRO:
- return GenericSchemaImpl.of(schemaInfo);
+ return GenericSchemaImpl.of(schemaInfo, false);
case PROTOBUF_NATIVE:
return GenericProtobufNativeSchema.of(schemaInfo);
case KEY_VALUE:
[pulsar] 40/46: Add streaming dispatcher. (#9056)
Posted by eo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 7e57cbe5de2e52b37706170e3c4ec1066cd9585b
Author: Marvin Cai <zx...@streamnative.io>
AuthorDate: Mon Feb 1 20:02:50 2021 -0800
Add streaming dispatcher. (#9056)
Related to #3804
Trying to streamline the dispatcher's read requests to manager ledger instead of micro batch.
Created a StreamingEntryReader that can streamline read request to managed ledger.
Created StreamingDispatcher interface with necessary method to interact with StreamingEntryReader.
Created PersistentStreamingDispatcherSingleActive/MultipleConsumer that make use of StreamingEntryReader to read entries from managed ledger.
Add config to use streaming dispatcher.
(cherry picked from commit 8cfaf48e1dc97e30050cb29c106b614d6b9bc69c)
---
build/run_unit_group.sh | 195 ++++++++++
.../bookkeeper/mledger/WaitingEntryCallBack.java | 30 ++
.../bookkeeper/mledger/impl/ManagedLedgerImpl.java | 26 +-
.../apache/bookkeeper/mledger/impl/OpAddEntry.java | 2 +
.../apache/pulsar/broker/ServiceConfiguration.java | 7 +
.../AbstractDispatcherMultipleConsumers.java | 2 +
.../PersistentDispatcherMultipleConsumers.java | 191 +++++----
.../PersistentDispatcherSingleActiveConsumer.java | 279 ++++++-------
...istentStreamingDispatcherMultipleConsumers.java | 191 +++++++++
...entStreamingDispatcherSingleActiveConsumer.java | 208 ++++++++++
.../service/persistent/PersistentSubscription.java | 18 +-
.../streamingdispatch/PendingReadEntryRequest.java | 76 ++++
.../streamingdispatch/StreamingDispatcher.java | 53 +++
.../streamingdispatch/StreamingEntryReader.java | 338 ++++++++++++++++
.../service/streamingdispatch/package-info.java | 19 +
.../PersistentDispatcherFailoverConsumerTest.java | 5 +-
.../broker/service/PersistentFailoverE2ETest.java | 5 +
.../pulsar/broker/service/PersistentTopicTest.java | 4 +-
...herFailoverConsumerStreamingDispatcherTest.java | 35 ++
...rsistentFailoverStreamingDispatcherE2ETest.java | 36 ++
...istentStreamingDispatcherBlockConsumerTest.java | 35 ++
...eDispatchStreamingDispatcherThrottlingTest.java | 36 ++
.../PersistentTopicStreamingDispatcherE2ETest.java | 35 ++
.../PersistentTopicStreamingDispatcherTest.java | 35 ++
...roducerConsumerTestStreamingDispatcherTest.java | 35 ++
.../StreamingEntryReaderTests.java | 433 +++++++++++++++++++++
.../client/api/DispatcherBlockConsumerTest.java | 1 +
.../SubscriptionMessageDispatchThrottlingTest.java | 9 +-
site2/docs/reference-configuration.md | 1 +
29 files changed, 2107 insertions(+), 233 deletions(-)
diff --git a/build/run_unit_group.sh b/build/run_unit_group.sh
new file mode 100755
index 0000000..b4e5530
--- /dev/null
+++ b/build/run_unit_group.sh
@@ -0,0 +1,195 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+set -e
+set -x
+set -o pipefail
+set -o errexit
+
+MVN_TEST_COMMAND='build/retry.sh mvn -B -ntp test'
+
+# Test Groups -- start --
+function broker_group_1() {
+ $MVN_TEST_COMMAND -pl pulsar-broker -Dinclude="**/AdminApiOffloadTest.java" \
+ -DtestForkCount=1 \
+ -DtestReuseFork=true
+
+ $MVN_TEST_COMMAND -pl pulsar-broker -Dinclude="org/apache/pulsar/broker/**/*.java" \
+ -Dexclude="org/apache/pulsar/broker/zookeeper/**/*.java,
+ org/apache/pulsar/broker/loadbalance/**/*.java,
+ org/apache/pulsar/broker/service/**/*.java,
+ **/AdminApiOffloadTest.java"
+
+}
+
+function broker_group_2() {
+ $MVN_TEST_COMMAND -pl pulsar-broker -Dinclude="**/MessagePublishBufferThrottleTest.java" \
+ -DtestForkCount=1 \
+ -DtestReuseFork=true
+
+ $MVN_TEST_COMMAND -pl pulsar-broker -Dinclude="**/ReplicatorTest.java" \
+ -DtestForkCount=1 \
+ -DtestReuseFork=true
+
+ $MVN_TEST_COMMAND -pl pulsar-broker -Dinclude="**/TopicOwnerTest.java" \
+ -DtestForkCount=1 \
+ -DtestReuseFork=true
+
+ $MVN_TEST_COMMAND -pl pulsar-broker -Dinclude="**/AntiAffinityNamespaceGroupTest.java" \
+ -DtestForkCount=1 \
+ -DtestReuseFork=true
+
+ $MVN_TEST_COMMAND -pl pulsar-broker -Dinclude="**/*StreamingDispatcher*Test.java" \
+ -DtestForkCount=1 \
+ -DtestReuseFork=true
+
+ $MVN_TEST_COMMAND -pl pulsar-broker -Dinclude="org/apache/pulsar/broker/zookeeper/**/*.java,
+ org/apache/pulsar/broker/loadbalance/**/*.java,
+ org/apache/pulsar/broker/service/**/*.java" \
+ -Dexclude="**/ReplicatorTest.java,
+ **/MessagePublishBufferThrottleTest.java,
+ **/TopicOwnerTest.java,
+ **/*StreamingDispatcher*Test.java,
+ **/AntiAffinityNamespaceGroupTest.java"
+}
+
+function broker_client_api() {
+ $MVN_TEST_COMMAND -pl pulsar-broker -Dinclude="**/DispatcherBlockConsumerTest.java" \
+ -DtestForkCount=1 \
+ -DtestReuseFork=true
+
+ $MVN_TEST_COMMAND -pl pulsar-broker -Dinclude="**/SimpleProducerConsumerTest.java" \
+ -DtestForkCount=1 \
+ -DtestReuseFork=true
+
+ $MVN_TEST_COMMAND -pl pulsar-broker -Dinclude="org/apache/pulsar/client/api/**/*.java" \
+ -Dexclude="**/DispatcherBlockConsumerTest.java,
+ **/SimpleProducerConsumerTest.java"
+}
+
+function broker_client_impl() {
+ $MVN_TEST_COMMAND -pl pulsar-broker -Dinclude="org/apache/pulsar/client/impl/**/*.java"
+}
+
+function broker_client_other() {
+ $MVN_TEST_COMMAND -pl pulsar-broker -Dexclude="org/apache/pulsar/broker/**/*.java,
+ org/apache/pulsar/client/**/*.java"
+}
+
+function proxy() {
+ $MVN_TEST_COMMAND -pl pulsar-proxy -DtestForkCount=1 \
+ -DtestReuseFork=true \
+ -Dinclude="**/ProxyRolesEnforcementTest.java" \
+ -DtestForkCount=1 \
+ -DtestReuseFork=true
+
+ $MVN_TEST_COMMAND -pl pulsar-proxy -DtestForkCount=1 \
+ -DtestReuseFork=true \
+ -Dinclude="**/ProxyAuthenticationTest.java" \
+ -DtestForkCount=1 \
+ -DtestReuseFork=true
+
+ $MVN_TEST_COMMAND -pl pulsar-proxy -DtestForkCount=1 \
+ -DtestReuseFork=true \
+ -Dinclude="**/ProxyTest.java" \
+ -DtestForkCount=1 \
+ -DtestReuseFork=true
+
+ $MVN_TEST_COMMAND -pl pulsar-proxy -DtestForkCount=1 \
+ -DtestReuseFork=true \
+ -Dinclude="**/MessagePublishBufferThrottleTest.java" \
+ -DtestForkCount=1 \
+ -DtestReuseFork=true
+
+ $MVN_TEST_COMMAND -pl pulsar-proxy -DtestForkCount=1 \
+ -Dexclude="**/ProxyRolesEnforcementTest.java,
+ **/ProxyAuthenticationTest.java,
+ **/ProxyTest.java,
+ **/MessagePublishBufferThrottleTest.java" \
+ -DtestReuseFork=true
+}
+
+function other() {
+ build/retry.sh mvn -B -ntp install -PbrokerSkipTest \
+ -Dexclude="org/apache/pulsar/proxy/**/*.java,
+ **/ManagedLedgerTest.java,
+ **/TestPulsarKeyValueSchemaHandler.java,
+ **/PrimitiveSchemaTest.java,
+ BlobStoreManagedLedgerOffloaderTest.java"
+
+ $MVN_TEST_COMMAND -pl managed-ledger -Dinclude="**/ManagedLedgerTest.java" \
+ -DtestForkCount=1 \
+ -DtestReuseFork=true
+
+ $MVN_TEST_COMMAND -pl pulsar-sql/presto-pulsar-plugin -Dinclude="**/TestPulsarKeyValueSchemaHandler.java" \
+ -DtestForkCount=1 \
+ -DtestReuseFork=true
+
+ $MVN_TEST_COMMAND -pl pulsar-client -Dinclude="**/PrimitiveSchemaTest.java" \
+ -DtestForkCount=1 \
+ -DtestReuseFork=true
+
+ $MVN_TEST_COMMAND -pl tiered-storage/jcloud -Dinclude="**/BlobStoreManagedLedgerOffloaderTest.java" \
+ -DtestForkCount=1 \
+ -DtestReuseFork=true
+}
+
+# Test Groups -- end --
+
+TEST_GROUP=$1
+
+echo -n "Test Group : $TEST_GROUP"
+
+case $TEST_GROUP in
+
+ BROKER_GROUP_1)
+ broker_group_1
+ ;;
+
+ BROKER_GROUP_2)
+ broker_group_2
+ ;;
+
+ BROKER_CLIENT_API)
+ broker_client_api
+ ;;
+
+ BROKER_CLIENT_IMPL)
+ broker_client_impl
+ ;;
+
+ BROKER_CLIENT_OTHER)
+ broker_client_other
+ ;;
+
+ PROXY)
+ proxy
+ ;;
+
+ OTHER)
+ other
+ ;;
+
+ *)
+ echo -n "INVALID TEST GROUP"
+ exit 1
+ ;;
+esac
+
diff --git a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/WaitingEntryCallBack.java b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/WaitingEntryCallBack.java
new file mode 100644
index 0000000..fc1f3b4
--- /dev/null
+++ b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/WaitingEntryCallBack.java
@@ -0,0 +1,30 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.bookkeeper.mledger;
+
+/**
+ * Contains callback that can be registered with {@link ManagedLedger} to wait for new entries to be available.
+ */
+public interface WaitingEntryCallBack {
+
+ /**
+ * The callback {@link ManagedLedger} will trigger when new entries are available.
+ */
+ void entriesAvailable();
+}
diff --git a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java
index c0a7dd4..01f3902 100644
--- a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java
+++ b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java
@@ -111,6 +111,7 @@ import org.apache.bookkeeper.mledger.ManagedLedgerException.NonRecoverableLedger
import org.apache.bookkeeper.mledger.ManagedLedgerException.TooManyRequestsException;
import org.apache.bookkeeper.mledger.ManagedLedgerMXBean;
import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.WaitingEntryCallBack;
import org.apache.bookkeeper.mledger.impl.ManagedCursorImpl.VoidCallback;
import org.apache.bookkeeper.mledger.impl.MetaStore.MetaStoreCallback;
import org.apache.bookkeeper.mledger.offload.OffloadUtils;
@@ -172,6 +173,9 @@ public class ManagedLedgerImpl implements ManagedLedger, CreateCallback {
// Cursors that are waiting to be notified when new entries are persisted
final ConcurrentLinkedQueue<ManagedCursorImpl> waitingCursors;
+ // Objects that are waiting to be notified when new entries are persisted
+ final ConcurrentLinkedQueue<WaitingEntryCallBack> waitingEntryCallBacks;
+
// This map is used for concurrent open cursor requests, where the 2nd request will attach a listener to the
// uninitialized cursor future from the 1st request
final Map<String, CompletableFuture<ManagedCursor>> uninitializedCursors;
@@ -291,6 +295,7 @@ public class ManagedLedgerImpl implements ManagedLedger, CreateCallback {
this.mbean = new ManagedLedgerMBeanImpl(this);
this.entryCache = factory.getEntryCacheManager().getEntryCache(this);
this.waitingCursors = Queues.newConcurrentLinkedQueue();
+ this.waitingEntryCallBacks = Queues.newConcurrentLinkedQueue();
this.uninitializedCursors = Maps.newHashMap();
this.clock = config.getClock();
@@ -1961,6 +1966,21 @@ public class ManagedLedgerImpl implements ManagedLedger, CreateCallback {
}
}
+ void notifyWaitingEntryCallBacks() {
+ while (true) {
+ final WaitingEntryCallBack cb = waitingEntryCallBacks.poll();
+ if (cb == null) {
+ break;
+ }
+
+ executor.execute(safeRun(cb::entriesAvailable));
+ }
+ }
+
+ public void addWaitingEntryCallBack(WaitingEntryCallBack cb) {
+ this.waitingEntryCallBacks.add(cb);
+ }
+
private void trimConsumedLedgersInBackground() {
trimConsumedLedgersInBackground(Futures.NULL_PROMISE);
}
@@ -2942,7 +2962,7 @@ public class ManagedLedgerImpl implements ManagedLedger, CreateCallback {
* the position to validate
* @return true if the position is valid, false otherwise
*/
- boolean isValidPosition(PositionImpl position) {
+ public boolean isValidPosition(PositionImpl position) {
PositionImpl last = lastConfirmedEntry;
if (log.isDebugEnabled()) {
log.debug("IsValid position: {} -- last: {}", position, last);
@@ -2986,7 +3006,9 @@ public class ManagedLedgerImpl implements ManagedLedger, CreateCallback {
next = getNextValidPositionInternal(position);
} catch (NullPointerException e) {
next = lastConfirmedEntry.getNext();
- log.error("[{}] Can't find next valid position, fail back to the next position of the last position.", name, e);
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Can't find next valid position, fall back to the next position of the last position.", name, e);
+ }
}
return next;
}
diff --git a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java
index 78ad312..d72d237 100644
--- a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java
+++ b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java
@@ -191,6 +191,7 @@ class OpAddEntry extends SafeRunnable implements AddCallback, CloseCallback {
if (cb != null) {
cb.addComplete(lastEntry, ctx);
ml.notifyCursors();
+ ml.notifyWaitingEntryCallBacks();
this.recycle();
}
}
@@ -214,6 +215,7 @@ class OpAddEntry extends SafeRunnable implements AddCallback, CloseCallback {
if (cb != null) {
cb.addComplete(PositionImpl.get(lh.getId(), entryId), ctx);
ml.notifyCursors();
+ ml.notifyWaitingEntryCallBacks();
this.recycle();
}
}
diff --git a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
index c9001a2..5979e63 100644
--- a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
+++ b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
@@ -665,6 +665,13 @@ public class ServiceConfiguration implements PulsarConfiguration {
private boolean preciseDispatcherFlowControl = false;
@FieldContext(
+ category = CATEGORY_SERVER,
+ doc = "Whether to use streaming read dispatcher. Currently is in preview and can be changed " +
+ "in subsequent release."
+ )
+ private boolean streamingDispatch = false;
+
+ @FieldContext(
dynamic = true,
category = CATEGORY_SERVER,
doc = "Max number of concurrent lookup request broker allows to throttle heavy incoming lookup traffic")
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractDispatcherMultipleConsumers.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractDispatcherMultipleConsumers.java
index 434ff2d..1ebea27 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractDispatcherMultipleConsumers.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractDispatcherMultipleConsumers.java
@@ -73,6 +73,8 @@ public abstract class AbstractDispatcherMultipleConsumers extends AbstractBaseDi
public abstract boolean isConsumerAvailable(Consumer consumer);
+ protected void cancelPendingRead() {}
+
/**
* <pre>
* Broker gives more priority while dispatching messages. Here, broker follows descending priorities. (eg:
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherMultipleConsumers.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherMultipleConsumers.java
index 27b5e2a..934873a 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherMultipleConsumers.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherMultipleConsumers.java
@@ -82,31 +82,36 @@ public class PersistentDispatcherMultipleConsumers extends AbstractDispatcherMul
protected volatile Range<PositionImpl> lastIndividualDeletedRangeFromCursorRecovery;
private CompletableFuture<Void> closeFuture = null;
- LongPairSet messagesToRedeliver = new ConcurrentSortedLongPairSet(128, 2);
+ protected LongPairSet messagesToRedeliver = new ConcurrentSortedLongPairSet(128, 2);
protected final RedeliveryTracker redeliveryTracker;
private Optional<DelayedDeliveryTracker> delayedDeliveryTracker = Optional.empty();
- private volatile boolean havePendingRead = false;
- private volatile boolean havePendingReplayRead = false;
- private boolean shouldRewindBeforeReadingOrReplaying = false;
+ protected volatile boolean havePendingRead = false;
+ protected volatile boolean havePendingReplayRead = false;
+ protected boolean shouldRewindBeforeReadingOrReplaying = false;
protected final String name;
protected static final AtomicIntegerFieldUpdater<PersistentDispatcherMultipleConsumers> TOTAL_AVAILABLE_PERMITS_UPDATER =
AtomicIntegerFieldUpdater.newUpdater(PersistentDispatcherMultipleConsumers.class, "totalAvailablePermits");
protected volatile int totalAvailablePermits = 0;
- private volatile int readBatchSize;
- private final Backoff readFailureBackoff = new Backoff(15, TimeUnit.SECONDS, 1, TimeUnit.MINUTES, 0, TimeUnit.MILLISECONDS);
- private static final AtomicIntegerFieldUpdater<PersistentDispatcherMultipleConsumers> TOTAL_UNACKED_MESSAGES_UPDATER =
- AtomicIntegerFieldUpdater.newUpdater(PersistentDispatcherMultipleConsumers.class, "totalUnackedMessages");
- private volatile int totalUnackedMessages = 0;
+ protected volatile int readBatchSize;
+ protected final Backoff readFailureBackoff = new Backoff(15, TimeUnit.SECONDS,
+ 1, TimeUnit.MINUTES, 0, TimeUnit.MILLISECONDS);
+ private static final AtomicIntegerFieldUpdater<PersistentDispatcherMultipleConsumers>
+ TOTAL_UNACKED_MESSAGES_UPDATER =
+ AtomicIntegerFieldUpdater.newUpdater(PersistentDispatcherMultipleConsumers.class,
+ "totalUnackedMessages");
+ protected volatile int totalUnackedMessages = 0;
private volatile int blockedDispatcherOnUnackedMsgs = FALSE;
- private static final AtomicIntegerFieldUpdater<PersistentDispatcherMultipleConsumers> BLOCKED_DISPATCHER_ON_UNACKMSG_UPDATER =
- AtomicIntegerFieldUpdater.newUpdater(PersistentDispatcherMultipleConsumers.class, "blockedDispatcherOnUnackedMsgs");
+ protected static final AtomicIntegerFieldUpdater<PersistentDispatcherMultipleConsumers>
+ BLOCKED_DISPATCHER_ON_UNACKMSG_UPDATER =
+ AtomicIntegerFieldUpdater.newUpdater(PersistentDispatcherMultipleConsumers.class,
+ "blockedDispatcherOnUnackedMsgs");
protected final ServiceConfiguration serviceConfig;
protected Optional<DispatchRateLimiter> dispatchRateLimiter = Optional.empty();
- enum ReadType {
+ protected enum ReadType {
Normal, Replay
}
@@ -193,9 +198,7 @@ public class PersistentDispatcherMultipleConsumers extends AbstractDispatcherMul
consumerList.remove(consumer);
log.info("Removed consumer {} with pending {} acks", consumer, consumer.getPendingAcks().size());
if (consumerList.isEmpty()) {
- if (havePendingRead && cursor.cancelPendingReadRequest()) {
- havePendingRead = false;
- }
+ cancelPendingRead();
messagesToRedeliver.clear();
redeliveryTracker.clear();
@@ -248,76 +251,13 @@ public class PersistentDispatcherMultipleConsumers extends AbstractDispatcherMul
// totalAvailablePermits may be updated by other threads
int currentTotalAvailablePermits = totalAvailablePermits;
if (currentTotalAvailablePermits > 0 && isAtleastOneConsumerAvailable()) {
- int messagesToRead = Math.min(currentTotalAvailablePermits, readBatchSize);
-
- Consumer c = getRandomConsumer();
- // if turn on precise dispatcher flow control, adjust the record to read
- if (c != null && c.isPreciseDispatcherFlowControl()) {
- messagesToRead = Math.min((int) Math.ceil(currentTotalAvailablePermits * 1.0 / c.getAvgMessagesPerEntry()), readBatchSize);
- }
-
- if (!isConsumerWritable()) {
- // If the connection is not currently writable, we issue the read request anyway, but for a single
- // message. The intent here is to keep use the request as a notification mechanism while avoiding to
- // read and dispatch a big batch of messages which will need to wait before getting written to the
- // socket.
- messagesToRead = 1;
- }
-
- // throttle only if: (1) cursor is not active (or flag for throttle-nonBacklogConsumer is enabled) bcz
- // active-cursor reads message from cache rather from bookkeeper (2) if topic has reached message-rate
- // threshold: then schedule the read after MESSAGE_RATE_BACKOFF_MS
- if (serviceConfig.isDispatchThrottlingOnNonBacklogConsumerEnabled() || !cursor.isActive()) {
- if (topic.getDispatchRateLimiter().isPresent() && topic.getDispatchRateLimiter().get().isDispatchRateLimitingEnabled()) {
- DispatchRateLimiter topicRateLimiter = topic.getDispatchRateLimiter().get();
- if (!topicRateLimiter.hasMessageDispatchPermit()) {
- if (log.isDebugEnabled()) {
- log.debug("[{}] message-read exceeded topic message-rate {}/{}, schedule after a {}", name,
- topicRateLimiter.getDispatchRateOnMsg(), topicRateLimiter.getDispatchRateOnByte(),
- MESSAGE_RATE_BACKOFF_MS);
- }
- topic.getBrokerService().executor().schedule(() -> readMoreEntries(), MESSAGE_RATE_BACKOFF_MS,
- TimeUnit.MILLISECONDS);
- return;
- } else {
- // if dispatch-rate is in msg then read only msg according to available permit
- long availablePermitsOnMsg = topicRateLimiter.getAvailableDispatchRateLimitOnMsg();
- if (availablePermitsOnMsg > 0) {
- messagesToRead = Math.min(messagesToRead, (int) availablePermitsOnMsg);
- }
- }
- }
+ int messagesToRead = calculateNumOfMessageToRead(currentTotalAvailablePermits);
- if (dispatchRateLimiter.isPresent() && dispatchRateLimiter.get().isDispatchRateLimitingEnabled()) {
- if (!dispatchRateLimiter.get().hasMessageDispatchPermit()) {
- if (log.isDebugEnabled()) {
- log.debug("[{}] message-read exceeded subscription message-rate {}/{}, schedule after a {}", name,
- dispatchRateLimiter.get().getDispatchRateOnMsg(), dispatchRateLimiter.get().getDispatchRateOnByte(),
- MESSAGE_RATE_BACKOFF_MS);
- }
- topic.getBrokerService().executor().schedule(() -> readMoreEntries(), MESSAGE_RATE_BACKOFF_MS,
- TimeUnit.MILLISECONDS);
- return;
- } else {
- // if dispatch-rate is in msg then read only msg according to available permit
- long availablePermitsOnMsg = dispatchRateLimiter.get().getAvailableDispatchRateLimitOnMsg();
- if (availablePermitsOnMsg > 0) {
- messagesToRead = Math.min(messagesToRead, (int) availablePermitsOnMsg);
- }
- }
- }
-
- }
-
- if (havePendingReplayRead) {
- if (log.isDebugEnabled()) {
- log.debug("[{}] Skipping replay while awaiting previous read to complete", name);
- }
+ if (-1 == messagesToRead) {
+ // Skip read as topic/dispatcher has exceed the dispatch rate or previous pending read hasn't complete.
return;
}
- // If messagesToRead is 0 or less, correct it to 1 to prevent IllegalArgumentException
- messagesToRead = Math.max(messagesToRead, 1);
Set<PositionImpl> messagesToReplayNow = getMessagesToReplayNow(messagesToRead);
if (!messagesToReplayNow.isEmpty()) {
@@ -360,6 +300,84 @@ public class PersistentDispatcherMultipleConsumers extends AbstractDispatcherMul
}
}
+ protected int calculateNumOfMessageToRead(int currentTotalAvailablePermits) {
+ int messagesToRead = Math.min(currentTotalAvailablePermits, readBatchSize);
+
+ Consumer c = getRandomConsumer();
+ // if turn on precise dispatcher flow control, adjust the record to read
+ if (c != null && c.isPreciseDispatcherFlowControl()) {
+ messagesToRead = Math.min(
+ (int) Math.ceil(currentTotalAvailablePermits * 1.0 / c.getAvgMessagesPerEntry()),
+ readBatchSize);
+ }
+
+ if (!isConsumerWritable()) {
+ // If the connection is not currently writable, we issue the read request anyway, but for a single
+ // message. The intent here is to keep use the request as a notification mechanism while avoiding to
+ // read and dispatch a big batch of messages which will need to wait before getting written to the
+ // socket.
+ messagesToRead = 1;
+ }
+
+ // throttle only if: (1) cursor is not active (or flag for throttle-nonBacklogConsumer is enabled) bcz
+ // active-cursor reads message from cache rather from bookkeeper (2) if topic has reached message-rate
+ // threshold: then schedule the read after MESSAGE_RATE_BACKOFF_MS
+ if (serviceConfig.isDispatchThrottlingOnNonBacklogConsumerEnabled() || !cursor.isActive()) {
+ if (topic.getDispatchRateLimiter().isPresent()
+ && topic.getDispatchRateLimiter().get().isDispatchRateLimitingEnabled()) {
+ DispatchRateLimiter topicRateLimiter = topic.getDispatchRateLimiter().get();
+ if (!topicRateLimiter.hasMessageDispatchPermit()) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] message-read exceeded topic message-rate {}/{}, schedule after a {}", name,
+ topicRateLimiter.getDispatchRateOnMsg(), topicRateLimiter.getDispatchRateOnByte(),
+ MESSAGE_RATE_BACKOFF_MS);
+ }
+ topic.getBrokerService().executor().schedule(() -> readMoreEntries(), MESSAGE_RATE_BACKOFF_MS,
+ TimeUnit.MILLISECONDS);
+ return -1;
+ } else {
+ // if dispatch-rate is in msg then read only msg according to available permit
+ long availablePermitsOnMsg = topicRateLimiter.getAvailableDispatchRateLimitOnMsg();
+ if (availablePermitsOnMsg > 0) {
+ messagesToRead = Math.min(messagesToRead, (int) availablePermitsOnMsg);
+ }
+ }
+ }
+
+ if (dispatchRateLimiter.isPresent() && dispatchRateLimiter.get().isDispatchRateLimitingEnabled()) {
+ if (!dispatchRateLimiter.get().hasMessageDispatchPermit()) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] message-read exceeded subscription message-rate {}/{},"
+ + " schedule after a {}", name,
+ dispatchRateLimiter.get().getDispatchRateOnMsg(),
+ dispatchRateLimiter.get().getDispatchRateOnByte(),
+ MESSAGE_RATE_BACKOFF_MS);
+ }
+ topic.getBrokerService().executor().schedule(() -> readMoreEntries(), MESSAGE_RATE_BACKOFF_MS,
+ TimeUnit.MILLISECONDS);
+ return -1;
+ } else {
+ // if dispatch-rate is in msg then read only msg according to available permit
+ long availablePermitsOnMsg = dispatchRateLimiter.get().getAvailableDispatchRateLimitOnMsg();
+ if (availablePermitsOnMsg > 0) {
+ messagesToRead = Math.min(messagesToRead, (int) availablePermitsOnMsg);
+ }
+ }
+ }
+
+ }
+
+ if (havePendingReplayRead) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Skipping replay while awaiting previous read to complete", name);
+ }
+ return -1;
+ }
+
+ // If messagesToRead is 0 or less, correct it to 1 to prevent IllegalArgumentException
+ return Math.max(messagesToRead, 1);
+ }
+
protected Set<? extends Position> asyncReplayEntries(Set<? extends Position> positions) {
return cursor.asyncReplayEntries(positions, this, ReadType.Replay);
}
@@ -407,14 +425,19 @@ public class PersistentDispatcherMultipleConsumers extends AbstractDispatcherMul
closeFuture.complete(null);
} else {
consumerList.forEach(consumer -> consumer.disconnect(isResetCursor));
- if (havePendingRead && cursor.cancelPendingReadRequest()) {
- havePendingRead = false;
- }
+ cancelPendingRead();
}
return closeFuture;
}
@Override
+ protected void cancelPendingRead() {
+ if (havePendingRead && cursor.cancelPendingReadRequest()) {
+ havePendingRead = false;
+ }
+ }
+
+ @Override
public CompletableFuture<Void> disconnectActiveConsumers(boolean isResetCursor) {
return disconnectAllConsumers(isResetCursor);
}
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherSingleActiveConsumer.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherSingleActiveConsumer.java
index a5c8a5f..48c82d1 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherSingleActiveConsumer.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherSingleActiveConsumer.java
@@ -63,18 +63,20 @@ import org.apache.pulsar.common.util.collections.LongPairSet;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-public final class PersistentDispatcherSingleActiveConsumer extends AbstractDispatcherSingleActiveConsumer implements Dispatcher, ReadEntriesCallback {
+public class PersistentDispatcherSingleActiveConsumer extends AbstractDispatcherSingleActiveConsumer
+ implements Dispatcher, ReadEntriesCallback {
- private final PersistentTopic topic;
- private final ManagedCursor cursor;
- private final String name;
+ protected final PersistentTopic topic;
+ protected final ManagedCursor cursor;
+ protected final String name;
private Optional<DispatchRateLimiter> dispatchRateLimiter = Optional.empty();
- private volatile boolean havePendingRead = false;
+ protected volatile boolean havePendingRead = false;
- private volatile int readBatchSize;
- private final Backoff readFailureBackoff = new Backoff(15, TimeUnit.SECONDS, 1, TimeUnit.MINUTES, 0, TimeUnit.MILLISECONDS);
- private final ServiceConfiguration serviceConfig;
+ protected volatile int readBatchSize;
+ protected final Backoff readFailureBackoff = new Backoff(15, TimeUnit.SECONDS,
+ 1, TimeUnit.MINUTES, 0, TimeUnit.MILLISECONDS);
+ protected final ServiceConfiguration serviceConfig;
private volatile ScheduledFuture<?> readOnActiveConsumerTask = null;
LongPairSet messagesToRedeliver;
@@ -98,9 +100,7 @@ public final class PersistentDispatcherSingleActiveConsumer extends AbstractDisp
}
protected void scheduleReadOnActiveConsumer() {
- if (havePendingRead && cursor.cancelPendingReadRequest()) {
- havePendingRead = false;
- }
+ cancelPendingRead();
if (havePendingRead) {
return;
@@ -175,6 +175,7 @@ public final class PersistentDispatcherSingleActiveConsumer extends AbstractDisp
return false;
}
+ @Override
protected void cancelPendingRead() {
if (havePendingRead && cursor.cancelPendingReadRequest()) {
havePendingRead = false;
@@ -246,46 +247,49 @@ public final class PersistentDispatcherSingleActiveConsumer extends AbstractDisp
EntryBatchSizes batchSizes = EntryBatchSizes.get(entries.size());
SendMessageInfo sendMessageInfo = SendMessageInfo.getThreadLocal();
EntryBatchIndexesAcks batchIndexesAcks = EntryBatchIndexesAcks.get(entries.size());
- filterEntriesForConsumer(entries, batchSizes, sendMessageInfo, batchIndexesAcks, cursor, isReplayRead);
-
- int totalMessages = sendMessageInfo.getTotalMessages();
- long totalBytes = sendMessageInfo.getTotalBytes();
-
- currentConsumer
- .sendMessages(entries, batchSizes, batchIndexesAcks, sendMessageInfo.getTotalMessages(),
- sendMessageInfo.getTotalBytes(), sendMessageInfo.getTotalChunkedMessages(),
- redeliveryTracker)
- .addListener(future -> {
- if (future.isSuccess()) {
- // acquire message-dispatch permits for already delivered messages
- if (serviceConfig.isDispatchThrottlingOnNonBacklogConsumerEnabled() || !cursor.isActive()) {
- if (topic.getDispatchRateLimiter().isPresent()) {
- topic.getDispatchRateLimiter().get().tryDispatchPermit(totalMessages, totalBytes);
- }
-
- dispatchRateLimiter.ifPresent(rateLimiter -> rateLimiter.tryDispatchPermit(totalMessages, totalBytes));
- }
+ filterEntriesForConsumer(entries, batchSizes, sendMessageInfo, batchIndexesAcks, cursor, false);
+ dispatchEntriesToConsumer(currentConsumer, entries, batchSizes, batchIndexesAcks, sendMessageInfo);
+ }
+ }
- // Schedule a new read batch operation only after the previous batch has been written to the
- // socket
- topic.getBrokerService().getTopicOrderedExecutor().executeOrdered(topicName,
- SafeRun.safeRun(() -> {
- synchronized (PersistentDispatcherSingleActiveConsumer.this) {
- Consumer newConsumer = getActiveConsumer();
- if (newConsumer != null && !havePendingRead) {
- readMoreEntries(newConsumer);
- } else {
- if (log.isDebugEnabled()) {
- log.debug(
- "[{}-{}] Ignoring write future complete. consumerAvailable={} havePendingRead={}",
- name, newConsumer, newConsumer != null, havePendingRead);
- }
- }
- }
- }));
+ protected void dispatchEntriesToConsumer(Consumer currentConsumer, List<Entry> entries,
+ EntryBatchSizes batchSizes, EntryBatchIndexesAcks batchIndexesAcks,
+ SendMessageInfo sendMessageInfo) {
+ currentConsumer
+ .sendMessages(entries, batchSizes, batchIndexesAcks, sendMessageInfo.getTotalMessages(),
+ sendMessageInfo.getTotalBytes(), sendMessageInfo.getTotalChunkedMessages(),
+ redeliveryTracker)
+ .addListener(future -> {
+ if (future.isSuccess()) {
+ // acquire message-dispatch permits for already delivered messages
+ if (serviceConfig.isDispatchThrottlingOnNonBacklogConsumerEnabled() || !cursor.isActive()) {
+ if (topic.getDispatchRateLimiter().isPresent()) {
+ topic.getDispatchRateLimiter().get().tryDispatchPermit(sendMessageInfo.getTotalMessages(),
+ sendMessageInfo.getTotalBytes());
}
- });
- }
+
+ dispatchRateLimiter.ifPresent(rateLimiter ->
+ rateLimiter.tryDispatchPermit(sendMessageInfo.getTotalMessages(),
+ sendMessageInfo.getTotalBytes()));
+ }
+
+ // Schedule a new read batch operation only after the previous batch has been written to the socket.
+ topic.getBrokerService().getTopicOrderedExecutor().executeOrdered(topicName,
+ SafeRun.safeRun(() -> {
+ synchronized (PersistentDispatcherSingleActiveConsumer.this) {
+ Consumer newConsumer = getActiveConsumer();
+ if (newConsumer != null && !havePendingRead) {
+ readMoreEntries(newConsumer);
+ } else {
+ log.debug(
+ "[{}-{}] Ignoring write future complete."
+ + " consumerAvailable={} havePendingRead={}",
+ name, newConsumer, newConsumer != null, havePendingRead);
+ }
+ }
+ }));
+ }
+ });
}
@Override
@@ -339,9 +343,7 @@ public final class PersistentDispatcherSingleActiveConsumer extends AbstractDisp
return;
}
- if (havePendingRead && cursor.cancelPendingReadRequest()) {
- havePendingRead = false;
- }
+ cancelPendingRead();
if (!havePendingRead) {
cursor.rewind();
@@ -371,90 +373,14 @@ public final class PersistentDispatcherSingleActiveConsumer extends AbstractDisp
return;
}
- int availablePermits = consumer.getAvailablePermits();
-
- if (availablePermits > 0) {
- if (!consumer.isWritable()) {
- // If the connection is not currently writable, we issue the read request anyway, but for a single
- // message. The intent here is to keep use the request as a notification mechanism while avoiding to
- // read and dispatch a big batch of messages which will need to wait before getting written to the
- // socket.
- availablePermits = 1;
- }
-
- int messagesToRead = Math.min(availablePermits, readBatchSize);
- // if turn of precise dispatcher flow control, adjust the records to read
- if (consumer.isPreciseDispatcherFlowControl()) {
- int avgMessagesPerEntry = consumer.getAvgMessagesPerEntry();
- messagesToRead = Math.min((int) Math.ceil(availablePermits * 1.0 / avgMessagesPerEntry), readBatchSize);
- }
+ if (consumer.getAvailablePermits() > 0) {
+ int messagesToRead = calculateNumOfMessageToRead(consumer);
- // throttle only if: (1) cursor is not active (or flag for throttle-nonBacklogConsumer is enabled) bcz
- // active-cursor reads message from cache rather from bookkeeper (2) if topic has reached message-rate
- // threshold: then schedule the read after MESSAGE_RATE_BACKOFF_MS
- if (serviceConfig.isDispatchThrottlingOnNonBacklogConsumerEnabled() || !cursor.isActive()) {
- if (topic.getDispatchRateLimiter().isPresent()
- && topic.getDispatchRateLimiter().get().isDispatchRateLimitingEnabled()) {
- DispatchRateLimiter topicRateLimiter = topic.getDispatchRateLimiter().get();
- if (!topicRateLimiter.hasMessageDispatchPermit()) {
- if (log.isDebugEnabled()) {
- log.debug("[{}] message-read exceeded topic message-rate {}/{}, schedule after a {}", name,
- topicRateLimiter.getDispatchRateOnMsg(), topicRateLimiter.getDispatchRateOnByte(),
- MESSAGE_RATE_BACKOFF_MS);
- }
- topic.getBrokerService().executor().schedule(() -> {
- Consumer currentConsumer = ACTIVE_CONSUMER_UPDATER.get(this);
- if (currentConsumer != null && !havePendingRead) {
- readMoreEntries(currentConsumer);
- } else {
- if (log.isDebugEnabled()) {
- log.debug("[{}] Skipping read retry for topic: Current Consumer {}, havePendingRead {}",
- topic.getName(), currentConsumer, havePendingRead);
- }
- }
- }, MESSAGE_RATE_BACKOFF_MS, TimeUnit.MILLISECONDS);
- return;
- } else {
- // if dispatch-rate is in msg then read only msg according to available permit
- long availablePermitsOnMsg = topicRateLimiter.getAvailableDispatchRateLimitOnMsg();
- if (availablePermitsOnMsg > 0) {
- messagesToRead = Math.min(messagesToRead, (int) availablePermitsOnMsg);
- }
- }
- }
-
- if (dispatchRateLimiter.isPresent() && dispatchRateLimiter.get().isDispatchRateLimitingEnabled()) {
- if (!dispatchRateLimiter.get().hasMessageDispatchPermit()) {
- if (log.isDebugEnabled()) {
- log.debug("[{}] message-read exceeded subscription message-rate {}/{}, schedule after a {}", name,
- dispatchRateLimiter.get().getDispatchRateOnMsg(), dispatchRateLimiter.get().getDispatchRateOnByte(),
- MESSAGE_RATE_BACKOFF_MS);
- }
- topic.getBrokerService().executor().schedule(() -> {
- Consumer currentConsumer = ACTIVE_CONSUMER_UPDATER.get(this);
- if (currentConsumer != null && !havePendingRead) {
- readMoreEntries(currentConsumer);
- } else {
- if (log.isDebugEnabled()) {
- log.debug("[{}] Skipping read retry: Current Consumer {}, havePendingRead {}",
- topic.getName(), currentConsumer, havePendingRead);
- }
- }
- }, MESSAGE_RATE_BACKOFF_MS, TimeUnit.MILLISECONDS);
- return;
- } else {
- // if dispatch-rate is in msg then read only msg according to available permit
- long subPermitsOnMsg = dispatchRateLimiter.get().getAvailableDispatchRateLimitOnMsg();
- if (subPermitsOnMsg > 0) {
- messagesToRead = Math.min(messagesToRead, (int) subPermitsOnMsg);
- }
- }
- }
+ if (-1 == messagesToRead) {
+ // Skip read as topic/dispatcher has exceed the dispatch rate.
+ return;
}
- // If messagesToRead is 0 or less, correct it to 1 to prevent IllegalArgumentException
- messagesToRead = Math.max(messagesToRead, 1);
-
// Schedule read
if (log.isDebugEnabled()) {
log.debug("[{}-{}] Schedule read of {} messages", name, consumer, messagesToRead);
@@ -488,6 +414,93 @@ public final class PersistentDispatcherSingleActiveConsumer extends AbstractDisp
}
}
+ protected int calculateNumOfMessageToRead(Consumer consumer) {
+ int availablePermits = consumer.getAvailablePermits();
+ if (!consumer.isWritable()) {
+ // If the connection is not currently writable, we issue the read request anyway, but for a single
+ // message. The intent here is to keep use the request as a notification mechanism while avoiding to
+ // read and dispatch a big batch of messages which will need to wait before getting written to the
+ // socket.
+ availablePermits = 1;
+ }
+
+ int messagesToRead = Math.min(availablePermits, readBatchSize);
+ // if turn of precise dispatcher flow control, adjust the records to read
+ if (consumer.isPreciseDispatcherFlowControl()) {
+ int avgMessagesPerEntry = consumer.getAvgMessagesPerEntry();
+ messagesToRead = Math.min((int) Math.ceil(availablePermits * 1.0 / avgMessagesPerEntry), readBatchSize);
+ }
+
+ // throttle only if: (1) cursor is not active (or flag for throttle-nonBacklogConsumer is enabled) bcz
+ // active-cursor reads message from cache rather from bookkeeper (2) if topic has reached message-rate
+ // threshold: then schedule the read after MESSAGE_RATE_BACKOFF_MS
+ if (serviceConfig.isDispatchThrottlingOnNonBacklogConsumerEnabled() || !cursor.isActive()) {
+ if (topic.getDispatchRateLimiter().isPresent()
+ && topic.getDispatchRateLimiter().get().isDispatchRateLimitingEnabled()) {
+ DispatchRateLimiter topicRateLimiter = topic.getDispatchRateLimiter().get();
+ if (!topicRateLimiter.hasMessageDispatchPermit()) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] message-read exceeded topic message-rate {}/{}, schedule after a {}", name,
+ topicRateLimiter.getDispatchRateOnMsg(), topicRateLimiter.getDispatchRateOnByte(),
+ MESSAGE_RATE_BACKOFF_MS);
+ }
+ topic.getBrokerService().executor().schedule(() -> {
+ Consumer currentConsumer = ACTIVE_CONSUMER_UPDATER.get(this);
+ if (currentConsumer != null && !havePendingRead) {
+ readMoreEntries(currentConsumer);
+ } else {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Skipping read retry for topic: Current Consumer {},"
+ + " havePendingRead {}",
+ topic.getName(), currentConsumer, havePendingRead);
+ }
+ }
+ }, MESSAGE_RATE_BACKOFF_MS, TimeUnit.MILLISECONDS);
+ return -1;
+ } else {
+ // if dispatch-rate is in msg then read only msg according to available permit
+ long availablePermitsOnMsg = topicRateLimiter.getAvailableDispatchRateLimitOnMsg();
+ if (availablePermitsOnMsg > 0) {
+ messagesToRead = Math.min(messagesToRead, (int) availablePermitsOnMsg);
+ }
+ }
+ }
+
+ if (dispatchRateLimiter.isPresent() && dispatchRateLimiter.get().isDispatchRateLimitingEnabled()) {
+ if (!dispatchRateLimiter.get().hasMessageDispatchPermit()) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] message-read exceeded subscription message-rate {}/{},"
+ + " schedule after a {}",
+ name, dispatchRateLimiter.get().getDispatchRateOnMsg(),
+ dispatchRateLimiter.get().getDispatchRateOnByte(),
+ MESSAGE_RATE_BACKOFF_MS);
+ }
+ topic.getBrokerService().executor().schedule(() -> {
+ Consumer currentConsumer = ACTIVE_CONSUMER_UPDATER.get(this);
+ if (currentConsumer != null && !havePendingRead) {
+ readMoreEntries(currentConsumer);
+ } else {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Skipping read retry: Current Consumer {}, havePendingRead {}",
+ topic.getName(), currentConsumer, havePendingRead);
+ }
+ }
+ }, MESSAGE_RATE_BACKOFF_MS, TimeUnit.MILLISECONDS);
+ return -1;
+ } else {
+ // if dispatch-rate is in msg then read only msg according to available permit
+ long subPermitsOnMsg = dispatchRateLimiter.get().getAvailableDispatchRateLimitOnMsg();
+ if (subPermitsOnMsg > 0) {
+ messagesToRead = Math.min(messagesToRead, (int) subPermitsOnMsg);
+ }
+ }
+ }
+ }
+
+ // If messagesToRead is 0 or less, correct it to 1 to prevent IllegalArgumentException
+ return Math.max(messagesToRead, 1);
+ }
+
@Override
public void readEntriesFailed(ManagedLedgerException exception, Object ctx) {
topic.getBrokerService().getTopicOrderedExecutor().executeOrdered(topicName, SafeRun.safeRun(() -> {
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStreamingDispatcherMultipleConsumers.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStreamingDispatcherMultipleConsumers.java
new file mode 100644
index 0000000..9340e17
--- /dev/null
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStreamingDispatcherMultipleConsumers.java
@@ -0,0 +1,191 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.service.persistent;
+
+import com.google.common.collect.Lists;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedCursor;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedCursorImpl;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.util.SafeRun;
+import org.apache.pulsar.broker.service.Consumer;
+import org.apache.pulsar.broker.service.Subscription;
+import org.apache.pulsar.broker.service.streamingdispatch.PendingReadEntryRequest;
+import org.apache.pulsar.broker.service.streamingdispatch.StreamingDispatcher;
+import org.apache.pulsar.broker.service.streamingdispatch.StreamingEntryReader;
+
+/**
+ * A {@link PersistentDispatcherMultipleConsumers} implemented {@link StreamingDispatcher}.
+ * It'll use {@link StreamingEntryReader} to read new entries instead read as micro batch from managed ledger.
+ */
+@Slf4j
+public class PersistentStreamingDispatcherMultipleConsumers extends PersistentDispatcherMultipleConsumers
+ implements StreamingDispatcher {
+
+ private final StreamingEntryReader streamingEntryReader = new StreamingEntryReader((ManagedCursorImpl) cursor,
+ this, topic);
+
+ public PersistentStreamingDispatcherMultipleConsumers(PersistentTopic topic, ManagedCursor cursor,
+ Subscription subscription) {
+ super(topic, cursor, subscription);
+ }
+
+ /**
+ * {@inheritDoc}
+ */
+ @Override
+ public synchronized void readEntryComplete(Entry entry, PendingReadEntryRequest ctx) {
+
+ ReadType readType = (ReadType) ctx.ctx;
+ if (ctx.isLast()) {
+ readFailureBackoff.reduceToHalf();
+ if (readType == ReadType.Normal) {
+ havePendingRead = false;
+ } else {
+ havePendingReplayRead = false;
+ }
+ }
+
+ if (readBatchSize < serviceConfig.getDispatcherMaxReadBatchSize()) {
+ int newReadBatchSize = Math.min(readBatchSize * 2, serviceConfig.getDispatcherMaxReadBatchSize());
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Increasing read batch size from {} to {}", name, readBatchSize, newReadBatchSize);
+ }
+ readBatchSize = newReadBatchSize;
+ }
+
+ if (shouldRewindBeforeReadingOrReplaying && readType == ReadType.Normal) {
+ // All consumers got disconnected before the completion of the read operation
+ entry.release();
+ cursor.rewind();
+ shouldRewindBeforeReadingOrReplaying = false;
+ readMoreEntries();
+ return;
+ }
+
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Distributing a messages to {} consumers", name, consumerList.size());
+ }
+
+ cursor.seek(((ManagedLedgerImpl) cursor.getManagedLedger())
+ .getNextValidPosition((PositionImpl) entry.getPosition()));
+ sendMessagesToConsumers(readType, Lists.newArrayList(entry));
+ ctx.recycle();
+ }
+
+ /**
+ * {@inheritDoc}
+ */
+ @Override
+ public void canReadMoreEntries(boolean withBackoff) {
+ havePendingRead = false;
+ topic.getBrokerService().executor().schedule(() -> {
+ topic.getBrokerService().getTopicOrderedExecutor().executeOrdered(topic.getName(), SafeRun.safeRun(() -> {
+ synchronized (PersistentStreamingDispatcherMultipleConsumers.this) {
+ if (!havePendingRead) {
+ log.info("[{}] Scheduling read operation", name);
+ readMoreEntries();
+ } else {
+ log.info("[{}] Skipping read since we have pendingRead", name);
+ }
+ }
+ }));
+ }, withBackoff
+ ? readFailureBackoff.next() : 0, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * {@inheritDoc}
+ */
+ @Override
+ public void notifyConsumersEndOfTopic() {
+ if (cursor.getNumberOfEntriesInBacklog(false) == 0) {
+ // Topic has been terminated and there are no more entries to read
+ // Notify the consumer only if all the messages were already acknowledged
+ consumerList.forEach(Consumer::reachedEndOfTopic);
+ }
+ }
+
+ @Override
+ protected void cancelPendingRead() {
+ if (havePendingRead && streamingEntryReader.cancelReadRequests()) {
+ havePendingRead = false;
+ }
+ }
+
+ @Override
+ public void readMoreEntries() {
+ // totalAvailablePermits may be updated by other threads
+ int currentTotalAvailablePermits = totalAvailablePermits;
+ if (currentTotalAvailablePermits > 0 && isAtleastOneConsumerAvailable()) {
+ int messagesToRead = calculateNumOfMessageToRead(currentTotalAvailablePermits);
+
+ if (-1 == messagesToRead) {
+ // Skip read as topic/dispatcher has exceed the dispatch rate or previous pending read hasn't complete.
+ return;
+ }
+
+ Set<PositionImpl> messagesToReplayNow = getMessagesToReplayNow(messagesToRead);
+
+ if (!messagesToReplayNow.isEmpty()) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Schedule replay of {} messages for {} consumers", name, messagesToReplayNow.size(),
+ consumerList.size());
+ }
+
+ havePendingReplayRead = true;
+ Set<? extends Position> deletedMessages = topic.isDelayedDeliveryEnabled()
+ ? asyncReplayEntriesInOrder(messagesToReplayNow) : asyncReplayEntries(messagesToReplayNow);
+ // clear already acked positions from replay bucket
+
+ deletedMessages.forEach(position -> messagesToRedeliver.remove(((PositionImpl) position).getLedgerId(),
+ ((PositionImpl) position).getEntryId()));
+ // if all the entries are acked-entries and cleared up from messagesToRedeliver, try to read
+ // next entries as readCompletedEntries-callback was never called
+ if ((messagesToReplayNow.size() - deletedMessages.size()) == 0) {
+ havePendingReplayRead = false;
+ readMoreEntries();
+ }
+ } else if (BLOCKED_DISPATCHER_ON_UNACKMSG_UPDATER.get(this) == TRUE) {
+ log.warn("[{}] Dispatcher read is blocked due to unackMessages {} reached to max {}", name,
+ totalUnackedMessages, topic.getMaxUnackedMessagesOnSubscription());
+ } else if (!havePendingRead) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Schedule read of {} messages for {} consumers", name, messagesToRead,
+ consumerList.size());
+ }
+ havePendingRead = true;
+ streamingEntryReader.asyncReadEntries(messagesToRead, serviceConfig.getDispatcherMaxReadSizeBytes(),
+ ReadType.Normal);
+ } else {
+ log.debug("[{}] Cannot schedule next read until previous one is done", name);
+ }
+ } else {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Consumer buffer is full, pause reading", name);
+ }
+ }
+ }
+
+}
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStreamingDispatcherSingleActiveConsumer.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStreamingDispatcherSingleActiveConsumer.java
new file mode 100644
index 0000000..b4e4ed3
--- /dev/null
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStreamingDispatcherSingleActiveConsumer.java
@@ -0,0 +1,208 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.service.persistent;
+
+import static org.apache.bookkeeper.mledger.util.SafeRun.safeRun;
+import com.google.common.collect.Lists;
+import java.util.concurrent.TimeUnit;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedCursor;
+import org.apache.bookkeeper.mledger.impl.ManagedCursorImpl;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.util.SafeRun;
+import org.apache.pulsar.broker.service.Consumer;
+import org.apache.pulsar.broker.service.EntryBatchIndexesAcks;
+import org.apache.pulsar.broker.service.EntryBatchSizes;
+import org.apache.pulsar.broker.service.SendMessageInfo;
+import org.apache.pulsar.broker.service.Subscription;
+import org.apache.pulsar.broker.service.streamingdispatch.PendingReadEntryRequest;
+import org.apache.pulsar.broker.service.streamingdispatch.StreamingDispatcher;
+import org.apache.pulsar.broker.service.streamingdispatch.StreamingEntryReader;
+import org.apache.pulsar.common.api.proto.CommandSubscribe.SubType;
+
+/**
+ * A {@link PersistentDispatcherSingleActiveConsumer} implemented {@link StreamingDispatcher}.
+ * It'll use {@link StreamingEntryReader} to read new entries instead read as micro batch from managed ledger.
+ */
+@Slf4j
+public class PersistentStreamingDispatcherSingleActiveConsumer extends PersistentDispatcherSingleActiveConsumer
+ implements StreamingDispatcher {
+
+ private final StreamingEntryReader streamingEntryReader = new StreamingEntryReader((ManagedCursorImpl) cursor,
+ this, topic);
+
+ public PersistentStreamingDispatcherSingleActiveConsumer(ManagedCursor cursor, SubType subscriptionType,
+ int partitionIndex, PersistentTopic topic,
+ Subscription subscription) {
+ super(cursor, subscriptionType, partitionIndex, topic, subscription);
+ }
+
+ /**
+ * {@inheritDoc}
+ */
+ @Override
+ public void canReadMoreEntries(boolean withBackoff) {
+ havePendingRead = false;
+ topic.getBrokerService().executor().schedule(() -> {
+ topic.getBrokerService().getTopicOrderedExecutor().executeOrdered(topicName, SafeRun.safeRun(() -> {
+ synchronized (PersistentStreamingDispatcherSingleActiveConsumer.this) {
+ Consumer currentConsumer = ACTIVE_CONSUMER_UPDATER.get(this);
+ if (currentConsumer != null && !havePendingRead) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}-{}] Scheduling read ", name, currentConsumer);
+ }
+ readMoreEntries(currentConsumer);
+ } else {
+ log.info("[{}-{}] Skipping read as we still havePendingRead {}", name,
+ currentConsumer);
+ }
+ }
+ }));
+ }, withBackoff
+ ? readFailureBackoff.next() : 0, TimeUnit.MILLISECONDS);
+ }
+
+ @Override
+ protected void cancelPendingRead() {
+ if (havePendingRead && streamingEntryReader.cancelReadRequests()) {
+ havePendingRead = false;
+ }
+ }
+
+ /**
+ * {@inheritDoc}
+ */
+ @Override
+ public synchronized void notifyConsumersEndOfTopic() {
+ if (cursor.getNumberOfEntriesInBacklog(false) == 0) {
+ // Topic has been terminated and there are no more entries to read
+ // Notify the consumer only if all the messages were already acknowledged
+ consumers.forEach(Consumer::reachedEndOfTopic);
+ }
+ }
+
+ @Override
+ public String getName() {
+ return name;
+ }
+
+ /**
+ * {@inheritDoc}
+ */
+ @Override
+ public void readEntryComplete(Entry entry, PendingReadEntryRequest ctx) {
+ topic.getBrokerService().getTopicOrderedExecutor().executeOrdered(name, safeRun(() -> {
+ internalReadEntryComplete(entry, ctx);
+ }));
+ }
+
+ public synchronized void internalReadEntryComplete(Entry entry, PendingReadEntryRequest ctx) {
+ if (ctx.isLast()) {
+ readFailureBackoff.reduceToHalf();
+ havePendingRead = false;
+ }
+
+ if (readBatchSize < serviceConfig.getDispatcherMaxReadBatchSize()) {
+ int newReadBatchSize = Math.min(readBatchSize * 2, serviceConfig.getDispatcherMaxReadBatchSize());
+ if (log.isDebugEnabled()) {
+ log.debug("[{}-{}] Increasing read batch size from {} to {}", name,
+ ((Consumer) ctx.ctx).consumerName(), readBatchSize, newReadBatchSize);
+ }
+ readBatchSize = newReadBatchSize;
+ }
+
+ Consumer currentConsumer = ACTIVE_CONSUMER_UPDATER.get(this);
+
+ if (isKeyHashRangeFiltered) {
+ byte[] key = peekStickyKey(entry.getDataBuffer());
+ Consumer consumer = stickyKeyConsumerSelector.select(key);
+ // Skip the entry if it's not for current active consumer.
+ if (consumer == null || currentConsumer != consumer) {
+ entry.release();
+ return;
+ }
+ }
+ Consumer consumer = (Consumer) ctx.ctx;
+ ctx.recycle();
+ if (currentConsumer == null || consumer != currentConsumer) {
+ // Active consumer has changed since the read request has been issued. We need to rewind the cursor and
+ // re-issue the read request for the new consumer
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Rewind because no available consumer found to dispatch message to.", name);
+ }
+
+ entry.release();
+ streamingEntryReader.cancelReadRequests();
+ havePendingRead = false;
+ if (currentConsumer != null) {
+ notifyActiveConsumerChanged(currentConsumer);
+ readMoreEntries(currentConsumer);
+ }
+ } else {
+ EntryBatchSizes batchSizes = EntryBatchSizes.get(1);
+ SendMessageInfo sendMessageInfo = SendMessageInfo.getThreadLocal();
+ EntryBatchIndexesAcks batchIndexesAcks = EntryBatchIndexesAcks.get(1);
+ filterEntriesForConsumer(Lists.newArrayList(entry), batchSizes, sendMessageInfo, batchIndexesAcks,
+ cursor, false);
+ // Update cursor's read position.
+ cursor.seek(((ManagedLedgerImpl) cursor.getManagedLedger())
+ .getNextValidPosition((PositionImpl) entry.getPosition()));
+ dispatchEntriesToConsumer(currentConsumer, Lists.newArrayList(entry), batchSizes,
+ batchIndexesAcks, sendMessageInfo);
+ }
+ }
+
+ @Override
+ protected void readMoreEntries(Consumer consumer) {
+ // consumer can be null when all consumers are disconnected from broker.
+ // so skip reading more entries if currently there is no active consumer.
+ if (null == consumer) {
+ return;
+ }
+
+ if (!havePendingRead && consumer.getAvailablePermits() > 0) {
+ int messagesToRead = calculateNumOfMessageToRead(consumer);
+
+ if (-1 == messagesToRead) {
+ // Skip read as topic/dispatcher has exceed the dispatch rate.
+ return;
+ }
+
+ // Schedule read
+ if (log.isDebugEnabled()) {
+ log.debug("[{}-{}] Schedule read of {} messages", name, consumer, messagesToRead);
+ }
+ havePendingRead = true;
+
+ if (consumer.readCompacted()) {
+ topic.getCompactedTopic().asyncReadEntriesOrWait(cursor, messagesToRead, this, consumer);
+ } else {
+ streamingEntryReader.asyncReadEntries(messagesToRead, serviceConfig.getDispatcherMaxReadSizeBytes(),
+ consumer);
+ }
+ } else {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}-{}] Consumer buffer is full, pause reading", name, consumer);
+ }
+ }
+ }
+
+}
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentSubscription.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentSubscription.java
index c39d4bf..b01e5cb 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentSubscription.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentSubscription.java
@@ -183,18 +183,24 @@ public class PersistentSubscription implements Subscription {
if (dispatcher == null || !dispatcher.isConsumerConnected()) {
Dispatcher previousDispatcher = null;
-
+ boolean useStreamingDispatcher = topic.getBrokerService().getPulsar()
+ .getConfiguration().isStreamingDispatch();
switch (consumer.subType()) {
case Exclusive:
if (dispatcher == null || dispatcher.getType() != SubType.Exclusive) {
previousDispatcher = dispatcher;
- dispatcher = new PersistentDispatcherSingleActiveConsumer(cursor, SubType.Exclusive, 0, topic, this);
+ dispatcher = useStreamingDispatcher ? new PersistentStreamingDispatcherSingleActiveConsumer(cursor,
+ SubType.Exclusive, 0, topic, this) :
+ new PersistentDispatcherSingleActiveConsumer(cursor, SubType.Exclusive, 0,
+ topic, this);
}
break;
case Shared:
if (dispatcher == null || dispatcher.getType() != SubType.Shared) {
previousDispatcher = dispatcher;
- dispatcher = new PersistentDispatcherMultipleConsumers(topic, cursor, this);
+ dispatcher = useStreamingDispatcher ? new PersistentStreamingDispatcherMultipleConsumers(topic,
+ cursor, this) : new PersistentDispatcherMultipleConsumers(topic,
+ cursor, this);
}
break;
case Failover:
@@ -207,8 +213,10 @@ public class PersistentSubscription implements Subscription {
if (dispatcher == null || dispatcher.getType() != SubType.Failover) {
previousDispatcher = dispatcher;
- dispatcher = new PersistentDispatcherSingleActiveConsumer(cursor, SubType.Failover, partitionIndex,
- topic, this);
+ dispatcher = useStreamingDispatcher ? new PersistentStreamingDispatcherSingleActiveConsumer(cursor,
+ SubType.Failover, partitionIndex, topic, this) :
+ new PersistentDispatcherSingleActiveConsumer(cursor, SubType.Failover,
+ partitionIndex, topic, this);
}
break;
case Key_Shared:
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/streamingdispatch/PendingReadEntryRequest.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/streamingdispatch/PendingReadEntryRequest.java
new file mode 100644
index 0000000..7989bbc
--- /dev/null
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/streamingdispatch/PendingReadEntryRequest.java
@@ -0,0 +1,76 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.service.streamingdispatch;
+
+import io.netty.util.Recycler;
+import lombok.Data;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedger;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+
+/**
+ * Representing a pending read request to read an entry from {@link ManagedLedger} carrying necessary context.
+ */
+@Data
+public class PendingReadEntryRequest {
+
+ private static final Recycler<PendingReadEntryRequest> RECYCLER = new Recycler<PendingReadEntryRequest>() {
+ protected PendingReadEntryRequest newObject(Recycler.Handle<PendingReadEntryRequest> handle) {
+ return new PendingReadEntryRequest(handle);
+ }
+ };
+
+ public static PendingReadEntryRequest create(Object ctx, PositionImpl position) {
+ PendingReadEntryRequest pendingReadEntryRequest = RECYCLER.get();
+ pendingReadEntryRequest.ctx = ctx;
+ pendingReadEntryRequest.position = position;
+ pendingReadEntryRequest.retry = 0;
+ pendingReadEntryRequest.isLast = false;
+ return pendingReadEntryRequest;
+ }
+
+ public void recycle() {
+ entry = null;
+ ctx = null;
+ position = null;
+ retry = -1;
+ recyclerHandle.recycle(this);
+ }
+
+ public boolean isLastRequest() {
+ return isLast;
+ }
+
+ private final Recycler.Handle<PendingReadEntryRequest> recyclerHandle;
+
+ // Entry read from ledger
+ public Entry entry;
+
+ // Passed in context that'll be pass to callback
+ public Object ctx;
+
+ // Position of entry to be read
+ public PositionImpl position;
+
+ // Number of time request has been retried.
+ int retry;
+
+ // If request is the last one of a set of requests.
+ boolean isLast;
+}
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/streamingdispatch/StreamingDispatcher.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/streamingdispatch/StreamingDispatcher.java
new file mode 100644
index 0000000..814a381
--- /dev/null
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/streamingdispatch/StreamingDispatcher.java
@@ -0,0 +1,53 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.service.streamingdispatch;
+
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedger;
+import org.apache.pulsar.broker.service.Dispatcher;
+import org.apache.pulsar.common.classification.InterfaceStability;
+
+/**
+ * A {@link Dispatcher} that'll use {@link StreamingEntryReader} to read entries from {@link ManagedLedger}.
+ */
+@InterfaceStability.Unstable
+public interface StreamingDispatcher extends Dispatcher {
+
+ /**
+ * Notify dispatcher issued read entry request has complete.
+ * @param entry Entry read.
+ * @param ctx Context passed in when issuing read entries request.
+ */
+ void readEntryComplete(Entry entry, PendingReadEntryRequest ctx);
+
+ /**
+ * Notify dispatcher can issue next read request.
+ */
+ void canReadMoreEntries(boolean withBackoff);
+
+ /**
+ * Notify dispatcher to inform consumers reached end of topic.
+ */
+ void notifyConsumersEndOfTopic();
+
+ /**
+ * @return Name of the dispatcher.
+ */
+ String getName();
+}
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/streamingdispatch/StreamingEntryReader.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/streamingdispatch/StreamingEntryReader.java
new file mode 100644
index 0000000..24f9bcc
--- /dev/null
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/streamingdispatch/StreamingEntryReader.java
@@ -0,0 +1,338 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.service.streamingdispatch;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import lombok.RequiredArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.WaitingEntryCallBack;
+import org.apache.bookkeeper.mledger.impl.ManagedCursorImpl;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.util.SafeRun;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.transaction.buffer.exceptions.TransactionNotSealedException;
+import org.apache.pulsar.client.impl.Backoff;
+
+/**
+ * Entry reader that fulfill read request by streamline the read instead of reading with micro batch.
+ */
+@Slf4j
+@RequiredArgsConstructor
+public class StreamingEntryReader implements AsyncCallbacks.ReadEntryCallback, WaitingEntryCallBack {
+
+ private final int maxRetry = 3;
+
+ // Queue for read request issued yet waiting for complete from managed ledger.
+ private ConcurrentLinkedQueue<PendingReadEntryRequest> issuedReads = new ConcurrentLinkedQueue<>();
+
+ // Queue for read request that's wait for new entries from managed ledger.
+ private ConcurrentLinkedQueue<PendingReadEntryRequest> pendingReads = new ConcurrentLinkedQueue<>();
+
+ private final ManagedCursorImpl cursor;
+
+ private final StreamingDispatcher dispatcher;
+
+ private final PersistentTopic topic;
+
+ private AtomicInteger currentReadSizeByte = new AtomicInteger(0);
+
+ private volatile State state;
+
+ private static final AtomicReferenceFieldUpdater<StreamingEntryReader, State> STATE_UPDATER =
+ AtomicReferenceFieldUpdater.newUpdater(StreamingEntryReader.class, State.class, "state");
+
+ private volatile int maxReadSizeByte;
+
+ private final Backoff readFailureBackoff = new Backoff(10, TimeUnit.MILLISECONDS,
+ 1, TimeUnit.SECONDS, 0, TimeUnit.MILLISECONDS);
+
+ /**
+ * Read entries in streaming way, that said instead reading with micro batch and send entries to consumer after all
+ * entries in the batch are read from ledger, this method will fire numEntriesToRead requests to managedLedger
+ * and send entry to consumer whenever it is read && all entries before it have been sent to consumer.
+ * @param numEntriesToRead number of entry to read from ledger.
+ * @param maxReadSizeByte maximum byte will be read from ledger.
+ * @param ctx Context send along with read request.
+ */
+ public synchronized void asyncReadEntries(int numEntriesToRead, int maxReadSizeByte, Object ctx) {
+ if (STATE_UPDATER.compareAndSet(this, State.Canceling, State.Canceled)) {
+ internalCancelReadRequests();
+ }
+
+ if (!issuedReads.isEmpty() || !pendingReads.isEmpty()) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] There's pending streaming read not completed yet. Not scheduling next read request.",
+ cursor.getName());
+ }
+ return;
+ }
+
+ PositionImpl nextReadPosition = (PositionImpl) cursor.getReadPosition();
+ ManagedLedgerImpl managedLedger = (ManagedLedgerImpl) cursor.getManagedLedger();
+ // Edge case, when a old ledger is full and new ledger is not yet opened, position can point to next
+ // position of the last confirmed position, but it'll be an invalid position. So try to update the position.
+ if (!managedLedger.isValidPosition(nextReadPosition)) {
+ nextReadPosition = managedLedger.getNextValidPosition(nextReadPosition);
+ }
+ boolean hasEntriesToRead = managedLedger.hasMoreEntries(nextReadPosition);
+ currentReadSizeByte.set(0);
+ STATE_UPDATER.set(this, State.Issued);
+ this.maxReadSizeByte = maxReadSizeByte;
+ for (int c = 0; c < numEntriesToRead; c++) {
+ PendingReadEntryRequest pendingReadEntryRequest = PendingReadEntryRequest.create(ctx, nextReadPosition);
+ // Make sure once we start putting request into pending requests queue, we won't put any following request
+ // to issued requests queue in order to guarantee the order.
+ if (hasEntriesToRead && managedLedger.hasMoreEntries(nextReadPosition)) {
+ issuedReads.offer(pendingReadEntryRequest);
+ } else {
+ pendingReads.offer(pendingReadEntryRequest);
+ }
+ nextReadPosition = managedLedger.getNextValidPosition(nextReadPosition);
+ }
+
+ // Issue requests.
+ for (PendingReadEntryRequest request : issuedReads) {
+ managedLedger.asyncReadEntry(request.position, this, request);
+ }
+
+ if (!pendingReads.isEmpty()) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}} Streaming entry reader has {} pending read requests waiting on new entry."
+ , cursor.getName(), pendingReads.size());
+ }
+ // If new entries are available after we put request into pending queue, fire read.
+ // Else register callback with managed ledger to get notify when new entries are available.
+ if (managedLedger.hasMoreEntries(pendingReads.peek().position)) {
+ entriesAvailable();
+ } else if (managedLedger.isTerminated()) {
+ dispatcher.notifyConsumersEndOfTopic();
+ cleanQueue(pendingReads);
+ if (issuedReads.size() == 0) {
+ dispatcher.canReadMoreEntries(true);
+ }
+ } else {
+ managedLedger.addWaitingEntryCallBack(this);
+ }
+ }
+ }
+
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ // Don't block caller thread, complete read entry with dispatcher dedicated thread.
+ topic.getBrokerService().getTopicOrderedExecutor().executeOrdered(dispatcher.getName(), SafeRun.safeRun(() -> {
+ internalReadEntryComplete(entry, ctx);
+ }));
+ }
+
+ private void internalReadEntryComplete(Entry entry, Object ctx) {
+ PendingReadEntryRequest pendingReadEntryRequest = (PendingReadEntryRequest) ctx;
+ pendingReadEntryRequest.entry = entry;
+ readFailureBackoff.reduceToHalf();
+ Entry readEntry;
+ // If we have entry to send to dispatcher.
+ if (!issuedReads.isEmpty() && issuedReads.peek() == pendingReadEntryRequest) {
+ while (!issuedReads.isEmpty() && issuedReads.peek().entry != null) {
+ PendingReadEntryRequest firstPendingReadEntryRequest = issuedReads.poll();
+ readEntry = firstPendingReadEntryRequest.entry;
+ currentReadSizeByte.addAndGet(readEntry.getLength());
+ //Cancel remaining requests and reset cursor if maxReadSizeByte exceeded.
+ if (currentReadSizeByte.get() > maxReadSizeByte) {
+ cancelReadRequests(readEntry.getPosition());
+ dispatcher.canReadMoreEntries(false);
+ STATE_UPDATER.set(this, State.Completed);
+ return;
+ } else {
+ // All request has been completed, mark returned entry as last.
+ if (issuedReads.isEmpty() && pendingReads.isEmpty()) {
+ firstPendingReadEntryRequest.isLast = true;
+ STATE_UPDATER.set(this, State.Completed);
+ }
+ dispatcher.readEntryComplete(readEntry, firstPendingReadEntryRequest);
+ }
+ }
+ } else if (!issuedReads.isEmpty() && issuedReads.peek().retry > maxRetry) {
+ cancelReadRequests(issuedReads.peek().position);
+ dispatcher.canReadMoreEntries(true);
+ STATE_UPDATER.set(this, State.Completed);
+ }
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ // Don't block caller thread, complete read entry fail with dispatcher dedicated thread.
+ topic.getBrokerService().getTopicOrderedExecutor().executeOrdered(dispatcher.getName(), SafeRun.safeRun(() -> {
+ internalReadEntryFailed(exception, ctx);
+ }));
+ }
+
+ private void internalReadEntryFailed(ManagedLedgerException exception, Object ctx) {
+ PendingReadEntryRequest pendingReadEntryRequest = (PendingReadEntryRequest) ctx;
+ PositionImpl readPosition = pendingReadEntryRequest.position;
+ pendingReadEntryRequest.retry++;
+ long waitTimeMillis = readFailureBackoff.next();
+ if (exception.getCause() instanceof TransactionNotSealedException) {
+ waitTimeMillis = 1;
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Error reading transaction entries : {}, - Retrying to read in {} seconds",
+ cursor.getName(), exception.getMessage(), waitTimeMillis / 1000.0);
+ }
+ } else if (!(exception instanceof ManagedLedgerException.TooManyRequestsException)) {
+ log.error("[{} Error reading entries at {} : {} - Retrying to read in {} seconds", cursor.getName(),
+ readPosition, exception.getMessage(), waitTimeMillis / 1000.0);
+ } else {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Got throttled by bookies while reading at {} : {} - Retrying to read in {} seconds",
+ cursor.getName(), readPosition, exception.getMessage(), waitTimeMillis / 1000.0);
+ }
+ }
+ if (!issuedReads.isEmpty()) {
+ if (issuedReads.peek().retry > maxRetry) {
+ cancelReadRequests(issuedReads.peek().position);
+ dispatcher.canReadMoreEntries(true);
+ STATE_UPDATER.set(this, State.Completed);
+ return;
+ }
+ if (pendingReadEntryRequest.retry <= maxRetry) {
+ retryReadRequest(pendingReadEntryRequest, waitTimeMillis);
+ }
+ }
+ }
+
+ // Cancel all issued and pending request and update cursor's read position.
+ private void cancelReadRequests(Position position) {
+ if (!issuedReads.isEmpty()) {
+ cleanQueue(issuedReads);
+ cursor.seek(position);
+ }
+
+ if (!pendingReads.isEmpty()) {
+ cleanQueue(pendingReads);
+ }
+ }
+
+ private void internalCancelReadRequests() {
+ Position readPosition = !issuedReads.isEmpty() ? issuedReads.peek().position : pendingReads.peek().position;
+ cancelReadRequests(readPosition);
+ }
+
+ public boolean cancelReadRequests() {
+ if (STATE_UPDATER.compareAndSet(this, State.Issued, State.Canceling)) {
+ // Don't block caller thread, complete cancel read with dispatcher dedicated thread.
+ topic.getBrokerService().getTopicOrderedExecutor().executeOrdered(topic.getName(), SafeRun.safeRun(() -> {
+ synchronized (StreamingEntryReader.this) {
+ if (STATE_UPDATER.compareAndSet(this, State.Canceling, State.Canceled)) {
+ internalCancelReadRequests();
+ }
+ }
+ }));
+ return true;
+ }
+ return false;
+ }
+
+ private void cleanQueue(Queue<PendingReadEntryRequest> queue) {
+ while (!queue.isEmpty()) {
+ PendingReadEntryRequest pendingReadEntryRequest = queue.poll();
+ if (pendingReadEntryRequest.entry != null) {
+ pendingReadEntryRequest.entry.release();
+ pendingReadEntryRequest.recycle();
+ }
+ }
+ }
+
+ private void retryReadRequest(PendingReadEntryRequest pendingReadEntryRequest, long delay) {
+ topic.getBrokerService().executor().schedule(() -> {
+ // Jump again into dispatcher dedicated thread
+ topic.getBrokerService().getTopicOrderedExecutor().executeOrdered(dispatcher.getName(),
+ SafeRun.safeRun(() -> {
+ ManagedLedgerImpl managedLedger = (ManagedLedgerImpl) cursor.getManagedLedger();
+ managedLedger.asyncReadEntry(pendingReadEntryRequest.position, this, pendingReadEntryRequest);
+ }));
+ }, delay, TimeUnit.MILLISECONDS);
+ }
+
+ @Override
+ public void entriesAvailable() {
+ topic.getBrokerService().getTopicOrderedExecutor().executeOrdered(dispatcher.getName(), SafeRun.safeRun(() -> {
+ internalEntriesAvailable();
+ }));
+ }
+
+ private synchronized void internalEntriesAvailable() {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}} Streaming entry reader get notification of newly added entries from managed ledger,"
+ + " trying to issued pending read requests.", cursor.getName());
+ }
+ ManagedLedgerImpl managedLedger = (ManagedLedgerImpl) cursor.getManagedLedger();
+ List<PendingReadEntryRequest> newlyIssuedRequests = new ArrayList<>();
+ if (!pendingReads.isEmpty()) {
+ // Edge case, when a old ledger is full and new ledger is not yet opened, position can point to next
+ // position of the last confirmed position, but it'll be an invalid position. So try to update the position.
+ if (!managedLedger.isValidPosition(pendingReads.peek().position)) {
+ pendingReads.peek().position = managedLedger.getNextValidPosition(pendingReads.peek().position);
+ }
+ while (!pendingReads.isEmpty() && managedLedger.hasMoreEntries(pendingReads.peek().position)) {
+ PendingReadEntryRequest next = pendingReads.poll();
+ issuedReads.offer(next);
+ newlyIssuedRequests.add(next);
+ // Need to update the position because when the PendingReadEntryRequest is created, the position could
+ // be all set to managed ledger's last confirmed position.
+ if (!pendingReads.isEmpty()) {
+ pendingReads.peek().position = managedLedger.getNextValidPosition(next.position);
+ }
+ }
+
+ for (PendingReadEntryRequest request : newlyIssuedRequests) {
+ managedLedger.asyncReadEntry(request.position, this, request);
+ }
+
+ if (!pendingReads.isEmpty()) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}} Streaming entry reader has {} pending read requests waiting on new entry."
+ , cursor.getName(), pendingReads.size());
+ }
+ if (managedLedger.hasMoreEntries(pendingReads.peek().position)) {
+ entriesAvailable();
+ } else {
+ managedLedger.addWaitingEntryCallBack(this);
+ }
+ }
+ }
+ }
+
+ protected State getState() {
+ return STATE_UPDATER.get(this);
+ }
+
+ enum State {
+ Issued, Canceling, Canceled, Completed;
+ }
+
+}
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/streamingdispatch/package-info.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/streamingdispatch/package-info.java
new file mode 100644
index 0000000..9a205ed
--- /dev/null
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/streamingdispatch/package-info.java
@@ -0,0 +1,19 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.service.streamingdispatch;
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/PersistentDispatcherFailoverConsumerTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/PersistentDispatcherFailoverConsumerTest.java
index bbf188e..10b96ae 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/PersistentDispatcherFailoverConsumerTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/PersistentDispatcherFailoverConsumerTest.java
@@ -61,6 +61,7 @@ import org.apache.bookkeeper.mledger.ManagedLedger;
import org.apache.bookkeeper.mledger.ManagedLedgerConfig;
import org.apache.bookkeeper.mledger.ManagedLedgerException;
import org.apache.bookkeeper.mledger.ManagedLedgerFactory;
+import org.apache.bookkeeper.mledger.impl.ManagedCursorImpl;
import org.apache.bookkeeper.mledger.impl.PositionImpl;
import org.apache.pulsar.broker.PulsarService;
import org.apache.pulsar.broker.ServiceConfiguration;
@@ -104,7 +105,7 @@ public class PersistentDispatcherFailoverConsumerTest {
private ChannelHandlerContext channelCtx;
private LinkedBlockingQueue<CommandActiveConsumerChange> consumerChanges;
private ZooKeeper mockZk;
- private PulsarService pulsar;
+ protected PulsarService pulsar;
final String successTopicName = "persistent://part-perf/global/perf.t1/ptopic";
final String failTopicName = "persistent://part-perf/global/perf.t1/pfailTopic";
@@ -215,7 +216,7 @@ public class PersistentDispatcherFailoverConsumerTest {
void setupMLAsyncCallbackMocks() {
ledgerMock = mock(ManagedLedger.class);
- cursorMock = mock(ManagedCursor.class);
+ cursorMock = mock(ManagedCursorImpl.class);
doReturn(new ArrayList<Object>()).when(ledgerMock).getCursors();
doReturn("mockCursor").when(cursorMock).getName();
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/PersistentFailoverE2ETest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/PersistentFailoverE2ETest.java
index 83445aa..bb017d4 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/PersistentFailoverE2ETest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/PersistentFailoverE2ETest.java
@@ -270,6 +270,11 @@ public class PersistentFailoverE2ETest extends BrokerTestBase {
@Test
public void testSimpleConsumerEventsWithPartition() throws Exception {
+ // Resetting ActiveConsumerFailoverDelayTimeMillis else if testActiveConsumerFailoverWithDelay get executed
+ // first could cause this test to fail.
+ conf.setActiveConsumerFailoverDelayTimeMillis(0);
+ restartBroker();
+
int numPartitions = 4;
final String topicName = "persistent://prop/use/ns-abc/testSimpleConsumerEventsWithPartition-" + System.nanoTime();
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/PersistentTopicTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/PersistentTopicTest.java
index b955f2e..7341791 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/PersistentTopicTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/PersistentTopicTest.java
@@ -139,7 +139,7 @@ import io.netty.buffer.Unpooled;
/**
*/
public class PersistentTopicTest extends MockedBookKeeperTestCase {
- private PulsarService pulsar;
+ protected PulsarService pulsar;
private BrokerService brokerService;
private ManagedLedgerFactory mlFactoryMock;
private ServerCnx serverCnx;
@@ -1136,7 +1136,7 @@ public class PersistentTopicTest extends MockedBookKeeperTestCase {
@SuppressWarnings("unchecked")
void setupMLAsyncCallbackMocks() {
ledgerMock = mock(ManagedLedger.class);
- cursorMock = mock(ManagedCursor.class);
+ cursorMock = mock(ManagedCursorImpl.class);
final CompletableFuture<Void> closeFuture = new CompletableFuture<>();
doReturn(new ArrayList<Object>()).when(ledgerMock).getCursors();
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherFailoverConsumerStreamingDispatcherTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherFailoverConsumerStreamingDispatcherTest.java
new file mode 100644
index 0000000..66f49c8
--- /dev/null
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherFailoverConsumerStreamingDispatcherTest.java
@@ -0,0 +1,35 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.service.persistent;
+
+import org.apache.pulsar.broker.service.PersistentDispatcherFailoverConsumerTest;
+import org.apache.pulsar.broker.service.streamingdispatch.StreamingDispatcher;
+import org.testng.annotations.BeforeMethod;
+
+/**
+ * PersistentDispatcherFailoverConsumerTest with {@link StreamingDispatcher}
+ */
+public class PersistentDispatcherFailoverConsumerStreamingDispatcherTest extends
+ PersistentDispatcherFailoverConsumerTest {
+ @BeforeMethod
+ public void setup() throws Exception {
+ super.setup();
+ pulsar.getConfiguration().setStreamingDispatch(true);
+ }
+}
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/persistent/PersistentFailoverStreamingDispatcherE2ETest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/persistent/PersistentFailoverStreamingDispatcherE2ETest.java
new file mode 100644
index 0000000..ee56b5c
--- /dev/null
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/persistent/PersistentFailoverStreamingDispatcherE2ETest.java
@@ -0,0 +1,36 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.service.persistent;
+
+import org.apache.pulsar.broker.service.PersistentFailoverE2ETest;
+import org.apache.pulsar.broker.service.streamingdispatch.StreamingDispatcher;
+import org.testng.annotations.BeforeClass;
+
+/**
+ * PersistentFailoverE2ETest with {@link StreamingDispatcher}
+ */
+public class PersistentFailoverStreamingDispatcherE2ETest extends PersistentFailoverE2ETest {
+
+ @BeforeClass
+ @Override
+ protected void setup() throws Exception {
+ conf.setStreamingDispatch(true);
+ super.setup();
+ }
+}
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/persistent/PersistentStreamingDispatcherBlockConsumerTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/persistent/PersistentStreamingDispatcherBlockConsumerTest.java
new file mode 100644
index 0000000..bdaf88e
--- /dev/null
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/persistent/PersistentStreamingDispatcherBlockConsumerTest.java
@@ -0,0 +1,35 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.service.persistent;
+
+import org.apache.pulsar.broker.service.streamingdispatch.StreamingDispatcher;
+import org.apache.pulsar.client.api.DispatcherBlockConsumerTest;
+import org.testng.annotations.BeforeMethod;
+
+/**
+ * DispatcherBlockConsumerTest with {@link StreamingDispatcher}
+ */
+public class PersistentStreamingDispatcherBlockConsumerTest extends DispatcherBlockConsumerTest {
+ @BeforeMethod
+ @Override
+ protected void setup() throws Exception {
+ super.setup();
+ conf.setStreamingDispatch(true);
+ }
+}
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/persistent/PersistentSubscriptionMessageDispatchStreamingDispatcherThrottlingTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/persistent/PersistentSubscriptionMessageDispatchStreamingDispatcherThrottlingTest.java
new file mode 100644
index 0000000..ff170ed
--- /dev/null
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/persistent/PersistentSubscriptionMessageDispatchStreamingDispatcherThrottlingTest.java
@@ -0,0 +1,36 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.service.persistent;
+
+import org.apache.pulsar.broker.service.streamingdispatch.StreamingDispatcher;
+import org.apache.pulsar.client.api.SubscriptionMessageDispatchThrottlingTest;
+import org.testng.annotations.BeforeMethod;
+
+/**
+ * SubscriptionMessageDispatchThrottlingTest with {@link StreamingDispatcher}
+ */
+public class PersistentSubscriptionMessageDispatchStreamingDispatcherThrottlingTest
+ extends SubscriptionMessageDispatchThrottlingTest {
+ @BeforeMethod
+ @Override
+ protected void setup() throws Exception {
+ super.setup();
+ conf.setStreamingDispatch(true);
+ }
+}
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/persistent/PersistentTopicStreamingDispatcherE2ETest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/persistent/PersistentTopicStreamingDispatcherE2ETest.java
new file mode 100644
index 0000000..b1efcf6
--- /dev/null
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/persistent/PersistentTopicStreamingDispatcherE2ETest.java
@@ -0,0 +1,35 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.service.persistent;
+
+import org.apache.pulsar.broker.service.PersistentTopicE2ETest;
+import org.apache.pulsar.broker.service.streamingdispatch.StreamingDispatcher;
+import org.testng.annotations.BeforeMethod;
+
+/**
+ * PersistentTopicE2ETest with {@link StreamingDispatcher}
+ */
+public class PersistentTopicStreamingDispatcherE2ETest extends PersistentTopicE2ETest {
+ @BeforeMethod
+ @Override
+ protected void setup() throws Exception {
+ conf.setStreamingDispatch(true);
+ super.baseSetup();
+ }
+}
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/persistent/PersistentTopicStreamingDispatcherTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/persistent/PersistentTopicStreamingDispatcherTest.java
new file mode 100644
index 0000000..f0057d9
--- /dev/null
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/persistent/PersistentTopicStreamingDispatcherTest.java
@@ -0,0 +1,35 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.service.persistent;
+
+import org.apache.pulsar.broker.service.PersistentTopicTest;
+import org.apache.pulsar.broker.service.streamingdispatch.StreamingDispatcher;
+import org.testng.annotations.BeforeMethod;
+
+/**
+ * PersistentTopicTest with {@link StreamingDispatcher}
+ */
+public class PersistentTopicStreamingDispatcherTest extends PersistentTopicTest {
+
+ @BeforeMethod
+ public void setup() throws Exception {
+ super.setup();
+ pulsar.getConfiguration().setStreamingDispatch(true);
+ }
+}
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/persistent/SimpleProducerConsumerTestStreamingDispatcherTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/persistent/SimpleProducerConsumerTestStreamingDispatcherTest.java
new file mode 100644
index 0000000..1bcdec7
--- /dev/null
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/persistent/SimpleProducerConsumerTestStreamingDispatcherTest.java
@@ -0,0 +1,35 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.service.persistent;
+
+import org.apache.pulsar.broker.service.streamingdispatch.StreamingDispatcher;
+import org.apache.pulsar.client.api.SimpleProducerConsumerTest;
+import org.testng.annotations.BeforeMethod;
+
+/**
+ * SimpleProducerConsumerTest with {@link StreamingDispatcher}
+ */
+public class SimpleProducerConsumerTestStreamingDispatcherTest extends SimpleProducerConsumerTest {
+ @BeforeMethod
+ @Override
+ protected void setup() throws Exception {
+ super.setup();
+ conf.setStreamingDispatch(true);
+ }
+}
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/streamingdispatch/StreamingEntryReaderTests.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/streamingdispatch/StreamingEntryReaderTests.java
new file mode 100644
index 0000000..332f431
--- /dev/null
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/streamingdispatch/StreamingEntryReaderTests.java
@@ -0,0 +1,433 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.service.streamingdispatch;
+
+import com.google.common.base.Charsets;
+import org.apache.bookkeeper.common.util.OrderedExecutor;
+import org.apache.bookkeeper.common.util.OrderedScheduler;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedCursor;
+import org.apache.bookkeeper.mledger.ManagedLedgerConfig;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.EntryImpl;
+import org.apache.bookkeeper.mledger.impl.ManagedCursorImpl;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.test.MockedBookKeeperTestCase;
+import org.apache.pulsar.broker.service.BrokerService;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.mockito.invocation.InvocationOnMock;
+import org.mockito.stubbing.Answer;
+import org.powermock.core.classloader.annotations.PrepareForTest;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Stack;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.awaitility.Awaitility.await;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyBoolean;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.reset;
+import static org.powermock.api.mockito.PowerMockito.doAnswer;
+import static org.powermock.api.mockito.PowerMockito.mock;
+import static org.powermock.api.mockito.PowerMockito.spy;
+import static org.powermock.api.mockito.PowerMockito.when;
+import static org.testng.Assert.assertEquals;
+
+/**
+ * Tests for {@link StreamingEntryReader}
+ */
+@PrepareForTest({ManagedLedgerImpl.class})
+public class StreamingEntryReaderTests extends MockedBookKeeperTestCase {
+
+ private static final Charset Encoding = Charsets.UTF_8;
+ private PersistentTopic mockTopic;
+ private StreamingDispatcher mockDispatcher;
+ private BrokerService mockBrokerService;
+ private ScheduledExecutorService scheduledExecutorService;
+ private OrderedExecutor orderedExecutor;
+ private ManagedLedgerConfig config;
+ private ManagedLedgerImpl ledger;
+ private ManagedCursor cursor;
+
+ @BeforeMethod
+ public void setup() throws Exception {
+ scheduledExecutorService = new ScheduledThreadPoolExecutor(1);
+ orderedExecutor = OrderedScheduler.newSchedulerBuilder()
+ .numThreads(1)
+ .name("StreamingEntryReaderTests").build();
+ mockTopic = mock(PersistentTopic.class);
+ mockBrokerService = mock(BrokerService.class);
+ mockDispatcher = mock(StreamingDispatcher.class);
+ config = new ManagedLedgerConfig().setMaxEntriesPerLedger(10);
+ ledger = spy((ManagedLedgerImpl) factory.open("my_test_ledger", config));
+ cursor = ledger.openCursor("test");
+ when(mockTopic.getBrokerService()).thenReturn(mockBrokerService);
+ when(mockBrokerService.executor()).thenReturn(scheduledExecutorService);
+ when(mockBrokerService.getTopicOrderedExecutor()).thenReturn(orderedExecutor);
+ doAnswer(new Answer<Void>() {
+ @Override
+ public Void answer(InvocationOnMock invocationOnMock) throws Throwable {
+ return null;
+ }
+ }).when(mockDispatcher).notifyConsumersEndOfTopic();
+ }
+
+ @Test
+ public void testCanReadEntryFromMLedgerHappyPath() throws Exception {
+ AtomicInteger entryCount = new AtomicInteger(0);
+ Stack<Position> positions = new Stack<>();
+
+ for (int i = 0; i < 150; i++) {
+ ledger.addEntry(String.format("message-%d", i).getBytes(Encoding));
+ }
+
+ StreamingEntryReader streamingEntryReader =new StreamingEntryReader((ManagedCursorImpl) cursor,
+ mockDispatcher, mockTopic);
+
+ doAnswer((InvocationOnMock invocationOnMock) -> {
+ Entry entry = invocationOnMock.getArgument(0, Entry.class);
+ positions.push(entry.getPosition());
+ assertEquals(new String(entry.getData()), String.format("message-%d", entryCount.getAndIncrement()));
+ cursor.seek(ledger.getNextValidPosition((PositionImpl) entry.getPosition()));
+ return null;
+ }
+ ).when(mockDispatcher).readEntryComplete(any(Entry.class), any(PendingReadEntryRequest.class));
+
+ streamingEntryReader.asyncReadEntries(50, 700, null);
+ await().atMost(500, TimeUnit.MILLISECONDS).until(() -> entryCount.get() == 50);
+ // Check cursor's read position has been properly updated
+ assertEquals(cursor.getReadPosition(), ledger.getNextValidPosition((PositionImpl) positions.peek()));
+ streamingEntryReader.asyncReadEntries(50, 700, null);
+ await().atMost(500, TimeUnit.MILLISECONDS).until(() -> entryCount.get() == 100);
+ assertEquals(cursor.getReadPosition(), ledger.getNextValidPosition((PositionImpl) positions.peek()));
+ streamingEntryReader.asyncReadEntries(50, 700, null);
+ await().atMost(500, TimeUnit.MILLISECONDS).until(() -> entryCount.get() == 150);
+ assertEquals(cursor.getReadPosition(), ledger.getNextValidPosition((PositionImpl) positions.peek()));
+ }
+
+ @Test
+ public void testCanReadEntryFromMLedgerSizeExceededLimit() throws Exception {
+ AtomicBoolean readComplete = new AtomicBoolean(false);
+ Stack<Position> positions = new Stack<>();
+ List<String> entries = new ArrayList<>();
+ int size = "mmmmmmmmmmessage-0".getBytes().length;
+ for (int i = 0; i < 15; i++) {
+ ledger.addEntry(String.format("mmmmmmmmmmessage-%d", i).getBytes(Encoding));
+ }
+
+ StreamingEntryReader streamingEntryReader =
+ new StreamingEntryReader((ManagedCursorImpl) cursor, mockDispatcher, mockTopic);
+
+ doAnswer((InvocationOnMock invocationOnMock) -> {
+ Entry entry = invocationOnMock.getArgument(0, Entry.class);
+ positions.push(entry.getPosition());
+ entries.add(new String(entry.getData()));
+ cursor.seek(ledger.getNextValidPosition((PositionImpl) entry.getPosition()));
+ return null;
+ }
+ ).when(mockDispatcher).readEntryComplete(any(Entry.class), any(PendingReadEntryRequest.class));
+
+ doAnswer((InvocationOnMock invocationOnMock) -> {
+ readComplete.set(true);
+ return null;
+ }
+ ).when(mockDispatcher).canReadMoreEntries(anyBoolean());
+
+ PositionImpl position = ledger.getPositionAfterN(ledger.getFirstPosition(), 3, ManagedLedgerImpl.PositionBound.startExcluded);
+ // Make reading from mledger return out of order.
+ doAnswer(new Answer<Void>() {
+ @Override
+ public Void answer(InvocationOnMock invocationOnMock) throws Throwable {
+ AsyncCallbacks.ReadEntryCallback cb = invocationOnMock.getArgument(1, AsyncCallbacks.ReadEntryCallback.class);
+ executor.schedule(() -> {
+ cb.readEntryComplete(EntryImpl.create(position.getLedgerId(), position.getEntryId(), "mmmmmmmmmmessage-2".getBytes()),
+ invocationOnMock.getArgument(2));
+ }, 200, TimeUnit.MILLISECONDS);
+ return null;
+ }
+ }).when(ledger).asyncReadEntry(eq(position), any(), any());
+
+ // Only 2 entries should be read with this request.
+ streamingEntryReader.asyncReadEntries(6, size * 2 + 1, null);
+ await().atMost(300, TimeUnit.MILLISECONDS).until(() -> readComplete.get());
+ assertEquals(entries.size(), 2);
+ // Assert cursor's read position has been properly updated to the third entry, since we should only read
+ // 2 retries with previous request
+ assertEquals(cursor.getReadPosition(), ledger.getNextValidPosition((PositionImpl) positions.peek()));
+ reset(ledger);
+ readComplete.set(false);
+ streamingEntryReader.asyncReadEntries(6, size * 2 + 1, null);
+ await().atMost(300, TimeUnit.MILLISECONDS).until(() -> readComplete.get());
+ readComplete.set(false);
+ streamingEntryReader.asyncReadEntries(6, size * 2 + 1, null);
+ await().atMost(300, TimeUnit.MILLISECONDS).until(() -> readComplete.get());
+ readComplete.set(false);
+ streamingEntryReader.asyncReadEntries(6, size * 2 + 1, null);
+ await().atMost(300, TimeUnit.MILLISECONDS).until(() -> readComplete.get());
+ assertEquals(cursor.getReadPosition(), ledger.getNextValidPosition((PositionImpl) positions.peek()));
+ assertEquals(entries.size(), 8);
+ for (int i = 0; i < entries.size(); i++) {
+ assertEquals(String.format("mmmmmmmmmmessage-%d", i), entries.get(i));
+ }
+ }
+
+ @Test
+ public void testCanReadEntryFromMLedgerWaitingForNewEntry() throws Exception {
+ AtomicInteger entryCount = new AtomicInteger(0);
+ AtomicBoolean entryProcessed = new AtomicBoolean(false);
+ Stack<Position> positions = new Stack<>();
+ List<String> entries = new ArrayList<>();
+ for (int i = 0; i < 7; i++) {
+ ledger.addEntry(String.format("message-%d", i).getBytes(Encoding));
+ }
+
+ StreamingEntryReader streamingEntryReader =
+ new StreamingEntryReader((ManagedCursorImpl) cursor, mockDispatcher, mockTopic);
+
+ doAnswer((InvocationOnMock invocationOnMock) -> {
+ Entry entry = invocationOnMock.getArgument(0, Entry.class);
+ positions.push(entry.getPosition());
+ entries.add(new String(entry.getData()));
+ entryCount.getAndIncrement();
+ cursor.seek(ledger.getNextValidPosition((PositionImpl) entry.getPosition()));
+ entryProcessed.set(true);
+ return null;
+ }
+ ).when(mockDispatcher).readEntryComplete(any(Entry.class), any(PendingReadEntryRequest.class));
+
+ streamingEntryReader.asyncReadEntries(5, 100, null);
+ await().atMost(300, TimeUnit.MILLISECONDS).until(() -> entryCount.get() == 5);
+ assertEquals(cursor.getReadPosition(), ledger.getNextValidPosition((PositionImpl) positions.peek()));
+ streamingEntryReader.asyncReadEntries(5, 100, null);
+ // We only write 7 entries initially so only 7 entries can be read.
+ await().atMost(300, TimeUnit.MILLISECONDS).until(() -> entryCount.get() == 7);
+ // Add new entry and await for it to be send to reader.
+ entryProcessed.set(false);
+ ledger.addEntry("message-7".getBytes(Encoding));
+ await().atMost(500, TimeUnit.MILLISECONDS).until(() -> entryProcessed.get());
+ assertEquals(entries.size(), 8);
+ entryProcessed.set(false);
+ ledger.addEntry("message-8".getBytes(Encoding));
+ await().atMost(500, TimeUnit.MILLISECONDS).until(() -> entryProcessed.get());
+ assertEquals(entries.size(), 9);
+ entryProcessed.set(false);
+ ledger.addEntry("message-9".getBytes(Encoding));
+ await().atMost(500, TimeUnit.MILLISECONDS).until(() -> entryProcessed.get());
+ assertEquals(entries.size(), 10);
+ assertEquals(cursor.getReadPosition(), ledger.getNextValidPosition((PositionImpl) positions.peek()));
+ for (int i = 0; i < entries.size(); i++) {
+ assertEquals(String.format("message-%d", i), entries.get(i));
+ }
+ }
+
+ @Test
+ public void testCanCancelReadEntryRequestAndResumeReading() throws Exception {
+ Map<Position, String> messages = new HashMap<>();
+ AtomicInteger count = new AtomicInteger(0);
+ Stack<Position> positions = new Stack<>();
+ List<String> entries = new ArrayList<>();
+
+ for (int i = 0; i < 20; i++) {
+ String msg = String.format("message-%d", i);
+ messages.put(ledger.addEntry(msg.getBytes(Encoding)), msg);
+ }
+
+ StreamingEntryReader streamingEntryReader =
+ new StreamingEntryReader((ManagedCursorImpl) cursor, mockDispatcher, mockTopic);
+
+ doAnswer((InvocationOnMock invocationOnMock) -> {
+ Entry entry = invocationOnMock.getArgument(0, Entry.class);
+ positions.push(entry.getPosition());
+ entries.add(new String(entry.getData()));
+ cursor.seek(ledger.getNextValidPosition((PositionImpl) entry.getPosition()));
+ return null;
+ }
+ ).when(mockDispatcher).readEntryComplete(any(Entry.class), any(PendingReadEntryRequest.class));
+
+ // Only return 5 entries
+ doAnswer(new Answer<Void>() {
+ @Override
+ public Void answer(InvocationOnMock invocationOnMock) throws Throwable {
+ AsyncCallbacks.ReadEntryCallback cb = invocationOnMock.getArgument(1, AsyncCallbacks.ReadEntryCallback.class);
+ PositionImpl position = invocationOnMock.getArgument(0, PositionImpl.class);
+ int c = count.getAndIncrement();
+ if (c < 5) {
+ cb.readEntryComplete(EntryImpl.create(position.getLedgerId(), position.getEntryId(),
+ messages.get(position).getBytes()),
+ invocationOnMock.getArgument(2));
+ }
+ return null;
+ }
+ }).when(ledger).asyncReadEntry(any(), any(), any());
+
+ streamingEntryReader.asyncReadEntries(20, 200, null);
+ streamingEntryReader.cancelReadRequests();
+ await().atMost(10000, TimeUnit.MILLISECONDS).until(() -> streamingEntryReader.getState() == StreamingEntryReader.State.Canceled);
+ // Only have 5 entry as we make ledger only return 5 entries and cancel the request.
+ assertEquals(entries.size(), 5);
+ assertEquals(cursor.getReadPosition(), ledger.getNextValidPosition((PositionImpl) positions.peek()));
+ // Clear mock and try to read remaining entries
+ reset(ledger);
+ streamingEntryReader.asyncReadEntries(15, 200, null);
+ streamingEntryReader.cancelReadRequests();
+ await().atMost(10000, TimeUnit.MILLISECONDS).until(() -> streamingEntryReader.getState() == StreamingEntryReader.State.Completed);
+ // Only have 5 entry as we make ledger only return 5 entries and cancel the request.
+ assertEquals(entries.size(), 20);
+ assertEquals(cursor.getReadPosition(), ledger.getNextValidPosition((PositionImpl) positions.peek()));
+ // Make sure message still returned in order
+ for (int i = 0; i < entries.size(); i++) {
+ assertEquals(String.format("message-%d", i), entries.get(i));
+ }
+ }
+
+ @Test
+ public void testCanHandleExceptionAndRetry() throws Exception {
+ Map<Position, String> messages = new HashMap<>();
+ AtomicBoolean entryProcessed = new AtomicBoolean(false);
+ AtomicInteger count = new AtomicInteger(0);
+ Stack<Position> positions = new Stack<>();
+ List<String> entries = new ArrayList<>();
+ for (int i = 0; i < 12; i++) {
+ String msg = String.format("message-%d", i);
+ messages.put(ledger.addEntry(msg.getBytes(Encoding)), msg);
+ }
+
+ StreamingEntryReader streamingEntryReader =
+ new StreamingEntryReader((ManagedCursorImpl) cursor, mockDispatcher, mockTopic);
+
+ doAnswer((InvocationOnMock invocationOnMock) -> {
+ Entry entry = invocationOnMock.getArgument(0, Entry.class);
+ positions.push(entry.getPosition());
+ entries.add(new String(entry.getData()));
+ cursor.seek(ledger.getNextValidPosition((PositionImpl) entry.getPosition()));
+
+ if (entries.size() == 6 || entries.size() == 12) {
+ entryProcessed.set(true);
+ }
+ return null;
+ }
+ ).when(mockDispatcher).readEntryComplete(any(Entry.class), any(PendingReadEntryRequest.class));
+
+ // Make reading from mledger throw exception randomly.
+ doAnswer(new Answer<Void>() {
+ @Override
+ public Void answer(InvocationOnMock invocationOnMock) throws Throwable {
+ AsyncCallbacks.ReadEntryCallback cb = invocationOnMock.getArgument(1, AsyncCallbacks.ReadEntryCallback.class);
+ PositionImpl position = invocationOnMock.getArgument(0, PositionImpl.class);
+ int c = count.getAndIncrement();
+ if (c >= 3 && c < 5 || c >= 9 && c < 11) {
+ cb.readEntryFailed(new ManagedLedgerException.TooManyRequestsException("Fake exception."),
+ invocationOnMock.getArgument(2));
+ } else {
+ cb.readEntryComplete(EntryImpl.create(position.getLedgerId(), position.getEntryId(),
+ messages.get(position).getBytes()),
+ invocationOnMock.getArgument(2));
+ }
+ return null;
+ }
+ }).when(ledger).asyncReadEntry(any(), any(), any());
+
+ streamingEntryReader.asyncReadEntries(6, 100, null);
+ await().atMost(3000, TimeUnit.MILLISECONDS).until(() -> entryProcessed.get());
+ assertEquals(entries.size(), 6);
+ assertEquals(cursor.getReadPosition(), ledger.getNextValidPosition((PositionImpl) positions.peek()));
+ entryProcessed.set(false);
+ streamingEntryReader.asyncReadEntries(6, 100, null);
+ await().atMost(3000, TimeUnit.MILLISECONDS).until(() -> entryProcessed.get());
+ assertEquals(entries.size(), 12);
+ assertEquals(cursor.getReadPosition(), ledger.getNextValidPosition((PositionImpl) positions.peek()));
+ // Make sure message still returned in order
+ for (int i = 0; i < entries.size(); i++) {
+ assertEquals(String.format("message-%d", i), entries.get(i));
+ }
+ }
+
+ @Test
+ public void testWillCancelReadAfterExhaustingRetry() throws Exception {
+ Map<Position, String> messages = new HashMap<>();
+ AtomicInteger count = new AtomicInteger(0);
+ Stack<Position> positions = new Stack<>();
+ List<String> entries = new ArrayList<>();
+ for (int i = 0; i < 12; i++) {
+ String msg = String.format("message-%d", i);
+ messages.put(ledger.addEntry(msg.getBytes(Encoding)), msg);
+ }
+
+ StreamingEntryReader streamingEntryReader =
+ new StreamingEntryReader((ManagedCursorImpl) cursor, mockDispatcher, mockTopic);
+
+ doAnswer((InvocationOnMock invocationOnMock) -> {
+ Entry entry = invocationOnMock.getArgument(0, Entry.class);
+ positions.push(entry.getPosition());
+ cursor.seek(ledger.getNextValidPosition((PositionImpl) entry.getPosition()));
+ entries.add(new String(entry.getData()));
+ return null;
+ }
+ ).when(mockDispatcher).readEntryComplete(any(Entry.class), any(PendingReadEntryRequest.class));
+
+ // Fail after first 3 read.
+ doAnswer(new Answer<Void>() {
+ @Override
+ public Void answer(InvocationOnMock invocationOnMock) throws Throwable {
+ AsyncCallbacks.ReadEntryCallback cb = invocationOnMock.getArgument(1, AsyncCallbacks.ReadEntryCallback.class);
+ PositionImpl position = invocationOnMock.getArgument(0, PositionImpl.class);
+ int c = count.getAndIncrement();
+ if (c >= 3) {
+ cb.readEntryFailed(new ManagedLedgerException.TooManyRequestsException("Fake exception."),
+ invocationOnMock.getArgument(2));
+ } else {
+ cb.readEntryComplete(EntryImpl.create(position.getLedgerId(), position.getEntryId(),
+ messages.get(position).getBytes()),
+ invocationOnMock.getArgument(2));
+ }
+ return null;
+ }
+ }).when(ledger).asyncReadEntry(any(), any(), any());
+
+ streamingEntryReader.asyncReadEntries(5, 100, null);
+ await().atMost(10, TimeUnit.SECONDS).until(() -> streamingEntryReader.getState() == StreamingEntryReader.State.Completed);
+ // Issued 5 read, should only have 3 entries as others were canceled after exhausting retries.
+ assertEquals(entries.size(), 3);
+ for (int i = 0; i < entries.size(); i++) {
+ assertEquals(String.format("message-%d", i), entries.get(i));
+ }
+ reset(ledger);
+ streamingEntryReader.asyncReadEntries(5, 100, null);
+ await().atMost(500, TimeUnit.MILLISECONDS).until(() -> streamingEntryReader.getState() == StreamingEntryReader.State.Completed);
+ assertEquals(entries.size(), 8);
+ for (int i = 0; i < entries.size(); i++) {
+ assertEquals(String.format("message-%d", i), entries.get(i));
+ }
+ }
+
+}
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/DispatcherBlockConsumerTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/DispatcherBlockConsumerTest.java
index f4783f5..604edc3 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/DispatcherBlockConsumerTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/DispatcherBlockConsumerTest.java
@@ -620,6 +620,7 @@ public class DispatcherBlockConsumerTest extends ProducerConsumerBase {
int receivedMsgCount = 0;
for (int i = 0; i < totalProducedMsgs; i++) {
Message<?> msg = consumer.receive(500, TimeUnit.MILLISECONDS);
+ assertTrue(msg != null);
if (!unackMessages.contains(i)) {
consumer.acknowledge(msg);
}
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SubscriptionMessageDispatchThrottlingTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SubscriptionMessageDispatchThrottlingTest.java
index 7ed88f5..2b63411 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SubscriptionMessageDispatchThrottlingTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SubscriptionMessageDispatchThrottlingTest.java
@@ -21,6 +21,7 @@ package org.apache.pulsar.client.api;
import com.google.common.collect.Sets;
import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
import org.apache.pulsar.broker.service.Dispatcher;
@@ -35,6 +36,8 @@ import org.slf4j.LoggerFactory;
import org.testng.Assert;
import org.testng.annotations.Test;
+import static org.awaitility.Awaitility.await;
+
public class SubscriptionMessageDispatchThrottlingTest extends MessageDispatchThrottlingTest {
private static final Logger log = LoggerFactory.getLogger(SubscriptionMessageDispatchThrottlingTest.class);
@@ -189,17 +192,13 @@ public class SubscriptionMessageDispatchThrottlingTest extends MessageDispatchTh
Assert.assertTrue(isMessageRateUpdate);
Assert.assertEquals(admin.namespaces().getSubscriptionDispatchRate(namespace), dispatchRate);
- long start = System.currentTimeMillis();
// Asynchronously produce messages
for (int i = 0; i < numProducedMessages; i++) {
final String message = "my-message-" + i;
producer.send(message.getBytes());
}
- latch.await();
+ await().atMost(2500, TimeUnit.MILLISECONDS).until(() -> latch.getCount() == 0);
Assert.assertEquals(totalReceived.get(), numProducedMessages);
- long end = System.currentTimeMillis();
-
- Assert.assertTrue((end - start) >= 2000);
consumer.close();
producer.close();
diff --git a/site2/docs/reference-configuration.md b/site2/docs/reference-configuration.md
index 67ed800..28cf79e 100644
--- a/site2/docs/reference-configuration.md
+++ b/site2/docs/reference-configuration.md
@@ -500,6 +500,7 @@ The value of 0 disables message-byte dispatch-throttling.|0|
|dispatcherMinReadBatchSize|The minimum number of entries to read from BookKeeper. By default, it is 1 entry. When there is an error occurred on reading entries from bookkeeper, the broker will backoff the batch size to this minimum number.|1|
|dispatcherMaxRoundRobinBatchSize|The maximum number of entries to dispatch for a shared subscription. By default, it is 20 entries.|20|
| preciseDispatcherFlowControl | Precise dispathcer flow control according to history message number of each entry. | false |
+| streamingDispatch | Whether to use streaming read dispatcher. It can be useful when there's a huge backlog to drain and instead of read with micro batch we can streamline the read from bookkeeper to make the most of consumer capacity till we hit bookkeeper read limit or consumer process limit, then we can use consumer flow control to tune the speed. This feature is currently in preview and can be changed in subsequent release. | false |
| maxConcurrentLookupRequest | Maximum number of concurrent lookup request that the broker allows to throttle heavy incoming lookup traffic. | 50000 |
| maxConcurrentTopicLoadRequest | Maximum number of concurrent topic loading request that the broker allows to control the number of zk-operations. | 5000 |
| maxConcurrentNonPersistentMessagePerConnection | Maximum number of concurrent non-persistent message that can be processed per connection. | 1000 |
[pulsar] 41/46: [pulsar-broker] Allow broker to discover and
unblock stuck subscription (#9789)
Posted by eo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit a1e1388111f5dafa6aa4f0fe0858033af76924cf
Author: Rajan Dhabalia <rd...@apache.org>
AuthorDate: Tue Mar 9 22:53:48 2021 -0800
[pulsar-broker] Allow broker to discover and unblock stuck subscription (#9789)
We have been frequently seeing issue where subscription gets stuck on different topics and broker is not dispatching messages though consumer has available-permits and no pending reads (example #9788). It can happen due to regression bug or unknown issue when expiry runs.. one of the workarounds is manually unload the topic and reload it which is not feasible if this happens frequently to many topics. Or broker should have the capability to discover such stuck subscriptions and unblock them.
Below example shows that:
subscription has available-permit>0, there is no pending reads, cursor's read-position is not moving forward and that builds the backlog until we unload the topic. It happens frequently due to unknown reason:
```
STATS-INTERNAL:
"sub1" : {
"markDeletePosition" : "11111111:15520",
"readPosition" : "11111111:15521",
"waitingReadOp" : false,
"pendingReadOps" : 0,
"messagesConsumedCounter" : 115521,
"cursorLedger" : 585099247,
"cursorLedgerLastEntry" : 597,
"individuallyDeletedMessages" : "[]",
"lastLedgerSwitchTimestamp" : "2021-02-25T19:55:50.357Z",
"state" : "Open",
"numberOfEntriesSinceFirstNotAckedMessage" : 1,
"totalNonContiguousDeletedMessagesRange" : 0,
STATS:
"sub1" : {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"msgRateRedeliver" : 0.0,
"msgBacklog" : 30350,
"blockedSubscriptionOnUnackedMsgs" : false,
"msgDelayed" : 0,
"unackedMessages" : 0,
"type" : "Shared",
"msgRateExpired" : 0.0,
"consumers" : [ {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"msgRateRedeliver" : 0.0,
"consumerName" : "C1",
"availablePermits" : 723,
"unackedMessages" : 0,
"blockedConsumerOnUnackedMsgs" : false,
"metadata" : { },
"connectedSince" : "2021-02-25T19:55:50.358285Z",
```
Add capability in broker to periodically check if subscription is stuck and unblock it if needed. This check is controlled by flag and for initial release it can be disabled by default (and we can enable by default in later release)
It helps broker to handle stuck subscription and logs the message for later debugging.
---
conf/broker.conf | 3 +
deployment/terraform-ansible/templates/broker.conf | 3 +
.../apache/bookkeeper/mledger/ManagedCursor.java | 6 ++
.../bookkeeper/mledger/impl/ManagedCursorImpl.java | 12 ++++
.../bookkeeper/mledger/impl/PositionImpl.java | 1 -
.../mledger/impl/ManagedCursorContainerTest.java | 5 ++
.../bookkeeper/mledger/impl/ManagedCursorTest.java | 61 +++++++++++++++-
.../apache/pulsar/broker/ServiceConfiguration.java | 7 ++
.../apache/pulsar/broker/service/Dispatcher.java | 7 ++
.../PersistentDispatcherMultipleConsumers.java | 15 ++++
.../PersistentDispatcherSingleActiveConsumer.java | 15 ++++
.../service/persistent/PersistentSubscription.java | 4 ++
.../broker/service/persistent/PersistentTopic.java | 4 ++
.../service/persistent/PersistentTopicTest.java | 83 +++++++++++++++++++++-
site2/docs/reference-configuration.md | 1 +
15 files changed, 224 insertions(+), 3 deletions(-)
diff --git a/conf/broker.conf b/conf/broker.conf
index f286961..92a2c1d 100644
--- a/conf/broker.conf
+++ b/conf/broker.conf
@@ -259,6 +259,9 @@ maxUnackedMessagesPerBroker=0
# limit/2 messages
maxUnackedMessagesPerSubscriptionOnBrokerBlocked=0.16
+# Broker periodically checks if subscription is stuck and unblock if flag is enabled. (Default is disabled)
+unblockStuckSubscriptionEnabled=false
+
# Tick time to schedule task that checks topic publish rate limiting across all topics
# Reducing to lower value can give more accuracy while throttling publish but
# it uses more CPU to perform frequent check. (Disable publish throttling with value 0)
diff --git a/deployment/terraform-ansible/templates/broker.conf b/deployment/terraform-ansible/templates/broker.conf
index 638c889..91dc813 100644
--- a/deployment/terraform-ansible/templates/broker.conf
+++ b/deployment/terraform-ansible/templates/broker.conf
@@ -227,6 +227,9 @@ maxUnackedMessagesPerBroker=0
# limit/2 messages
maxUnackedMessagesPerSubscriptionOnBrokerBlocked=0.16
+# Broker periodically checks if subscription is stuck and unblock if flag is enabled. (Default is disabled)
+unblockStuckSubscriptionEnabled=false
+
# Tick time to schedule task that checks topic publish rate limiting across all topics
# Reducing to lower value can give more accuracy while throttling publish but
# it uses more CPU to perform frequent check. (Disable publish throttling with value 0)
diff --git a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedCursor.java b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedCursor.java
index a9724de..320a030 100644
--- a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedCursor.java
+++ b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedCursor.java
@@ -673,4 +673,10 @@ public interface ManagedCursor {
*/
ManagedCursorMXBean getStats();
+ /**
+ * Checks if read position changed since this method was called last time.
+ *
+ * @return if read position changed
+ */
+ boolean checkAndUpdateReadPositionChanged();
}
diff --git a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java
index f732a75..f114f6d 100644
--- a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java
+++ b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java
@@ -119,6 +119,8 @@ public class ManagedCursorImpl implements ManagedCursor {
protected static final AtomicReferenceFieldUpdater<ManagedCursorImpl, PositionImpl> READ_POSITION_UPDATER =
AtomicReferenceFieldUpdater.newUpdater(ManagedCursorImpl.class, PositionImpl.class, "readPosition");
protected volatile PositionImpl readPosition;
+ // keeps sample of last read-position for validation and monitoring if read-position is not moving forward.
+ protected volatile PositionImpl statsLastReadPosition;
protected static final AtomicReferenceFieldUpdater<ManagedCursorImpl, MarkDeleteEntry> LAST_MARK_DELETE_ENTRY_UPDATER =
AtomicReferenceFieldUpdater.newUpdater(ManagedCursorImpl.class, MarkDeleteEntry.class, "lastMarkDeleteEntry");
@@ -2969,5 +2971,15 @@ public class ManagedCursorImpl implements ManagedCursor {
return Math.min(maxEntriesBasedOnSize, maxEntries);
}
+ @Override
+ public boolean checkAndUpdateReadPositionChanged() {
+ PositionImpl lastEntry = ledger.lastConfirmedEntry;
+ boolean isReadPositionOnTail = lastEntry == null || readPosition == null
+ || !(lastEntry.compareTo(readPosition) > 0);
+ boolean isReadPositionChanged = readPosition != null && !readPosition.equals(statsLastReadPosition);
+ statsLastReadPosition = readPosition;
+ return isReadPositionOnTail || isReadPositionChanged;
+ }
+
private static final Logger log = LoggerFactory.getLogger(ManagedCursorImpl.class);
}
diff --git a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/PositionImpl.java b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/PositionImpl.java
index ef445f9..e4a3060 100644
--- a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/PositionImpl.java
+++ b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/PositionImpl.java
@@ -125,7 +125,6 @@ public class PositionImpl implements Position, Comparable<PositionImpl> {
PositionImpl other = (PositionImpl) obj;
return ledgerId == other.ledgerId && entryId == other.entryId;
}
-
return false;
}
diff --git a/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedCursorContainerTest.java b/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedCursorContainerTest.java
index d0b0b2c..4801aab 100644
--- a/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedCursorContainerTest.java
+++ b/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedCursorContainerTest.java
@@ -357,6 +357,11 @@ public class ManagedCursorContainerTest {
throws InterruptedException, ManagedLedgerException {
return null;
}
+
+ @Override
+ public boolean checkAndUpdateReadPositionChanged() {
+ return false;
+ }
}
@Test
diff --git a/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedCursorTest.java b/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedCursorTest.java
index 4f6e1a0..f34cc8c 100644
--- a/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedCursorTest.java
+++ b/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedCursorTest.java
@@ -3414,12 +3414,71 @@ public class ManagedCursorTest extends MockedBookKeeperTestCase {
} finally {
factory2.shutdown();
- }
+ }
});
factory1.shutdown();
dirtyFactory.shutdown();
}
+ @Test
+ public void testCursorCheckReadPositionChanged() throws Exception {
+ ManagedLedger ledger = factory.open("my_test_ledger", new ManagedLedgerConfig());
+ ManagedCursor c1 = ledger.openCursor("c1");
+
+ // check empty ledger
+ assertTrue(c1.checkAndUpdateReadPositionChanged());
+ assertTrue(c1.checkAndUpdateReadPositionChanged());
+
+ ledger.addEntry("dummy-entry-1".getBytes(Encoding));
+ ledger.addEntry("dummy-entry-1".getBytes(Encoding));
+ ledger.addEntry("dummy-entry-1".getBytes(Encoding));
+ ledger.addEntry("dummy-entry-1".getBytes(Encoding));
+
+ // read-position has not been moved
+ assertFalse(c1.checkAndUpdateReadPositionChanged());
+
+ List<Entry> entries = c1.readEntries(2);
+ entries.forEach(e -> {
+ try {
+ c1.markDelete(e.getPosition());
+ e.release();
+ } catch (Exception e1) {
+ // Ok
+ }
+ });
+
+ // read-position is moved
+ assertTrue(c1.checkAndUpdateReadPositionChanged());
+ // read-position has not been moved since last read
+ assertFalse(c1.checkAndUpdateReadPositionChanged());
+
+ c1.close();
+ ledger.close();
+
+ ledger = factory.open("my_test_ledger", new ManagedLedgerConfig());
+ // recover cursor
+ ManagedCursor c2 = ledger.openCursor("c1");
+ assertTrue(c2.checkAndUpdateReadPositionChanged());
+ assertFalse(c2.checkAndUpdateReadPositionChanged());
+
+ entries = c2.readEntries(2);
+ entries.forEach(e -> {
+ try {
+ c2.markDelete(e.getPosition());
+ e.release();
+ } catch (Exception e1) {
+ // Ok
+ }
+ });
+
+ assertTrue(c2.checkAndUpdateReadPositionChanged());
+ // returns true because read-position is on tail
+ assertTrue(c2.checkAndUpdateReadPositionChanged());
+ assertTrue(c2.checkAndUpdateReadPositionChanged());
+
+ ledger.close();
+ }
+
private static final Logger log = LoggerFactory.getLogger(ManagedCursorTest.class);
}
diff --git a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
index 5979e63..2614706 100644
--- a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
+++ b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
@@ -509,6 +509,13 @@ public class ServiceConfiguration implements PulsarConfiguration {
@FieldContext(
category = CATEGORY_POLICIES,
dynamic = true,
+ doc = "Broker periodically checks if subscription is stuck and unblock if flag is enabled. "
+ + "(Default is disabled)"
+ )
+ private boolean unblockStuckSubscriptionEnabled = false;
+ @FieldContext(
+ category = CATEGORY_POLICIES,
+ dynamic = true,
doc = "Tick time to schedule task that checks topic publish rate limiting across all topics "
+ "Reducing to lower value can give more accuracy while throttling publish but "
+ "it uses more CPU to perform frequent check. (Disable publish throttling with value 0)"
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Dispatcher.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Dispatcher.java
index 43b02d1..e30d54c 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Dispatcher.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Dispatcher.java
@@ -125,4 +125,11 @@ public interface Dispatcher {
// No-op
}
+ /**
+ * Checks if dispatcher is stuck and unblocks the dispatch if needed.
+ */
+ default boolean checkAndUnblockIfStuck() {
+ return false;
+ }
+
}
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherMultipleConsumers.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherMultipleConsumers.java
index 934873a..f3c6d94 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherMultipleConsumers.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherMultipleConsumers.java
@@ -865,6 +865,21 @@ public class PersistentDispatcherMultipleConsumers extends AbstractDispatcherMul
this.messagesToRedeliver.add(ledgerId, entryId);
}
+ @Override
+ public boolean checkAndUnblockIfStuck() {
+ if (cursor.checkAndUpdateReadPositionChanged()) {
+ return false;
+ }
+ // consider dispatch is stuck if : dispatcher has backlog, available-permits and there is no pending read
+ if (totalAvailablePermits > 0 && !havePendingReplayRead && !havePendingRead
+ && cursor.getNumberOfEntriesInBacklog(false) > 0) {
+ log.warn("{}-{} Dispatcher is stuck and unblocking by issuing reads", topic.getName(), name);
+ readMoreEntries();
+ return true;
+ }
+ return false;
+ }
+
public PersistentTopic getTopic() {
return topic;
}
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherSingleActiveConsumer.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherSingleActiveConsumer.java
index 48c82d1..468f413 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherSingleActiveConsumer.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherSingleActiveConsumer.java
@@ -615,5 +615,20 @@ public class PersistentDispatcherSingleActiveConsumer extends AbstractDispatcher
this.messagesToRedeliver.add(ledgerId, entryId);
}
+ public boolean checkAndUnblockIfStuck() {
+ if (cursor.checkAndUpdateReadPositionChanged()) {
+ return false;
+ }
+ Consumer consumer = ACTIVE_CONSUMER_UPDATER.get(this);
+ int totalAvailablePermits = consumer.getAvailablePermits();
+ // consider dispatch is stuck if : dispatcher has backlog, available-permits and there is no pending read
+ if (totalAvailablePermits > 0 && !havePendingRead && cursor.getNumberOfEntriesInBacklog(false) > 0) {
+ log.warn("{}-{} Dispatcher is stuck and unblocking by issuing reads", topic.getName(), name);
+ readMoreEntries(consumer);
+ return true;
+ }
+ return false;
+ }
+
private static final Logger log = LoggerFactory.getLogger(PersistentDispatcherSingleActiveConsumer.class);
}
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentSubscription.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentSubscription.java
index b01e5cb..57281cb 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentSubscription.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentSubscription.java
@@ -1089,5 +1089,9 @@ public class PersistentSubscription implements Subscription {
return this.pendingAckHandle.checkIsCanDeleteConsumerPendingAck(position);
}
+ public boolean checkAndUnblockIfStuck() {
+ return dispatcher.checkAndUnblockIfStuck();
+ }
+
private static final Logger log = LoggerFactory.getLogger(PersistentSubscription.class);
}
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
index d87707a..3ba512a 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
@@ -1534,6 +1534,10 @@ public class PersistentTopic extends AbstractTopic implements Topic, AddEntryCal
topicStatsHelper.aggMsgRateOut += subMsgRateOut;
topicStatsHelper.aggMsgThroughputOut += subMsgThroughputOut;
nsStats.msgBacklog += subscription.getNumberOfEntriesInBacklog(false);
+ // check stuck subscription
+ if (brokerService.getPulsar().getConfig().isUnblockStuckSubscriptionEnabled()) {
+ subscription.checkAndUnblockIfStuck();
+ }
} catch (Exception e) {
log.error("Got exception when creating consumer stats for subscription {}: {}", subscriptionName,
e.getMessage(), e);
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/persistent/PersistentTopicTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/persistent/PersistentTopicTest.java
index f4d11b8..547baef 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/persistent/PersistentTopicTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/persistent/PersistentTopicTest.java
@@ -21,14 +21,32 @@ package org.apache.pulsar.broker.service.persistent;
import static org.mockito.Mockito.spy;
import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.doNothing;
+import static org.mockito.Mockito.mock;
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertNotNull;
+import static org.testng.Assert.assertNull;
import java.util.UUID;
import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
import com.google.common.collect.Sets;
import org.apache.pulsar.broker.service.BrokerTestBase;
import org.apache.pulsar.common.policies.data.Policies;
import org.awaitility.Awaitility;
+import org.apache.pulsar.client.api.Consumer;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.MessageRoutingMode;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.SubscriptionType;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.naming.TopicName;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;
@@ -72,4 +90,67 @@ public class PersistentTopicTest extends BrokerTestBase {
persistentTopic.onPoliciesUpdate(policies);
verify(persistentTopic, times(1)).checkReplicationAndRetryOnFailure();
}
-}
\ No newline at end of file
+
+ /**
+ * Test validates if topic's dispatcher is stuck then broker can doscover and unblock it.
+ *
+ * @throws Exception
+ */
+ @Test
+ public void testUnblockStuckSubscription() throws Exception {
+ final String topicName = "persistent://prop/ns-abc/stuckSubscriptionTopic";
+ final String sharedSubName = "shared";
+ final String failoverSubName = "failOver";
+
+ Consumer<String> consumer1 = pulsarClient.newConsumer(Schema.STRING).topic(topicName)
+ .subscriptionType(SubscriptionType.Shared).subscriptionName(sharedSubName).subscribe();
+ Consumer<String> consumer2 = pulsarClient.newConsumer(Schema.STRING).topic(topicName)
+ .subscriptionType(SubscriptionType.Failover).subscriptionName(failoverSubName).subscribe();
+ Producer<String> producer = pulsarClient.newProducer(Schema.STRING).topic(topicName).create();
+
+ PersistentTopic topic = (PersistentTopic) pulsar.getBrokerService().getTopicReference(topicName).get();
+ PersistentSubscription sharedSub = topic.getSubscription(sharedSubName);
+ PersistentSubscription failOverSub = topic.getSubscription(failoverSubName);
+
+ PersistentDispatcherMultipleConsumers sharedDispatcher = (PersistentDispatcherMultipleConsumers) sharedSub
+ .getDispatcher();
+ PersistentDispatcherSingleActiveConsumer failOverDispatcher = (PersistentDispatcherSingleActiveConsumer) failOverSub
+ .getDispatcher();
+
+ // build backlog
+ consumer1.close();
+ consumer2.close();
+
+ // block sub to read messages
+ sharedDispatcher.havePendingRead = true;
+ failOverDispatcher.havePendingRead = true;
+
+ producer.newMessage().value("test").eventTime(5).send();
+ producer.newMessage().value("test").eventTime(5).send();
+
+ consumer1 = pulsarClient.newConsumer(Schema.STRING).topic(topicName).subscriptionType(SubscriptionType.Shared)
+ .subscriptionName(sharedSubName).subscribe();
+ consumer2 = pulsarClient.newConsumer(Schema.STRING).topic(topicName).subscriptionType(SubscriptionType.Failover)
+ .subscriptionName(failoverSubName).subscribe();
+ Message<String> msg = consumer1.receive(2, TimeUnit.SECONDS);
+ assertNull(msg);
+ msg = consumer2.receive(2, TimeUnit.SECONDS);
+ assertNull(msg);
+
+ // allow reads but dispatchers are still blocked
+ sharedDispatcher.havePendingRead = false;
+ failOverDispatcher.havePendingRead = false;
+
+ // run task to unblock stuck dispatcher: first iteration sets the lastReadPosition and next iteration will
+ // unblock the dispatcher read because read-position has not been moved since last iteration.
+ sharedSub.checkAndUnblockIfStuck();
+ failOverDispatcher.checkAndUnblockIfStuck();
+ assertTrue(sharedSub.checkAndUnblockIfStuck());
+ assertTrue(failOverDispatcher.checkAndUnblockIfStuck());
+
+ msg = consumer1.receive(5, TimeUnit.SECONDS);
+ assertNotNull(msg);
+ msg = consumer2.receive(5, TimeUnit.SECONDS);
+ assertNotNull(msg);
+ }
+}
diff --git a/site2/docs/reference-configuration.md b/site2/docs/reference-configuration.md
index 28cf79e..eb4076c 100644
--- a/site2/docs/reference-configuration.md
+++ b/site2/docs/reference-configuration.md
@@ -480,6 +480,7 @@ The [`pulsar-client`](reference-cli-tools.md#pulsar-client) CLI tool can be used
|maxUnackedMessagesPerSubscription| The same as above, except per subscription rather than per consumer. |200000|
| maxUnackedMessagesPerBroker | Maximum number of unacknowledged messages allowed per broker. Once this limit reaches, the broker stops dispatching messages to all shared subscriptions which has a higher number of unacknowledged messages until subscriptions start acknowledging messages back and unacknowledged messages count reaches to limit/2. When the value is set to 0, unacknowledged message limit check is disabled and broker does not block dispatchers. | 0 |
| maxUnackedMessagesPerSubscriptionOnBrokerBlocked | Once the broker reaches maxUnackedMessagesPerBroker limit, it blocks subscriptions which have higher unacknowledged messages than this percentage limit and subscription does not receive any new messages until that subscription acknowledges messages back. | 0.16 |
+| unblockStuckSubscriptionEnabled|Broker periodically checks if subscription is stuck and unblock if flag is enabled.|false|
|maxNumPartitionsPerPartitionedTopic|Max number of partitions per partitioned topic. Use 0 or negative number to disable the check|0|
|zookeeperSessionExpiredPolicy|There are two policies when ZooKeeper session expired happens, "shutdown" and "reconnect". If it is set to "shutdown" policy, when ZooKeeper session expired happens, the broker is shutdown. If it is set to "reconnect" policy, the broker tries to reconnect to ZooKeeper server and re-register metadata to ZooKeeper. Note: the "reconnect" policy is an experiment feature.|shutdown|
| topicPublisherThrottlingTickTimeMillis | Tick time to schedule task that checks topic publish rate limiting across all topics. A lower value can improve accuracy while throttling publish but it uses more CPU to perform frequent check. (Disable publish throttling with value 0) | 10|
[pulsar] 36/46: PIP-85 Add Schema Information to Message in Java
Client API (#10476)
Posted by eo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 4acd720eb2aea434a9839c6e738943a313e87b20
Author: Enrico Olivelli <eo...@gmail.com>
AuthorDate: Wed May 12 12:33:42 2021 +0200
PIP-85 Add Schema Information to Message in Java Client API (#10476)
This is the implementation for PIP-85
https://docs.google.com/document/d/1VWi5LHP44V31nP4bCui9d5RXwH6xc_phrUes6tvNguk/
- introduce `Optional<Schema<?>> Message.getReaderSchema()` - new Public API
- introduce `Optional<Object>` SchemaReader.getNativeSchema()` - new Public API
- introduce `Schema<?> atSchemaVersion(schemaVersion)` to `AbstractSchema` (private API)
- make `KeyValueSchema` extend AbstractSchema (private API)
- rename MessageImpl and TopicMessageImpl `getSchema` to `getSchemaInternal`
The schema returned by Message.getReaderSchema has these properties:
- getSchemaInfo returns the actual schema used to decode the Message
- getNativeSchema returns the original native (ie. Avro) Schema for the correct version
- it can be used for "decoding" payloads
- it cannot be used for "encoding" payloads
(cherry picked from commit 42241107dcf1f74a27a02297bc47901b3684b3cf)
---
.../apache/pulsar/client/api/InterceptorsTest.java | 4 +-
.../apache/pulsar/client/api/SimpleSchemaTest.java | 79 +++++++++-
.../java/org/apache/pulsar/schema/SchemaTest.java | 174 +++++++++++++++++++--
.../java/org/apache/pulsar/client/api/Message.java | 13 ++
.../pulsar/client/api/schema/SchemaReader.java | 10 ++
.../org/apache/pulsar/client/impl/MessageImpl.java | 35 ++++-
.../apache/pulsar/client/impl/ProducerImpl.java | 12 +-
.../pulsar/client/impl/TopicMessageImpl.java | 10 +-
.../pulsar/client/impl/schema/AbstractSchema.java | 18 +++
.../client/impl/schema/AbstractStructSchema.java | 84 +++++++++-
.../client/impl/schema/AutoConsumeSchema.java | 10 ++
.../pulsar/client/impl/schema/KeyValueSchema.java | 36 ++++-
.../impl/schema/generic/GenericAvroReader.java | 6 +
.../generic/GenericProtobufNativeReader.java | 6 +
.../impl/schema/generic/GenericSchemaImpl.java | 1 +
.../schema/reader/AbstractMultiVersionReader.java | 8 +-
.../client/impl/schema/reader/AvroReader.java | 10 ++
.../pulsar/functions/source/PulsarSource.java | 4 +-
.../pulsar/functions/source/PulsarSourceTest.java | 5 +-
19 files changed, 485 insertions(+), 40 deletions(-)
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/InterceptorsTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/InterceptorsTest.java
index d5fe225..5c42251 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/InterceptorsTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/InterceptorsTest.java
@@ -106,14 +106,14 @@ public class InterceptorsTest extends ProducerConsumerBase {
@Override
public boolean eligible(Message message) {
return SchemaType.STRING.equals(
- ((MessageImpl)message).getSchema().getSchemaInfo().getType());
+ ((MessageImpl)message).getSchemaInternal().getSchemaInfo().getType());
}
};
BaseInterceptor interceptor3 = new BaseInterceptor("int3") {
@Override
public boolean eligible(Message message) {
return SchemaType.INT32.equals(
- ((MessageImpl)message).getSchema().getSchemaInfo().getType());
+ ((MessageImpl)message).getSchemaInternal().getSchemaInfo().getType());
}
};
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleSchemaTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleSchemaTest.java
index 802d059..0e1d756 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleSchemaTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleSchemaTest.java
@@ -25,6 +25,7 @@ import lombok.NoArgsConstructor;
import static java.nio.charset.StandardCharsets.UTF_8;
import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertNotEquals;
import static org.testng.Assert.assertNotNull;
import static org.testng.Assert.assertTrue;
import static org.testng.Assert.fail;
@@ -33,7 +34,7 @@ import lombok.extern.slf4j.Slf4j;
import org.apache.avro.reflect.ReflectData;
import org.apache.avro.Schema.Parser;
import org.apache.pulsar.client.impl.MessageImpl;
-import org.apache.pulsar.common.protocol.schema.BytesSchemaVersion;
+import org.apache.pulsar.client.impl.schema.KeyValueSchema;
import org.apache.pulsar.common.schema.LongSchemaVersion;
import org.apache.pulsar.client.admin.PulsarAdminException;
import org.apache.pulsar.client.api.PulsarClientException.IncompatibleSchemaException;
@@ -79,7 +80,10 @@ public class SimpleSchemaTest extends ProducerConsumerBase {
@DataProvider(name = "schemaValidationModes")
public static Object[][] schemaValidationModes() {
- return new Object[][] { { true }, { false } };
+ return new Object[][] {
+ { true },
+ { false }
+ };
}
@DataProvider(name = "topicDomain")
@@ -530,8 +534,11 @@ public class SimpleSchemaTest extends ProducerConsumerBase {
assertEquals(data.getValue().getField("i"), i);
MessageImpl impl = (MessageImpl) data;
- org.apache.avro.Schema avroSchema = (org.apache.avro.Schema) impl.getSchema().getNativeSchema().get();
+ org.apache.avro.Schema avroSchema = (org.apache.avro.Schema) impl.getSchemaInternal().getNativeSchema().get();
assertNotNull(avroSchema);
+
+ org.apache.avro.Schema avroSchema2 = (org.apache.avro.Schema) data.getReaderSchema().get().getNativeSchema().get();
+ assertNotNull(avroSchema2);
}
}
@@ -616,6 +623,9 @@ public class SimpleSchemaTest extends ProducerConsumerBase {
assertEquals(data.getValue().getKey().getField("i"), i * 100);
assertEquals(data.getValue().getValue().getField("i"), i * 1000);
c1.acknowledge(data);
+ KeyValueSchema keyValueSchema = (KeyValueSchema) data.getReaderSchema().get();
+ assertNotNull(keyValueSchema.getKeySchema());
+ assertNotNull(keyValueSchema.getValueSchema());
}
// verify c2
@@ -625,6 +635,9 @@ public class SimpleSchemaTest extends ProducerConsumerBase {
assertEquals(data.getValue().getKey().i, i * 100);
assertEquals(data.getValue().getValue().i, i * 1000);
c2.acknowledge(data);
+ KeyValueSchema keyValueSchema = (KeyValueSchema) data.getReaderSchema().get();
+ assertNotNull(keyValueSchema.getKeySchema());
+ assertNotNull(keyValueSchema.getValueSchema());
}
// verify c3
@@ -634,6 +647,9 @@ public class SimpleSchemaTest extends ProducerConsumerBase {
assertEquals(data.getValue().getKey().getField("i"), i * 100);
assertEquals(data.getValue().getValue().i, i * 1000);
c3.acknowledge(data);
+ KeyValueSchema keyValueSchema = (KeyValueSchema) data.getReaderSchema().get();
+ assertNotNull(keyValueSchema.getKeySchema());
+ assertNotNull(keyValueSchema.getValueSchema());
}
// verify c4
@@ -784,12 +800,18 @@ public class SimpleSchemaTest extends ProducerConsumerBase {
// verify c0
for (int i = 0; i < numMessages; i++) {
Message<GenericRecord> wrapper = c0.receive();
- log.info("schema version {}", BytesSchemaVersion.of(wrapper.getSchemaVersion()));
KeyValue<GenericRecord, GenericRecord> data = (KeyValue<GenericRecord, GenericRecord>) wrapper.getValue().getNativeObject();
assertNotNull(wrapper.getSchemaVersion());
assertEquals(data.getKey().getField("i"), i * 100);
assertEquals(data.getValue().getField("i"), i * 1000);
c0.acknowledge(wrapper);
+ KeyValueSchema keyValueSchema = (KeyValueSchema) wrapper.getReaderSchema().get();
+ assertNotNull(keyValueSchema.getKeySchema());
+ assertNotNull(keyValueSchema.getValueSchema());
+ assertTrue(keyValueSchema.getKeySchema().getSchemaInfo().getSchemaDefinition().contains("V1Data"));
+ assertTrue(keyValueSchema.getValueSchema().getSchemaInfo().getSchemaDefinition().contains("V1Data"));
+ assertTrue(keyValueSchema.getKeySchema().getNativeSchema().isPresent());
+ assertTrue(keyValueSchema.getValueSchema().getNativeSchema().isPresent());
}
@@ -813,15 +835,20 @@ public class SimpleSchemaTest extends ProducerConsumerBase {
// verify c0
for (int i = 0; i < numMessages; i++) {
Message<GenericRecord> wrapper = c0.receive();
- log.info("schema version {}", BytesSchemaVersion.of(wrapper.getSchemaVersion()));
KeyValue<GenericRecord, GenericRecord> data = (KeyValue<GenericRecord, GenericRecord>) wrapper.getValue().getNativeObject();
assertNotNull(wrapper.getSchemaVersion());
assertEquals(data.getKey().getField("i"), i * 100);
assertEquals(data.getValue().getField("i"), i * 1000);
assertEquals(data.getKey().getField("j"), i);
assertEquals(data.getValue().getField("j"), i * 20);
+ KeyValueSchema keyValueSchema = (KeyValueSchema) wrapper.getReaderSchema().get();
+ assertNotNull(keyValueSchema.getKeySchema());
+ assertNotNull(keyValueSchema.getValueSchema());
+ assertTrue(keyValueSchema.getKeySchema().getSchemaInfo().getSchemaDefinition().contains("V2Data"));
+ assertTrue(keyValueSchema.getValueSchema().getSchemaInfo().getSchemaDefinition().contains("V2Data"));
+ assertTrue(keyValueSchema.getKeySchema().getNativeSchema().isPresent());
+ assertTrue(keyValueSchema.getValueSchema().getNativeSchema().isPresent());
}
-
}
}
}
@@ -852,6 +879,46 @@ public class SimpleSchemaTest extends ProducerConsumerBase {
Assert.assertTrue(binaryLookupService.getSchema(TopicName.get(topic), ByteBuffer.allocate(8).putLong(1).array()).get().isPresent());
}
+ @Test
+ public void testGetNativeSchemaWithAutoConsumeWithMultiVersion() throws Exception {
+ final String topic = "persistent://my-property/my-ns/testGetSchemaWithMultiVersion";
+
+ @Cleanup
+ Consumer<?> consumer = pulsarClient.newConsumer(Schema.AUTO_CONSUME())
+ .subscriptionName("test")
+ .topic(topic)
+ .subscribe();
+
+ @Cleanup
+ Producer<V1Data> v1DataProducer = pulsarClient.newProducer(Schema.AVRO(V1Data.class))
+ .topic(topic)
+ .create();
+
+ @Cleanup
+ Producer<V2Data> v2DataProducer = pulsarClient.newProducer(Schema.AVRO(V2Data.class))
+ .topic(topic)
+ .create();
+
+ Assert.assertEquals(admin.schemas().getAllSchemas(topic).size(), 2);
+
+ v1DataProducer.send(new V1Data());
+ v2DataProducer.send(new V2Data());
+
+ Message<?> messageV1 = consumer.receive();
+ Schema<?> schemaV1 = messageV1.getReaderSchema().get();
+ Message<?> messageV2 = consumer.receive();
+ Schema<?> schemaV2 = messageV2.getReaderSchema().get();
+ log.info("schemaV1 {} {}", schemaV1.getSchemaInfo(), schemaV1.getNativeSchema());
+ log.info("schemaV2 {} {}", schemaV2.getSchemaInfo(), schemaV2.getNativeSchema());
+ assertTrue(schemaV1.getSchemaInfo().getSchemaDefinition().contains("V1Data"));
+ assertTrue(schemaV2.getSchemaInfo().getSchemaDefinition().contains("V2Data"));
+ org.apache.avro.Schema avroSchemaV1 = (org.apache.avro.Schema) schemaV1.getNativeSchema().get();
+ org.apache.avro.Schema avroSchemaV2 = (org.apache.avro.Schema) schemaV2.getNativeSchema().get();
+ assertNotEquals(avroSchemaV1.toString(false), avroSchemaV2.toString(false));
+ assertTrue(avroSchemaV1.toString(false).contains("V1Data"));
+ assertTrue(avroSchemaV2.toString(false).contains("V2Data"));
+ }
+
@Test(dataProvider = "topicDomain")
public void testAutoCreatedSchema(String domain) throws Exception {
final String topic1 = domain + "my-property/my-ns/testAutoCreatedSchema-1";
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/schema/SchemaTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/schema/SchemaTest.java
index 7672cce..9542a36 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/schema/SchemaTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/schema/SchemaTest.java
@@ -47,6 +47,7 @@ import org.apache.pulsar.client.api.Producer;
import org.apache.pulsar.client.api.Schema;
import org.apache.pulsar.client.api.schema.GenericRecord;
import org.apache.pulsar.client.api.schema.SchemaDefinition;
+import org.apache.pulsar.client.impl.schema.KeyValueSchema;
import org.apache.pulsar.common.naming.TopicDomain;
import org.apache.pulsar.common.naming.TopicName;
import org.apache.pulsar.common.policies.data.ClusterData;
@@ -121,10 +122,11 @@ public class SchemaTest extends MockedPulsarServiceBaseTest {
(false).withSupportSchemaVersioning(true).
withPojo(Schemas.PersonTwo.class).build()).getSchemaInfo());
- admin.schemas().createSchema(fqtnTwo, Schema.AVRO(
+ Schema<Schemas.PersonTwo> personTwoSchema = Schema.AVRO(
SchemaDefinition.<Schemas.PersonTwo>builder().withAlwaysAllowNull
(false).withSupportSchemaVersioning(true).
- withPojo(Schemas.PersonTwo.class).build()).getSchemaInfo());
+ withPojo(Schemas.PersonTwo.class).build());
+ admin.schemas().createSchema(fqtnTwo, personTwoSchema.getSchemaInfo());
Producer<Schemas.PersonTwo> producer = pulsarClient.newProducer(Schema.AVRO(
SchemaDefinition.<Schemas.PersonTwo>builder().withAlwaysAllowNull
@@ -146,18 +148,144 @@ public class SchemaTest extends MockedPulsarServiceBaseTest {
.topic(fqtnOne, fqtnTwo)
.subscribe();
+ Consumer<GenericRecord> consumer2 = pulsarClient.newConsumer(Schema.AUTO_CONSUME())
+ .subscriptionName("test2")
+ .topic(fqtnOne, fqtnTwo)
+ .subscribe();
+
producer.send(personTwo);
Schemas.PersonTwo personConsume = consumer.receive().getValue();
assertEquals("Tom", personConsume.getName());
assertEquals(1, personConsume.getId());
+ Message<Schemas.PersonTwo> message = consumer.receive();
+ Schemas.PersonTwo personConsume = message.getValue();
+ assertEquals(personConsume.getName(), "Tom");
+ assertEquals(personConsume.getId(), 1);
+ Schema<?> schema = message.getReaderSchema().get();
+ log.info("the-schema {}", schema);
+ assertEquals(personTwoSchema.getSchemaInfo(), schema.getSchemaInfo());
+ org.apache.avro.Schema nativeSchema = (org.apache.avro.Schema) schema.getNativeSchema().get();
+ log.info("nativeSchema-schema {}", nativeSchema);
+ assertNotNull(nativeSchema);
+
+ // verify that with AUTO_CONSUME we can access the original schema
+ // and the Native AVRO schema
+ Message<?> message2 = consumer2.receive();
+ Schema<?> schema2 = message2.getReaderSchema().get();
+ log.info("the-schema {}", schema2);
+ assertEquals(personTwoSchema.getSchemaInfo(), schema2.getSchemaInfo());
+ org.apache.avro.Schema nativeSchema2 = (org.apache.avro.Schema) schema.getNativeSchema().get();
+ log.info("nativeSchema-schema {}", nativeSchema2);
+ assertNotNull(nativeSchema2);
+
+ producer.close();
+ consumer.close();
+ }
+
+ @Test
+ public void testMultiTopicSetSchemaProviderWithKeyValue() throws Exception {
+ final String tenant = PUBLIC_TENANT;
+ final String namespace = "test-namespace-" + randomName(16);
+ final String topicOne = "test-multi-version-schema-one";
+ final String topicTwo = "test-multi-version-schema-two";
+ final String fqtnOne = TopicName.get(
+ TopicDomain.persistent.value(),
+ tenant,
+ namespace,
+ topicOne
+ ).toString();
+
+ final String fqtnTwo = TopicName.get(
+ TopicDomain.persistent.value(),
+ tenant,
+ namespace,
+ topicTwo
+ ).toString();
+
+
+ admin.namespaces().createNamespace(
+ tenant + "/" + namespace,
+ Sets.newHashSet(CLUSTER_NAME)
+ );
+
+ admin.topics().createPartitionedTopic(fqtnOne, 3);
+ admin.topics().createPartitionedTopic(fqtnTwo, 3);
+
+ Schema<Schemas.PersonOne> schemaOne = Schema.AVRO(
+ SchemaDefinition.<Schemas.PersonOne>builder().withAlwaysAllowNull
+ (false).withSupportSchemaVersioning(true).
+ withPojo(Schemas.PersonOne.class).build());
+ admin.schemas().createSchema(fqtnOne, Schema.KeyValue(Schema.STRING, schemaOne).getSchemaInfo());
+
+ Schema<Schemas.PersonTwo> schemaTwo = Schema.AVRO(
+ SchemaDefinition.<Schemas.PersonTwo>builder().withAlwaysAllowNull
+ (false).withSupportSchemaVersioning(true).
+ withPojo(Schemas.PersonTwo.class).build());
+ admin.schemas().createSchema(fqtnOne, Schema.KeyValue(Schema.STRING, schemaTwo).getSchemaInfo());
+
+ Schema<Schemas.PersonTwo> personTwoSchema = Schema.AVRO(
+ SchemaDefinition.<Schemas.PersonTwo>builder().withAlwaysAllowNull
+ (false).withSupportSchemaVersioning(true).
+ withPojo(Schemas.PersonTwo.class).build());
+ admin.schemas().createSchema(fqtnTwo, Schema.KeyValue(Schema.STRING, schemaTwo).getSchemaInfo());
+
+ Producer<KeyValue<String, Schemas.PersonTwo>> producer = pulsarClient.newProducer(Schema.KeyValue(Schema.STRING, Schema.AVRO(
+ SchemaDefinition.<Schemas.PersonTwo>builder().withAlwaysAllowNull
+ (false).withSupportSchemaVersioning(true).
+ withPojo(Schemas.PersonTwo.class).build())))
+ .topic(fqtnOne)
+ .create();
+
+ Schemas.PersonTwo personTwo = new Schemas.PersonTwo();
+ personTwo.setId(1);
+ personTwo.setName("Tom");
+
+
+ Consumer<KeyValue<String, Schemas.PersonTwo>> consumer = pulsarClient.newConsumer(Schema.KeyValue(Schema.STRING, Schema.AVRO(
+ SchemaDefinition.<Schemas.PersonTwo>builder().withAlwaysAllowNull
+ (false).withSupportSchemaVersioning(true).
+ withPojo(Schemas.PersonTwo.class).build())))
+ .subscriptionName("test")
+ .topic(fqtnOne, fqtnTwo)
+ .subscribe();
+
+ Consumer<GenericRecord> consumer2 = pulsarClient.newConsumer(Schema.AUTO_CONSUME())
+ .subscriptionName("test2")
+ .topic(fqtnOne, fqtnTwo)
+ .subscribe();
+
+ producer.send(new KeyValue<>("foo", personTwo));
+
+ Message<KeyValue<String, Schemas.PersonTwo>> message = consumer.receive();
+ assertEquals("foo", message.getValue().getKey());
+ Schemas.PersonTwo personConsume = message.getValue().getValue();
+ assertEquals(personConsume.getName(), "Tom");
+ assertEquals(personConsume.getId(), 1);
+ KeyValueSchema schema = (KeyValueSchema) message.getReaderSchema().get();
+ log.info("the-schema {}", schema);
+ assertEquals(personTwoSchema.getSchemaInfo(), schema.getValueSchema().getSchemaInfo());
+ org.apache.avro.Schema nativeSchema = (org.apache.avro.Schema) schema.getValueSchema().getNativeSchema().get();
+ log.info("nativeSchema-schema {}", nativeSchema);
+ assertNotNull(nativeSchema);
+
+ // verify that with AUTO_CONSUME we can access the original schema
+ // and the Native AVRO schema
+ Message<?> message2 = consumer2.receive();
+ KeyValueSchema schema2 = (KeyValueSchema) message2.getReaderSchema().get();
+ log.info("the-schema {}", schema2);
+ assertEquals(personTwoSchema.getSchemaInfo(), schema2.getValueSchema().getSchemaInfo());
+ org.apache.avro.Schema nativeSchema2 = (org.apache.avro.Schema) schema.getValueSchema().getNativeSchema().get();
+ log.info("nativeSchema-schema {}", nativeSchema2);
+ assertNotNull(nativeSchema2);
+
producer.close();
consumer.close();
}
@Test
- public void testBytesSchemaDeserialize() throws Exception {
+ public void testJSONSchemaDeserialize() throws Exception {
final String tenant = PUBLIC_TENANT;
final String namespace = "test-namespace-" + randomName(16);
final String topicName = "test-bytes-schema";
@@ -203,6 +331,12 @@ public class SchemaTest extends MockedPulsarServiceBaseTest {
assertEquals(message.getValue().getField("address").getClass(),
message1.getValue().getAddress().getClass());
+ Schema<?> schema = message.getReaderSchema().get();
+ Schema<?> schema1 = message1.getReaderSchema().get();
+ log.info("schema {}", schema);
+ log.info("schema1 {}", schema1);
+ assertEquals(schema.getSchemaInfo(), schema1.getSchemaInfo());
+
producer.close();
consumer.close();
consumer1.close();
@@ -246,12 +380,14 @@ public class SchemaTest extends MockedPulsarServiceBaseTest {
producer.send("foo");
Message<String> message = consumer.receive();
- Message<GenericRecord> message3 = consumer2.receive();
+ Message<GenericRecord> message2 = consumer2.receive();
+ assertEquals(SchemaType.STRING, message.getReaderSchema().get().getSchemaInfo().getType());
+ assertEquals(SchemaType.STRING, message2.getReaderSchema().get().getSchemaInfo().getType());
assertEquals("foo", message.getValue());
- assertEquals(message3.getValue().getClass().getName(), "org.apache.pulsar.client.impl.schema.GenericObjectWrapper");
- assertEquals(SchemaType.STRING, message3.getValue().getSchemaType());
- assertEquals("foo", message3.getValue().getNativeObject());
+ assertEquals(message2.getValue().getClass().getName(), "org.apache.pulsar.client.impl.schema.GenericObjectWrapper");
+ assertEquals(SchemaType.STRING, message2.getValue().getSchemaType());
+ assertEquals("foo", message2.getValue().getNativeObject());
producer.close();
consumer.close();
@@ -296,12 +432,22 @@ public class SchemaTest extends MockedPulsarServiceBaseTest {
producer.send("foo".getBytes(StandardCharsets.UTF_8));
Message<byte[]> message = consumer.receive();
- Message<GenericRecord> message3 = consumer2.receive();
+ Message<GenericRecord> message2 = consumer2.receive();
+ if (schema == SchemaType.BYTES) {
+ assertEquals(schema, message.getReaderSchema().get().getSchemaInfo().getType());
+ assertEquals(schema, message2.getReaderSchema().get().getSchemaInfo().getType());
+ } else if (schema == SchemaType.NONE) {
+ // schema NONE is always reported as BYTES
+ assertEquals(SchemaType.BYTES, message.getReaderSchema().get().getSchemaInfo().getType());
+ assertEquals(SchemaType.BYTES, message2.getReaderSchema().get().getSchemaInfo().getType());
+ } else {
+ fail();
+ }
assertEquals("foo".getBytes(StandardCharsets.UTF_8), message.getValue());
- assertEquals(message3.getValue().getClass().getName(), "org.apache.pulsar.client.impl.schema.GenericObjectWrapper");
- assertEquals(SchemaType.BYTES, message3.getValue().getSchemaType());
- assertEquals("foo".getBytes(StandardCharsets.UTF_8), message3.getValue().getNativeObject());
+ assertEquals(message2.getValue().getClass().getName(), "org.apache.pulsar.client.impl.schema.GenericObjectWrapper");
+ assertEquals(SchemaType.BYTES, message2.getValue().getSchemaType());
+ assertEquals("foo".getBytes(StandardCharsets.UTF_8), message2.getValue().getNativeObject());
producer.close();
consumer.close();
@@ -420,6 +566,12 @@ public class SchemaTest extends MockedPulsarServiceBaseTest {
Message<GenericRecord> message2 = consumer2.receive();
assertEquals(message.getValue(), message2.getValue().getNativeObject());
+ Schema<?> schema = message.getReaderSchema().get();
+ Schema<?> schemaFromGenericRecord = message.getReaderSchema().get();
+ KeyValueSchema keyValueSchema = (KeyValueSchema) schema;
+ KeyValueSchema keyValueSchemaFromGenericRecord = (KeyValueSchema) schemaFromGenericRecord;
+ assertEquals(keyValueSchema.getSchemaInfo(), keyValueSchemaFromGenericRecord.getSchemaInfo());
+
if (keyValueEncodingType == KeyValueEncodingType.SEPARATED) {
// with "SEPARATED encoding the routing key is the key of the KeyValue
assertNotNull(message.getKeyBytes());
diff --git a/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Message.java b/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Message.java
index 324c2ba..d41054e 100644
--- a/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Message.java
+++ b/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Message.java
@@ -202,6 +202,19 @@ public interface Message<T> {
byte[] getSchemaVersion();
/**
+ * Get the schema associated to the message.
+ * Please note that this schema is usually equal to the Schema you passed
+ * during the construction of the Consumer or the Reader.
+ * But if you are consuming the topic using the GenericObject interface
+ * this method will return the schema associated with the message.
+ * @return The schema used to decode the payload of message.
+ * @see Schema#AUTO_CONSUME()
+ */
+ default Optional<Schema<?>> getReaderSchema() {
+ return Optional.empty();
+ }
+
+ /**
* Check whether the message is replicated from other cluster.
*
* @since 2.4.0
diff --git a/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/schema/SchemaReader.java b/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/schema/SchemaReader.java
index dc18c4e..c5561bf 100644
--- a/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/schema/SchemaReader.java
+++ b/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/schema/SchemaReader.java
@@ -19,6 +19,8 @@
package org.apache.pulsar.client.api.schema;
import java.io.InputStream;
+import java.util.Optional;
+
import org.apache.pulsar.common.classification.InterfaceAudience;
import org.apache.pulsar.common.classification.InterfaceStability;
@@ -86,4 +88,12 @@ public interface SchemaReader<T> {
*/
default void setSchemaInfoProvider(SchemaInfoProvider schemaInfoProvider) {
}
+
+ /**
+ * Returns the underling Schema if possible
+ * @return the schema, or an empty Optional if it is not possible to access it
+ */
+ default Optional<Object> getNativeSchema() {
+ return Optional.empty();
+ }
}
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageImpl.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageImpl.java
index a148677..1eb7d3f 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageImpl.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageImpl.java
@@ -41,6 +41,7 @@ import java.util.stream.Collectors;
import org.apache.pulsar.client.api.Message;
import org.apache.pulsar.client.api.MessageId;
import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.impl.schema.AbstractSchema;
import org.apache.pulsar.client.impl.schema.AutoConsumeSchema;
import org.apache.pulsar.client.impl.schema.KeyValueSchema;
import org.apache.pulsar.common.protocol.Commands;
@@ -267,11 +268,38 @@ public class MessageImpl<T> implements Message<T> {
}
}
- public Schema getSchema() {
+ @Override
+ public int size() {
+ if (msgMetadata.isNullValue()) {
+ return 0;
+ }
+ return payload.readableBytes();
+ }
+
+ public Schema<T> getSchemaInternal() {
return this.schema;
}
@Override
+ public Optional<Schema<?>> getReaderSchema() {
+ ensureSchemaIsLoaded();
+ if (schema == null) {
+ return Optional.empty();
+ }
+ if (schema instanceof AutoConsumeSchema) {
+ byte[] schemaVersion = getSchemaVersion();
+ return Optional.of(((AutoConsumeSchema) schema)
+ .atSchemaVersion(schemaVersion));
+ } else if (schema instanceof AbstractSchema) {
+ byte[] schemaVersion = getSchemaVersion();
+ return Optional.of(((AbstractSchema) schema)
+ .atSchemaVersion(schemaVersion));
+ } else {
+ return Optional.of(schema);
+ }
+ }
+
+ @Override
public byte[] getSchemaVersion() {
if (msgMetadataBuilder != null && msgMetadataBuilder.hasSchemaVersion()) {
return msgMetadataBuilder.getSchemaVersion().toByteArray();
@@ -280,10 +308,13 @@ public class MessageImpl<T> implements Message<T> {
}
}
- private SchemaInfo getSchemaInfo() {
+ private void ensureSchemaIsLoaded() {
if (schema instanceof AutoConsumeSchema) {
((AutoConsumeSchema) schema).fetchSchemaIfNeeded();
}
+ }
+ private SchemaInfo getSchemaInfo() {
+ ensureSchemaIsLoaded();
return schema.getSchemaInfo();
}
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java
index 786af84..96e4042 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java
@@ -569,7 +569,7 @@ public class ProducerImpl<T> extends ProducerBase<T> implements TimerTask, Conne
private boolean populateMessageSchema(MessageImpl msg, SendCallback callback) {
MessageMetadata.Builder msgMetadataBuilder = msg.getMessageBuilder();
- if (msg.getSchema() == schema) {
+ if (msg.getSchemaInternal() == schema) {
schemaVersion.ifPresent(v -> msgMetadataBuilder.setSchemaVersion(ByteString.copyFrom(v)));
msg.setSchemaState(MessageImpl.SchemaState.Ready);
return true;
@@ -581,7 +581,7 @@ public class ProducerImpl<T> extends ProducerBase<T> implements TimerTask, Conne
completeCallbackAndReleaseSemaphore(callback, e);
return false;
}
- SchemaHash schemaHash = SchemaHash.of(msg.getSchema());
+ SchemaHash schemaHash = SchemaHash.of(msg.getSchemaInternal());
byte[] schemaVersion = schemaCache.get(schemaHash);
if (schemaVersion != null) {
msgMetadataBuilder.setSchemaVersion(ByteString.copyFrom(schemaVersion));
@@ -591,7 +591,7 @@ public class ProducerImpl<T> extends ProducerBase<T> implements TimerTask, Conne
}
private boolean rePopulateMessageSchema(MessageImpl msg) {
- SchemaHash schemaHash = SchemaHash.of(msg.getSchema());
+ SchemaHash schemaHash = SchemaHash.of(msg.getSchemaInternal());
byte[] schemaVersion = schemaCache.get(schemaHash);
if (schemaVersion == null) {
return false;
@@ -605,7 +605,7 @@ public class ProducerImpl<T> extends ProducerBase<T> implements TimerTask, Conne
if (!changeToRegisteringSchemaState()) {
return;
}
- SchemaInfo schemaInfo = Optional.ofNullable(msg.getSchema())
+ SchemaInfo schemaInfo = Optional.ofNullable(msg.getSchemaInternal())
.map(Schema::getSchemaInfo)
.filter(si -> si.getType().getValue() > 0)
.orElse(Schema.BYTES.getSchemaInfo());
@@ -619,7 +619,7 @@ public class ProducerImpl<T> extends ProducerBase<T> implements TimerTask, Conne
}
} else {
log.warn("[{}] [{}] GetOrCreateSchema succeed", topic, producerName);
- SchemaHash schemaHash = SchemaHash.of(msg.getSchema());
+ SchemaHash schemaHash = SchemaHash.of(msg.getSchemaInternal());
schemaCache.putIfAbsent(schemaHash, v);
msg.getMessageBuilder().setSchemaVersion(ByteString.copyFrom(v));
msg.setSchemaState(MessageImpl.SchemaState.Ready);
@@ -1831,4 +1831,4 @@ public class ProducerImpl<T> extends ProducerBase<T> implements TimerTask, Conne
}
private static final Logger log = LoggerFactory.getLogger(ProducerImpl.class);
-}
\ No newline at end of file
+}
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/TopicMessageImpl.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/TopicMessageImpl.java
index 00869f1..784c56b 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/TopicMessageImpl.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/TopicMessageImpl.java
@@ -173,11 +173,17 @@ public class TopicMessageImpl<T> implements Message<T> {
return msg;
}
- public Schema<T> getSchema() {
+ public Schema<T> getSchemaInternal() {
if (this.msg instanceof MessageImpl) {
MessageImpl message = (MessageImpl) this.msg;
- return message.getSchema();
+ return message.getSchemaInternal();
}
return null;
}
+
+ @Override
+ public Optional<Schema<?>> getReaderSchema() {
+ return msg.getReaderSchema();
+ }
+
}
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AbstractSchema.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AbstractSchema.java
index 255a491..4328e7b 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AbstractSchema.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AbstractSchema.java
@@ -22,6 +22,8 @@ import io.netty.buffer.ByteBuf;
import org.apache.pulsar.client.api.Schema;
import org.apache.pulsar.client.api.SchemaSerializationException;
+import java.util.Objects;
+
public abstract class AbstractSchema<T> implements Schema<T> {
/**
@@ -66,4 +68,20 @@ public abstract class AbstractSchema<T> implements Schema<T> {
public Schema<T> clone() {
return this;
}
+
+ /**
+ * Return an instance of this schema at the given version.
+ * @param schemaVersion the version
+ * @return the schema at that specific version
+ * @throws SchemaSerializationException in case of unknown schema version
+ * @throws NullPointerException in case of null schemaVersion
+ */
+ public Schema<?> atSchemaVersion(byte[] schemaVersion) throws SchemaSerializationException {
+ Objects.requireNonNull(schemaVersion);
+ if (!supportSchemaVersioning()) {
+ return this;
+ } else {
+ throw new SchemaSerializationException("Not implemented for " + this.getClass());
+ }
+ }
}
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AbstractStructSchema.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AbstractStructSchema.java
index 015ec4d..c4444e7 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AbstractStructSchema.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AbstractStructSchema.java
@@ -20,14 +20,21 @@ package org.apache.pulsar.client.impl.schema;
import io.netty.buffer.ByteBuf;
import io.netty.buffer.ByteBufInputStream;
-
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.SchemaSerializationException;
import org.apache.pulsar.client.api.schema.SchemaInfoProvider;
import org.apache.pulsar.client.api.schema.SchemaReader;
import org.apache.pulsar.client.api.schema.SchemaWriter;
+import org.apache.pulsar.client.impl.schema.reader.AbstractMultiVersionReader;
+import org.apache.pulsar.common.protocol.schema.BytesSchemaVersion;
import org.apache.pulsar.common.schema.SchemaInfo;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.concurrent.ExecutionException;
+
/**
* minimal abstract StructSchema
*/
@@ -80,6 +87,81 @@ public abstract class AbstractStructSchema<T> extends AbstractSchema<T> {
if (reader != null) {
this.reader.setSchemaInfoProvider(schemaInfoProvider);
}
+ this.schemaInfoProvider = schemaInfoProvider;
+ }
+
+ @Override
+ public Schema<T> atSchemaVersion(byte[] schemaVersion) throws SchemaSerializationException {
+ Objects.requireNonNull(schemaVersion);
+ if (schemaInfoProvider == null) {
+ // this schema is not downloaded from the registry
+ return this;
+ }
+ try {
+ SchemaInfo schemaInfo = schemaInfoProvider.getSchemaByVersion(schemaVersion).get();
+ if (schemaInfo == null) {
+ throw new SchemaSerializationException("Unknown version "+ BytesSchemaVersion.of(schemaVersion));
+ }
+ return getAbstractStructSchemaAtVersion(schemaVersion, schemaInfo);
+ } catch (ExecutionException err) {
+ throw new SchemaSerializationException(err.getCause());
+ } catch (InterruptedException err) {
+ Thread.currentThread().interrupt();
+ throw new SchemaSerializationException(err);
+ }
+ }
+
+ private static class WrappedVersionedSchema<T> extends AbstractStructSchema<T> {
+ private final byte[] schemaVersion;
+ private final AbstractStructSchema<T> parent;
+ public WrappedVersionedSchema(SchemaInfo schemaInfo, final byte[] schemaVersion,
+ AbstractStructSchema<T> parent) {
+ super(schemaInfo);
+ this.schemaVersion = schemaVersion;
+ this.writer = null;
+ this.reader = parent.reader;
+ this.schemaInfoProvider = parent.schemaInfoProvider;
+ this.parent = parent;
+ }
+
+ @Override
+ public boolean requireFetchingSchemaInfo() {
+ return true;
+ }
+
+ @Override
+ public T decode(byte[] bytes) {
+ return decode(bytes, schemaVersion);
+ }
+
+ @Override
+ public T decode(ByteBuf byteBuf) {
+ return decode(byteBuf, schemaVersion);
+ }
+
+ @Override
+ public byte[] encode(T message) {
+ throw new UnsupportedOperationException("This schema is not meant to be used for encoding");
+ }
+
+ @Override
+ public Optional<Object> getNativeSchema() {
+ if (reader instanceof AbstractMultiVersionReader) {
+ AbstractMultiVersionReader abstractMultiVersionReader = (AbstractMultiVersionReader) reader;
+ try {
+ SchemaReader schemaReader = abstractMultiVersionReader.getSchemaReader(schemaVersion);
+ return schemaReader.getNativeSchema();
+ } catch (ExecutionException err) {
+ throw new RuntimeException(err.getCause());
+ }
+ } else {
+ return Optional.empty();
+ }
+ }
+ }
+
+ private AbstractStructSchema<T> getAbstractStructSchemaAtVersion(byte[] schemaVersion, SchemaInfo schemaInfo) {
+ return new WrappedVersionedSchema<>(schemaInfo, schemaVersion, this);
}
protected void setWriter(SchemaWriter<T> writer) {
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java
index 3ec21a2..160a3a7 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java
@@ -78,6 +78,16 @@ public class AutoConsumeSchema implements Schema<GenericRecord> {
return schema == null || schema.supportSchemaVersioning();
}
+ public Schema<?> atSchemaVersion(byte[] schemaVersion) {
+ fetchSchemaIfNeeded();
+ ensureSchemaInitialized();
+ if (schema.supportSchemaVersioning() && schema instanceof AbstractSchema) {
+ return ((AbstractSchema) schema).atSchemaVersion(schemaVersion);
+ } else {
+ return schema;
+ }
+ }
+
@Override
public GenericRecord decode(byte[] bytes, byte[] schemaVersion) {
fetchSchemaIfNeeded();
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/KeyValueSchema.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/KeyValueSchema.java
index 4f7f921..f572bbf 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/KeyValueSchema.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/KeyValueSchema.java
@@ -21,6 +21,9 @@ package org.apache.pulsar.client.impl.schema;
import static com.google.common.base.Preconditions.checkArgument;
import java.util.concurrent.CompletableFuture;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.ByteBufUtil;
import lombok.Getter;
import lombok.extern.slf4j.Slf4j;
import org.apache.pulsar.client.api.Schema;
@@ -35,7 +38,7 @@ import org.apache.pulsar.common.schema.SchemaType;
* [Key, Value] pair schema definition
*/
@Slf4j
-public class KeyValueSchema<K, V> implements Schema<KeyValue<K, V>> {
+public class KeyValueSchema<K, V> extends AbstractSchema<KeyValue<K, V>> {
@Getter
private final Schema<K> keySchema;
@@ -139,18 +142,29 @@ public class KeyValueSchema<K, V> implements Schema<KeyValue<K, V>> {
}
}
+ @Override
public KeyValue<K, V> decode(byte[] bytes) {
return decode(bytes, null);
}
+ @Override
public KeyValue<K, V> decode(byte[] bytes, byte[] schemaVersion) {
if (this.keyValueEncodingType == KeyValueEncodingType.SEPARATED) {
throw new SchemaSerializationException("This method cannot be used under this SEPARATED encoding type");
}
-
return KeyValue.decode(bytes, (keyBytes, valueBytes) -> decode(keyBytes, valueBytes, schemaVersion));
}
+ @Override
+ public KeyValue<K, V> decode(ByteBuf byteBuf) {
+ return decode(ByteBufUtil.getBytes(byteBuf));
+ }
+
+ @Override
+ public KeyValue<K, V> decode(ByteBuf byteBuf, byte[] schemaVersion) {
+ return decode(ByteBufUtil.getBytes(byteBuf), schemaVersion);
+ }
+
public KeyValue<K, V> decode(byte[] keyBytes, byte[] valueBytes, byte[] schemaVersion) {
K k;
if (keyBytes == null) {
@@ -214,7 +228,6 @@ public class KeyValueSchema<K, V> implements Schema<KeyValue<K, V>> {
this.schemaInfo = KeyValueSchemaInfo.encodeKeyValueSchemaInfo(
keySchema, valueSchema, keyValueEncodingType
);
-
this.keySchema.setSchemaInfoProvider(new SchemaInfoProvider() {
@Override
public CompletableFuture<SchemaInfo> getSchemaByVersion(byte[] schemaVersion) {
@@ -253,4 +266,21 @@ public class KeyValueSchema<K, V> implements Schema<KeyValue<K, V>> {
}
});
}
+
+ @Override
+ public String toString() {
+ return "KeyValueSchema(" + keyValueEncodingType + "," + keySchema + "," + valueSchema + ")";
+ }
+
+ @Override
+ public Schema<?> atSchemaVersion(byte[] schemaVersion) throws SchemaSerializationException {
+ if (!supportSchemaVersioning()) {
+ return this;
+ } else {
+ Schema<?> keySchema = this.keySchema instanceof AbstractSchema ? ((AbstractSchema) this.keySchema).atSchemaVersion(schemaVersion) : this.keySchema;
+ Schema<?> valueSchema = this.valueSchema instanceof AbstractSchema ? ((AbstractSchema) this.valueSchema).atSchemaVersion(schemaVersion) : this.valueSchema;
+ return KeyValueSchema.of(keySchema, valueSchema, keyValueEncodingType);
+ }
+ }
+
}
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericAvroReader.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericAvroReader.java
index 22c63a94..d9a7384 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericAvroReader.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericAvroReader.java
@@ -36,6 +36,7 @@ import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.util.List;
+import java.util.Optional;
import java.util.stream.Collectors;
@@ -117,5 +118,10 @@ public class GenericAvroReader implements SchemaReader<GenericRecord> {
return offset;
}
+ @Override
+ public Optional<Object> getNativeSchema() {
+ return Optional.of(schema);
+ }
+
private static final Logger log = LoggerFactory.getLogger(GenericAvroReader.class);
}
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericProtobufNativeReader.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericProtobufNativeReader.java
index eccc482..c41767c 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericProtobufNativeReader.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericProtobufNativeReader.java
@@ -32,6 +32,7 @@ import java.io.IOException;
import java.io.InputStream;
import java.util.Arrays;
import java.util.List;
+import java.util.Optional;
import java.util.stream.Collectors;
public class GenericProtobufNativeReader implements SchemaReader<GenericRecord> {
@@ -79,5 +80,10 @@ public class GenericProtobufNativeReader implements SchemaReader<GenericRecord>
}
}
+ @Override
+ public Optional<Object> getNativeSchema() {
+ return Optional.of(descriptor);
+ }
+
private static final Logger log = LoggerFactory.getLogger(GenericProtobufNativeReader.class);
}
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericSchemaImpl.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericSchemaImpl.java
index 59e748c..e7d963c 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericSchemaImpl.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericSchemaImpl.java
@@ -25,6 +25,7 @@ import org.apache.pulsar.client.impl.schema.AvroBaseStructSchema;
import org.apache.pulsar.common.schema.SchemaInfo;
import java.util.List;
+import java.util.Optional;
import java.util.stream.Collectors;
/**
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/reader/AbstractMultiVersionReader.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/reader/AbstractMultiVersionReader.java
index ea0df25..f1a775d 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/reader/AbstractMultiVersionReader.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/reader/AbstractMultiVersionReader.java
@@ -71,7 +71,7 @@ public abstract class AbstractMultiVersionReader<T> implements SchemaReader<T> {
public T read(InputStream inputStream, byte[] schemaVersion) {
try {
return schemaVersion == null ? read(inputStream) :
- readerCache.get(BytesSchemaVersion.of(schemaVersion)).read(inputStream);
+ getSchemaReader(schemaVersion).read(inputStream);
} catch (ExecutionException e) {
LOG.error("Can't get generic schema for topic {} schema version {}",
schemaInfoProvider.getTopicName(), Hex.encodeHexString(schemaVersion), e);
@@ -79,11 +79,15 @@ public abstract class AbstractMultiVersionReader<T> implements SchemaReader<T> {
}
}
+ public SchemaReader<T> getSchemaReader(byte[] schemaVersion) throws ExecutionException {
+ return readerCache.get(BytesSchemaVersion.of(schemaVersion));
+ }
+
@Override
public T read(byte[] bytes, byte[] schemaVersion) {
try {
return schemaVersion == null ? read(bytes) :
- readerCache.get(BytesSchemaVersion.of(schemaVersion)).read(bytes);
+ getSchemaReader(schemaVersion).read(bytes);
} catch (ExecutionException | AvroTypeException e) {
if (e instanceof AvroTypeException) {
throw new SchemaSerializationException(e);
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/reader/AvroReader.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/reader/AvroReader.java
index e1b58a4..63885ba 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/reader/AvroReader.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/reader/AvroReader.java
@@ -32,18 +32,22 @@ import org.slf4j.LoggerFactory;
import java.io.IOException;
import java.io.InputStream;
+import java.util.Optional;
public class AvroReader<T> implements SchemaReader<T> {
private ReflectDatumReader<T> reader;
private static final ThreadLocal<BinaryDecoder> decoders =
new ThreadLocal<>();
+ private final Schema schema;
public AvroReader(Schema schema) {
this.reader = new ReflectDatumReader<>(schema);
+ this.schema = schema;
}
public AvroReader(Schema schema, ClassLoader classLoader, boolean jsr310ConversionEnabled) {
+ this.schema = schema;
if (classLoader != null) {
ReflectData reflectData = new ReflectData(classLoader);
AvroSchema.addLogicalTypeConversions(reflectData, jsr310ConversionEnabled);
@@ -55,6 +59,7 @@ public class AvroReader<T> implements SchemaReader<T> {
public AvroReader(Schema writerSchema, Schema readerSchema, ClassLoader classLoader,
boolean jsr310ConversionEnabled) {
+ this.schema = readerSchema;
if (classLoader != null) {
ReflectData reflectData = new ReflectData(classLoader);
AvroSchema.addLogicalTypeConversions(reflectData, jsr310ConversionEnabled);
@@ -98,6 +103,11 @@ public class AvroReader<T> implements SchemaReader<T> {
}
}
+ @Override
+ public Optional<Object> getNativeSchema() {
+ return Optional.of(schema);
+ }
+
private static final Logger log = LoggerFactory.getLogger(AvroReader.class);
}
diff --git a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/source/PulsarSource.java b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/source/PulsarSource.java
index e1ec9f6..6f8d326 100644
--- a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/source/PulsarSource.java
+++ b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/source/PulsarSource.java
@@ -132,10 +132,10 @@ public class PulsarSource<T> extends PushSource<T> implements MessageListener<T>
Schema<T> schema = null;
if (message instanceof MessageImpl) {
MessageImpl impl = (MessageImpl) message;
- schema = impl.getSchema();
+ schema = impl.getSchemaInternal();
} else if (message instanceof TopicMessageImpl) {
TopicMessageImpl impl = (TopicMessageImpl) message;
- schema = impl.getSchema();
+ schema = impl.getSchemaInternal();
}
Record<T> record = PulsarRecord.<T>builder()
.message(message)
diff --git a/pulsar-functions/instance/src/test/java/org/apache/pulsar/functions/source/PulsarSourceTest.java b/pulsar-functions/instance/src/test/java/org/apache/pulsar/functions/source/PulsarSourceTest.java
index be39508..65474ff 100644
--- a/pulsar-functions/instance/src/test/java/org/apache/pulsar/functions/source/PulsarSourceTest.java
+++ b/pulsar-functions/instance/src/test/java/org/apache/pulsar/functions/source/PulsarSourceTest.java
@@ -40,7 +40,6 @@ import lombok.extern.slf4j.Slf4j;
import org.apache.pulsar.client.api.Consumer;
import org.apache.pulsar.client.api.ConsumerBuilder;
-import org.apache.pulsar.client.api.Message;
import org.apache.pulsar.client.api.PulsarClientException;
import org.apache.pulsar.client.api.Schema;
import org.apache.pulsar.client.api.SubscriptionInitialPosition;
@@ -286,9 +285,9 @@ public class PulsarSourceTest {
Consumer consumer = mock(Consumer.class);
MessageImpl messageImpl = mock(MessageImpl.class);
Schema schema = mock(Schema.class);
- when(messageImpl.getSchema()).thenReturn(schema);
+ when(messageImpl.getSchemaInternal()).thenReturn(schema);
pulsarSource.received(consumer, (Message) messageImpl);
- verify(messageImpl.getSchema(), times(1));
+ verify(messageImpl.getSchemaInternal(), times(1));
Record<GenericRecord> pushed = pulsarSource.read();
assertSame(pushed.getSchema(), schema);
}
[pulsar] 26/46: Pulsar IO: Allow to develop Sinks that support
Schema but without setting it at build time (Sink) (#10034)
Posted by eo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit b116efd0944855174b422e43cecd13822c148dc6
Author: Enrico Olivelli <eo...@gmail.com>
AuthorDate: Sun Apr 11 16:44:33 2021 +0200
Pulsar IO: Allow to develop Sinks that support Schema but without setting it at build time (Sink<GenericObject>) (#10034)
(cherry picked from commit 42e92a7190fa36baa8f4ad48f980f3a8d879178c)
---
.../pulsar/functions/source/TopicSchema.java | 8 +-
.../integration/io/TestGenericObjectSink.java | 57 ++++++
.../io/PulsarGenericObjectSinkTest.java | 203 +++++++++++++++++++++
.../src/test/resources/pulsar-function-state.xml | 1 +
4 files changed, 265 insertions(+), 4 deletions(-)
diff --git a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/source/TopicSchema.java b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/source/TopicSchema.java
index ca689d2..a5cb58c 100644
--- a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/source/TopicSchema.java
+++ b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/source/TopicSchema.java
@@ -29,7 +29,7 @@ import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.apache.pulsar.client.api.PulsarClient;
import org.apache.pulsar.client.api.Schema;
-import org.apache.pulsar.client.api.schema.GenericRecord;
+import org.apache.pulsar.client.api.schema.GenericObject;
import org.apache.pulsar.client.api.schema.SchemaDefinition;
import org.apache.pulsar.client.impl.PulsarClientImpl;
import org.apache.pulsar.client.impl.schema.AvroSchema;
@@ -98,7 +98,7 @@ public class TopicSchema {
* If the topic is already created, we should be able to fetch the schema type (avro, json, ...)
*/
private SchemaType getSchemaTypeOrDefault(String topic, Class<?> clazz) {
- if (GenericRecord.class.isAssignableFrom(clazz)) {
+ if (GenericObject.class.isAssignableFrom(clazz)) {
return SchemaType.AUTO_CONSUME;
} else if (byte[].class.equals(clazz)
|| ByteBuf.class.equals(clazz)
@@ -124,8 +124,8 @@ public class TopicSchema {
|| ByteBuf.class.equals(clazz)
|| ByteBuffer.class.equals(clazz)) {
return SchemaType.NONE;
- } else if (GenericRecord.class.isAssignableFrom(clazz)) {
- // the function is taking generic record, so we do auto schema detection
+ } else if (GenericObject.class.isAssignableFrom(clazz)) {
+ // the function is taking generic record/object, so we do auto schema detection
return SchemaType.AUTO_CONSUME;
} else if (String.class.equals(clazz)) {
// If type is String, then we use schema type string, otherwise we fallback on default schema
diff --git a/tests/docker-images/java-test-functions/src/main/java/org/apache/pulsar/tests/integration/io/TestGenericObjectSink.java b/tests/docker-images/java-test-functions/src/main/java/org/apache/pulsar/tests/integration/io/TestGenericObjectSink.java
new file mode 100644
index 0000000..6b0da0b
--- /dev/null
+++ b/tests/docker-images/java-test-functions/src/main/java/org/apache/pulsar/tests/integration/io/TestGenericObjectSink.java
@@ -0,0 +1,57 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.tests.integration.io;
+
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.client.api.schema.GenericObject;
+import org.apache.pulsar.common.schema.KeyValue;
+import org.apache.pulsar.common.schema.SchemaType;
+import org.apache.pulsar.functions.api.KVRecord;
+import org.apache.pulsar.functions.api.Record;
+import org.apache.pulsar.io.core.Sink;
+import org.apache.pulsar.io.core.SinkContext;
+import org.apache.pulsar.io.core.Source;
+import org.apache.pulsar.io.core.SourceContext;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+
+@Slf4j
+public class TestGenericObjectSink implements Sink<GenericObject> {
+
+ @Override
+ public void open(Map<String, Object> config, SinkContext sourceContext) throws Exception {
+ }
+
+ public void write(Record<GenericObject> record) {
+ log.info("received record {} {}", record, record.getClass());
+ log.info("schema {}", record.getSchema());
+ log.info("native schema {}", record.getSchema().getNativeSchema().orElse(null));
+
+ log.info("value {}", record.getValue());
+ log.info("value schema type {}", record.getValue().getSchemaType());
+ log.info("value native object {}", record.getValue().getNativeObject());
+ }
+
+ @Override
+ public void close() throws Exception {
+
+ }
+}
diff --git a/tests/integration/src/test/java/org/apache/pulsar/tests/integration/io/PulsarGenericObjectSinkTest.java b/tests/integration/src/test/java/org/apache/pulsar/tests/integration/io/PulsarGenericObjectSinkTest.java
new file mode 100644
index 0000000..25f1776
--- /dev/null
+++ b/tests/integration/src/test/java/org/apache/pulsar/tests/integration/io/PulsarGenericObjectSinkTest.java
@@ -0,0 +1,203 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.tests.integration.io;
+
+import lombok.Builder;
+import lombok.Cleanup;
+import lombok.Data;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.pulsar.client.admin.PulsarAdmin;
+import org.apache.pulsar.client.api.Producer;
+import org.apache.pulsar.client.api.PulsarClient;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.common.policies.data.SinkStatus;
+import org.apache.pulsar.common.schema.KeyValue;
+import org.apache.pulsar.tests.integration.docker.ContainerExecException;
+import org.apache.pulsar.tests.integration.docker.ContainerExecResult;
+import org.apache.pulsar.tests.integration.suites.PulsarStandaloneTestSuite;
+import org.apache.pulsar.tests.integration.topologies.PulsarCluster;
+import org.awaitility.Awaitility;
+import org.testng.annotations.Test;
+
+import java.util.Arrays;
+import java.util.List;
+
+import static org.apache.pulsar.tests.integration.functions.utils.CommandGenerator.JAVAJAR;
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertTrue;
+import static org.testng.Assert.fail;
+
+/**
+ * Test behaviour of simple sinks
+ */
+@Slf4j
+public class PulsarGenericObjectSinkTest extends PulsarStandaloneTestSuite {
+
+ private static final class SinkSpec<T> {
+ final String outputTopicName;
+ final String sinkName;
+ final Schema<T> schema;
+ final T testValue;
+
+ public SinkSpec(String outputTopicName, String sinkName, Schema<T> schema, T testValue) {
+ this.outputTopicName = outputTopicName;
+ this.sinkName = sinkName;
+ this.schema = schema;
+ this.testValue = testValue;
+ }
+ }
+
+ @Data
+ @Builder
+ public static final class Pojo {
+ private String field1;
+ private int field2;
+ }
+
+ @Test(groups = {"sink"})
+ public void testGenericObjectSink() throws Exception {
+ // we are not using a parametrized test in order to save resources
+ // we create N sinks, send the records and verify each sink
+ // sinks execution happens in parallel
+ List<SinkSpec> specs = Arrays.asList(
+ new SinkSpec("test-kv-sink-input-string-" + randomName(8), "test-kv-sink-string-" + randomName(8), Schema.STRING, "foo"),
+ new SinkSpec("test-kv-sink-input-int-" + randomName(8), "test-kv-sink-int-" + randomName(8), Schema.INT32, 123),
+ new SinkSpec("test-kv-sink-input-avro-" + randomName(8), "test-kv-sink-avro-" + randomName(8), Schema.AVRO(Pojo.class), Pojo.builder().field1("a").field2(2).build()),
+ new SinkSpec("test-kv-sink-input-json-" + randomName(8), "test-kv-sink-json-" + randomName(8), Schema.JSON(Pojo.class), Pojo.builder().field1("a").field2(2).build())
+ );
+ // submit all sinks
+ for (SinkSpec spec : specs) {
+ submitSinkConnector(spec.sinkName, spec.outputTopicName, "org.apache.pulsar.tests.integration.io.TestGenericObjectSink", JAVAJAR);
+ }
+ // check all sinks
+ for (SinkSpec spec : specs) {
+ // get sink info
+ getSinkInfoSuccess(spec.sinkName);
+ getSinkStatus(spec.sinkName);
+ }
+
+ @Cleanup PulsarClient client = PulsarClient.builder()
+ .serviceUrl(container.getPlainTextServiceUrl())
+ .build();
+
+ final int numRecords = 10;
+
+ for (SinkSpec spec : specs) {
+ @Cleanup Producer<Object> producer = client.newProducer(spec.schema)
+ .topic(spec.outputTopicName)
+ .create();
+ for (int i = 0; i < numRecords; i++) {
+ producer.send(spec.testValue);
+ }
+ }
+
+ // wait that all sinks processed all records without errors
+ try (PulsarAdmin admin = PulsarAdmin.builder().serviceHttpUrl(container.getHttpServiceUrl()).build()) {
+
+ for (SinkSpec spec : specs) {
+ Awaitility.await().ignoreExceptions().untilAsserted(() -> {
+ SinkStatus status = admin.sinks().getSinkStatus("public", "default", spec.sinkName);
+ assertEquals(status.getInstances().size(), 1);
+ assertTrue(status.getInstances().get(0).getStatus().numReadFromPulsar >= numRecords);
+ assertTrue(status.getInstances().get(0).getStatus().numSinkExceptions == 0);
+ assertTrue(status.getInstances().get(0).getStatus().numSystemExceptions == 0);
+ });
+ }
+ }
+
+
+ for (SinkSpec spec : specs) {
+ deleteSink(spec.sinkName);
+ getSinkInfoNotFound(spec.sinkName);
+ }
+ }
+
+ private void submitSinkConnector(String sinkName,
+ String inputTopicName,
+ String className,
+ String archive) throws Exception {
+ String[] commands = {
+ PulsarCluster.ADMIN_SCRIPT,
+ "sinks", "create",
+ "--name", sinkName,
+ "-i", inputTopicName,
+ "--archive", archive,
+ "--classname", className
+ };
+ log.info("Run command : {}", StringUtils.join(commands, ' '));
+ ContainerExecResult result = container.execCmd(commands);
+ assertTrue(
+ result.getStdout().contains("\"Created successfully\""),
+ result.getStdout());
+ }
+
+ private void getSinkInfoSuccess(String sinkName) throws Exception {
+ ContainerExecResult result = container.execCmd(
+ PulsarCluster.ADMIN_SCRIPT,
+ "sinks",
+ "get",
+ "--tenant", "public",
+ "--namespace", "default",
+ "--name", sinkName
+ );
+ assertTrue(result.getStdout().contains("\"name\": \"" + sinkName + "\""));
+ }
+
+ private void getSinkStatus(String sinkName) throws Exception {
+ ContainerExecResult result = container.execCmd(
+ PulsarCluster.ADMIN_SCRIPT,
+ "sinks",
+ "status",
+ "--tenant", "public",
+ "--namespace", "default",
+ "--name", sinkName
+ );
+ assertTrue(result.getStdout().contains("\"running\" : true"));
+ }
+
+ private void deleteSink(String sinkName) throws Exception {
+ ContainerExecResult result = container.execCmd(
+ PulsarCluster.ADMIN_SCRIPT,
+ "sinks",
+ "delete",
+ "--tenant", "public",
+ "--namespace", "default",
+ "--name", sinkName
+ );
+ assertTrue(result.getStdout().contains("successfully"));
+ result.assertNoStderr();
+ }
+
+ private void getSinkInfoNotFound(String sinkName) throws Exception {
+ try {
+ container.execCmd(
+ PulsarCluster.ADMIN_SCRIPT,
+ "sinks",
+ "get",
+ "--tenant", "public",
+ "--namespace", "default",
+ "--name", sinkName);
+ fail("Command should have exited with non-zero");
+ } catch (ContainerExecException e) {
+ assertTrue(e.getResult().getStderr().contains(sinkName + " doesn't exist"));
+ }
+ }
+}
+
diff --git a/tests/integration/src/test/resources/pulsar-function-state.xml b/tests/integration/src/test/resources/pulsar-function-state.xml
index ebf3392..c8ba8bc 100644
--- a/tests/integration/src/test/resources/pulsar-function-state.xml
+++ b/tests/integration/src/test/resources/pulsar-function-state.xml
@@ -24,6 +24,7 @@
<classes>
<class name="org.apache.pulsar.tests.integration.functions.PulsarStateTest" />
<class name="org.apache.pulsar.tests.integration.io.PulsarSourcePropertyTest"/>
+ <class name="org.apache.pulsar.tests.integration.io.PulsarGenericObjectSinkTest"/>
</classes>
</test>
</suite>
[pulsar] 01/46: Start 2.7.2_1.0.0 Luna Streaming Disto Fork
Posted by eo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 37d2dfacdf87e68feaed7c7a26cd7b16f947c65e
Author: Enrico Olivelli <eo...@apache.org>
AuthorDate: Fri May 7 15:09:59 2021 +0200
Start 2.7.2_1.0.0 Luna Streaming Disto Fork
---
bouncy-castle/bc/pom.xml | 2 +-
bouncy-castle/bcfips-include-test/pom.xml | 2 +-
bouncy-castle/bcfips/pom.xml | 2 +-
bouncy-castle/pom.xml | 2 +-
buildtools/pom.xml | 2 +-
dashboard/pom.xml | 2 +-
distribution/io/pom.xml | 2 +-
distribution/offloaders/pom.xml | 2 +-
distribution/pom.xml | 2 +-
distribution/server/pom.xml | 2 +-
docker/grafana/pom.xml | 2 +-
docker/pom.xml | 2 +-
docker/pulsar-all/pom.xml | 2 +-
docker/pulsar-standalone/pom.xml | 2 +-
docker/pulsar/pom.xml | 2 +-
jclouds-shaded/pom.xml | 2 +-
kafka-connect-avro-converter-shaded/pom.xml | 2 +-
managed-ledger/pom.xml | 2 +-
pom.xml | 2 +-
protobuf-shaded/pom.xml | 2 +-
pulsar-broker-auth-athenz/pom.xml | 2 +-
pulsar-broker-auth-sasl/pom.xml | 2 +-
pulsar-broker-common/pom.xml | 2 +-
pulsar-broker-shaded/pom.xml | 2 +-
pulsar-broker/pom.xml | 2 +-
pulsar-client-1x-base/pom.xml | 2 +-
pulsar-client-1x-base/pulsar-client-1x/pom.xml | 2 +-
pulsar-client-1x-base/pulsar-client-2x-shaded/pom.xml | 2 +-
pulsar-client-admin-shaded/pom.xml | 2 +-
pulsar-client-admin/pom.xml | 2 +-
pulsar-client-all/pom.xml | 2 +-
pulsar-client-api/pom.xml | 2 +-
pulsar-client-auth-athenz/pom.xml | 2 +-
pulsar-client-auth-sasl/pom.xml | 2 +-
pulsar-client-messagecrypto-bc/pom.xml | 2 +-
pulsar-client-shaded/pom.xml | 2 +-
pulsar-client-tools-test/pom.xml | 2 +-
pulsar-client-tools/pom.xml | 2 +-
pulsar-client/pom.xml | 2 +-
pulsar-common/pom.xml | 2 +-
pulsar-config-validation/pom.xml | 2 +-
pulsar-discovery-service/pom.xml | 2 +-
pulsar-functions/api-java/pom.xml | 2 +-
pulsar-functions/instance/pom.xml | 2 +-
pulsar-functions/java-examples/pom.xml | 2 +-
pulsar-functions/localrun-shaded/pom.xml | 2 +-
pulsar-functions/localrun/pom.xml | 2 +-
pulsar-functions/pom.xml | 2 +-
pulsar-functions/proto/pom.xml | 2 +-
pulsar-functions/runtime-all/pom.xml | 2 +-
pulsar-functions/runtime/pom.xml | 2 +-
pulsar-functions/secrets/pom.xml | 2 +-
pulsar-functions/utils/pom.xml | 2 +-
pulsar-functions/worker/pom.xml | 2 +-
pulsar-io/aerospike/pom.xml | 2 +-
pulsar-io/aws/pom.xml | 2 +-
pulsar-io/batch-data-generator/pom.xml | 2 +-
pulsar-io/batch-discovery-triggerers/pom.xml | 2 +-
pulsar-io/canal/pom.xml | 2 +-
pulsar-io/cassandra/pom.xml | 2 +-
pulsar-io/common/pom.xml | 2 +-
pulsar-io/core/pom.xml | 2 +-
pulsar-io/data-generator/pom.xml | 2 +-
pulsar-io/debezium/core/pom.xml | 2 +-
pulsar-io/debezium/mongodb/pom.xml | 2 +-
pulsar-io/debezium/mysql/pom.xml | 2 +-
pulsar-io/debezium/pom.xml | 2 +-
pulsar-io/debezium/postgres/pom.xml | 2 +-
pulsar-io/docs/pom.xml | 2 +-
pulsar-io/dynamodb/pom.xml | 2 +-
pulsar-io/elastic-search/pom.xml | 2 +-
pulsar-io/file/pom.xml | 2 +-
pulsar-io/flume/pom.xml | 2 +-
pulsar-io/hbase/pom.xml | 2 +-
pulsar-io/hdfs2/pom.xml | 2 +-
pulsar-io/hdfs3/pom.xml | 2 +-
pulsar-io/influxdb/pom.xml | 2 +-
pulsar-io/jdbc/clickhouse/pom.xml | 2 +-
pulsar-io/jdbc/core/pom.xml | 2 +-
pulsar-io/jdbc/mariadb/pom.xml | 2 +-
pulsar-io/jdbc/pom.xml | 2 +-
pulsar-io/jdbc/postgres/pom.xml | 2 +-
pulsar-io/jdbc/sqlite/pom.xml | 2 +-
pulsar-io/kafka-connect-adaptor/pom.xml | 2 +-
pulsar-io/kafka/pom.xml | 2 +-
pulsar-io/kinesis/pom.xml | 2 +-
pulsar-io/mongo/pom.xml | 2 +-
pulsar-io/netty/pom.xml | 2 +-
pulsar-io/nsq/pom.xml | 2 +-
pulsar-io/pom.xml | 2 +-
pulsar-io/rabbitmq/pom.xml | 2 +-
pulsar-io/redis/pom.xml | 2 +-
pulsar-io/solr/pom.xml | 2 +-
pulsar-io/twitter/pom.xml | 2 +-
pulsar-metadata/pom.xml | 2 +-
pulsar-proxy/pom.xml | 2 +-
pulsar-sql/pom.xml | 2 +-
pulsar-sql/presto-distribution/pom.xml | 2 +-
pulsar-sql/presto-pulsar-plugin/pom.xml | 2 +-
pulsar-sql/presto-pulsar/pom.xml | 2 +-
pulsar-testclient/pom.xml | 2 +-
pulsar-transaction/common/pom.xml | 2 +-
pulsar-transaction/coordinator/pom.xml | 2 +-
pulsar-transaction/pom.xml | 2 +-
pulsar-websocket/pom.xml | 2 +-
pulsar-zookeeper-utils/pom.xml | 2 +-
pulsar-zookeeper/pom.xml | 2 +-
testmocks/pom.xml | 2 +-
tests/bc_2_0_0/pom.xml | 2 +-
tests/bc_2_0_1/pom.xml | 2 +-
tests/docker-images/java-test-functions/pom.xml | 2 +-
tests/docker-images/latest-version-image/pom.xml | 2 +-
tests/docker-images/pom.xml | 2 +-
tests/integration/pom.xml | 2 +-
tests/pom.xml | 2 +-
tests/pulsar-client-admin-shade-test/pom.xml | 2 +-
tests/pulsar-client-all-shade-test/pom.xml | 2 +-
tests/pulsar-client-shade-test/pom.xml | 2 +-
tiered-storage/file-system/pom.xml | 2 +-
tiered-storage/jcloud/pom.xml | 2 +-
tiered-storage/pom.xml | 2 +-
121 files changed, 121 insertions(+), 121 deletions(-)
diff --git a/bouncy-castle/bc/pom.xml b/bouncy-castle/bc/pom.xml
index 54c5504..77fac2c 100644
--- a/bouncy-castle/bc/pom.xml
+++ b/bouncy-castle/bc/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>bouncy-castle-parent</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/bouncy-castle/bcfips-include-test/pom.xml b/bouncy-castle/bcfips-include-test/pom.xml
index cf0d193..0e51db6 100644
--- a/bouncy-castle/bcfips-include-test/pom.xml
+++ b/bouncy-castle/bcfips-include-test/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>bouncy-castle-parent</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/bouncy-castle/bcfips/pom.xml b/bouncy-castle/bcfips/pom.xml
index 339da61..45c2ebf 100644
--- a/bouncy-castle/bcfips/pom.xml
+++ b/bouncy-castle/bcfips/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>bouncy-castle-parent</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/bouncy-castle/pom.xml b/bouncy-castle/pom.xml
index ff8a09c..14061a4 100644
--- a/bouncy-castle/pom.xml
+++ b/bouncy-castle/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/buildtools/pom.xml b/buildtools/pom.xml
index bfed8d2..6656abe 100644
--- a/buildtools/pom.xml
+++ b/buildtools/pom.xml
@@ -30,7 +30,7 @@
<groupId>org.apache.pulsar</groupId>
<artifactId>buildtools</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<packaging>jar</packaging>
<name>Pulsar Build Tools</name>
diff --git a/dashboard/pom.xml b/dashboard/pom.xml
index 2eed258..5a23db5 100644
--- a/dashboard/pom.xml
+++ b/dashboard/pom.xml
@@ -23,7 +23,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>docker-images</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<relativePath>../docker</relativePath>
</parent>
<modelVersion>4.0.0</modelVersion>
diff --git a/distribution/io/pom.xml b/distribution/io/pom.xml
index 91038e0..cbbe128 100644
--- a/distribution/io/pom.xml
+++ b/distribution/io/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>distribution</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/distribution/offloaders/pom.xml b/distribution/offloaders/pom.xml
index 7e06228..2b4ef1a 100644
--- a/distribution/offloaders/pom.xml
+++ b/distribution/offloaders/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>distribution</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/distribution/pom.xml b/distribution/pom.xml
index 3aab9fc..9e90d26 100644
--- a/distribution/pom.xml
+++ b/distribution/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/distribution/server/pom.xml b/distribution/server/pom.xml
index 961e79b..0ae9d04 100644
--- a/distribution/server/pom.xml
+++ b/distribution/server/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>distribution</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/docker/grafana/pom.xml b/docker/grafana/pom.xml
index b26e596..3899e3b 100644
--- a/docker/grafana/pom.xml
+++ b/docker/grafana/pom.xml
@@ -23,7 +23,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>docker-images</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>grafana-docker-image</artifactId>
diff --git a/docker/pom.xml b/docker/pom.xml
index 1e145bb..7ff30bf 100644
--- a/docker/pom.xml
+++ b/docker/pom.xml
@@ -26,7 +26,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>docker-images</artifactId>
<name>Apache Pulsar :: Docker Images</name>
diff --git a/docker/pulsar-all/pom.xml b/docker/pulsar-all/pom.xml
index 2747eb0..272fa8a 100644
--- a/docker/pulsar-all/pom.xml
+++ b/docker/pulsar-all/pom.xml
@@ -23,7 +23,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>docker-images</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>pulsar-all-docker-image</artifactId>
diff --git a/docker/pulsar-standalone/pom.xml b/docker/pulsar-standalone/pom.xml
index b908aa2..b5e5ba1 100644
--- a/docker/pulsar-standalone/pom.xml
+++ b/docker/pulsar-standalone/pom.xml
@@ -23,7 +23,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>docker-images</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>pulsar-standalone-docker-image</artifactId>
diff --git a/docker/pulsar/pom.xml b/docker/pulsar/pom.xml
index 2704130..9055441 100644
--- a/docker/pulsar/pom.xml
+++ b/docker/pulsar/pom.xml
@@ -23,7 +23,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>docker-images</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>pulsar-docker-image</artifactId>
diff --git a/jclouds-shaded/pom.xml b/jclouds-shaded/pom.xml
index 4021786..43e85e5 100644
--- a/jclouds-shaded/pom.xml
+++ b/jclouds-shaded/pom.xml
@@ -26,7 +26,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/kafka-connect-avro-converter-shaded/pom.xml b/kafka-connect-avro-converter-shaded/pom.xml
index cd41bf2..9f26f7c 100644
--- a/kafka-connect-avro-converter-shaded/pom.xml
+++ b/kafka-connect-avro-converter-shaded/pom.xml
@@ -26,7 +26,7 @@
<parent>
<artifactId>pulsar</artifactId>
<groupId>org.apache.pulsar</groupId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/managed-ledger/pom.xml b/managed-ledger/pom.xml
index fc63df4..8f7efea 100644
--- a/managed-ledger/pom.xml
+++ b/managed-ledger/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pom.xml b/pom.xml
index eb35a96..c0f3a24 100644
--- a/pom.xml
+++ b/pom.xml
@@ -32,7 +32,7 @@
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<name>Pulsar</name>
<description>Pulsar is a distributed pub-sub messaging platform with a very
diff --git a/protobuf-shaded/pom.xml b/protobuf-shaded/pom.xml
index 66c214f..49a1201 100644
--- a/protobuf-shaded/pom.xml
+++ b/protobuf-shaded/pom.xml
@@ -26,7 +26,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-broker-auth-athenz/pom.xml b/pulsar-broker-auth-athenz/pom.xml
index 37fef50..4c74389 100644
--- a/pulsar-broker-auth-athenz/pom.xml
+++ b/pulsar-broker-auth-athenz/pom.xml
@@ -26,7 +26,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-broker-auth-athenz</artifactId>
diff --git a/pulsar-broker-auth-sasl/pom.xml b/pulsar-broker-auth-sasl/pom.xml
index e54e895..3ced729 100644
--- a/pulsar-broker-auth-sasl/pom.xml
+++ b/pulsar-broker-auth-sasl/pom.xml
@@ -26,7 +26,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-broker-auth-sasl</artifactId>
diff --git a/pulsar-broker-common/pom.xml b/pulsar-broker-common/pom.xml
index 4c9665a..ed8d978 100644
--- a/pulsar-broker-common/pom.xml
+++ b/pulsar-broker-common/pom.xml
@@ -26,7 +26,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-broker-common</artifactId>
diff --git a/pulsar-broker-shaded/pom.xml b/pulsar-broker-shaded/pom.xml
index b3cde57..22add27 100644
--- a/pulsar-broker-shaded/pom.xml
+++ b/pulsar-broker-shaded/pom.xml
@@ -26,7 +26,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-broker/pom.xml b/pulsar-broker/pom.xml
index 73b7de7..c811db6 100644
--- a/pulsar-broker/pom.xml
+++ b/pulsar-broker/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-client-1x-base/pom.xml b/pulsar-client-1x-base/pom.xml
index 9766d41..c4baae7 100644
--- a/pulsar-client-1x-base/pom.xml
+++ b/pulsar-client-1x-base/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-client-1x-base/pulsar-client-1x/pom.xml b/pulsar-client-1x-base/pulsar-client-1x/pom.xml
index 41fc22c..8754d7e 100644
--- a/pulsar-client-1x-base/pulsar-client-1x/pom.xml
+++ b/pulsar-client-1x-base/pulsar-client-1x/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-client-1x-base</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-client-1x-base/pulsar-client-2x-shaded/pom.xml b/pulsar-client-1x-base/pulsar-client-2x-shaded/pom.xml
index 451f452..108b367 100644
--- a/pulsar-client-1x-base/pulsar-client-2x-shaded/pom.xml
+++ b/pulsar-client-1x-base/pulsar-client-2x-shaded/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-client-1x-base</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-client-admin-shaded/pom.xml b/pulsar-client-admin-shaded/pom.xml
index 3393548..d0e0cc5 100644
--- a/pulsar-client-admin-shaded/pom.xml
+++ b/pulsar-client-admin-shaded/pom.xml
@@ -26,7 +26,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-client-admin/pom.xml b/pulsar-client-admin/pom.xml
index b8f93a0..432c33b 100644
--- a/pulsar-client-admin/pom.xml
+++ b/pulsar-client-admin/pom.xml
@@ -26,7 +26,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-client-all/pom.xml b/pulsar-client-all/pom.xml
index 22dcc76..4623f57 100644
--- a/pulsar-client-all/pom.xml
+++ b/pulsar-client-all/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-client-api/pom.xml b/pulsar-client-api/pom.xml
index 636e6be..ffb69a4 100644
--- a/pulsar-client-api/pom.xml
+++ b/pulsar-client-api/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-client-auth-athenz/pom.xml b/pulsar-client-auth-athenz/pom.xml
index 700ef1d..b71fcca 100644
--- a/pulsar-client-auth-athenz/pom.xml
+++ b/pulsar-client-auth-athenz/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-client-auth-sasl/pom.xml b/pulsar-client-auth-sasl/pom.xml
index 48dd039..d42172b 100644
--- a/pulsar-client-auth-sasl/pom.xml
+++ b/pulsar-client-auth-sasl/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-client-messagecrypto-bc/pom.xml b/pulsar-client-messagecrypto-bc/pom.xml
index 994e49f..bf0882b 100644
--- a/pulsar-client-messagecrypto-bc/pom.xml
+++ b/pulsar-client-messagecrypto-bc/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-client-shaded/pom.xml b/pulsar-client-shaded/pom.xml
index c879858..66b65b1 100644
--- a/pulsar-client-shaded/pom.xml
+++ b/pulsar-client-shaded/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-client-tools-test/pom.xml b/pulsar-client-tools-test/pom.xml
index 23533a1..6e66a8e 100644
--- a/pulsar-client-tools-test/pom.xml
+++ b/pulsar-client-tools-test/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-client-tools/pom.xml b/pulsar-client-tools/pom.xml
index 1c0aa4f..89b103c 100644
--- a/pulsar-client-tools/pom.xml
+++ b/pulsar-client-tools/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-client/pom.xml b/pulsar-client/pom.xml
index 655d3e0..f2d6370 100644
--- a/pulsar-client/pom.xml
+++ b/pulsar-client/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-common/pom.xml b/pulsar-common/pom.xml
index 445ecc7..2fcf9fd 100644
--- a/pulsar-common/pom.xml
+++ b/pulsar-common/pom.xml
@@ -26,7 +26,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-config-validation/pom.xml b/pulsar-config-validation/pom.xml
index cd336ae..5dd64c4 100644
--- a/pulsar-config-validation/pom.xml
+++ b/pulsar-config-validation/pom.xml
@@ -26,7 +26,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-discovery-service/pom.xml b/pulsar-discovery-service/pom.xml
index d897597..65bd2b2 100644
--- a/pulsar-discovery-service/pom.xml
+++ b/pulsar-discovery-service/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-functions/api-java/pom.xml b/pulsar-functions/api-java/pom.xml
index 24fe816..0bb9733 100644
--- a/pulsar-functions/api-java/pom.xml
+++ b/pulsar-functions/api-java/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-functions</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-functions-api</artifactId>
diff --git a/pulsar-functions/instance/pom.xml b/pulsar-functions/instance/pom.xml
index ec4c5d7..a6d05d5 100644
--- a/pulsar-functions/instance/pom.xml
+++ b/pulsar-functions/instance/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-functions</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-functions-instance</artifactId>
diff --git a/pulsar-functions/java-examples/pom.xml b/pulsar-functions/java-examples/pom.xml
index 73ebd6a..fa8f400 100644
--- a/pulsar-functions/java-examples/pom.xml
+++ b/pulsar-functions/java-examples/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-functions</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-functions-api-examples</artifactId>
diff --git a/pulsar-functions/localrun-shaded/pom.xml b/pulsar-functions/localrun-shaded/pom.xml
index d07463f..858f724 100644
--- a/pulsar-functions/localrun-shaded/pom.xml
+++ b/pulsar-functions/localrun-shaded/pom.xml
@@ -26,7 +26,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-functions</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-functions/localrun/pom.xml b/pulsar-functions/localrun/pom.xml
index fb4a0f2..6d13bcb 100644
--- a/pulsar-functions/localrun/pom.xml
+++ b/pulsar-functions/localrun/pom.xml
@@ -26,7 +26,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-functions</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-functions/pom.xml b/pulsar-functions/pom.xml
index 413dcc4..0147e7e 100644
--- a/pulsar-functions/pom.xml
+++ b/pulsar-functions/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-functions</artifactId>
diff --git a/pulsar-functions/proto/pom.xml b/pulsar-functions/proto/pom.xml
index a8a8d6e..5d35a6f 100644
--- a/pulsar-functions/proto/pom.xml
+++ b/pulsar-functions/proto/pom.xml
@@ -27,7 +27,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-functions</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-functions-proto</artifactId>
diff --git a/pulsar-functions/runtime-all/pom.xml b/pulsar-functions/runtime-all/pom.xml
index fbadfcc..0d25d27 100644
--- a/pulsar-functions/runtime-all/pom.xml
+++ b/pulsar-functions/runtime-all/pom.xml
@@ -26,7 +26,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-functions</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-functions/runtime/pom.xml b/pulsar-functions/runtime/pom.xml
index eeca33a..2a5e77f 100644
--- a/pulsar-functions/runtime/pom.xml
+++ b/pulsar-functions/runtime/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-functions</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-functions-runtime</artifactId>
diff --git a/pulsar-functions/secrets/pom.xml b/pulsar-functions/secrets/pom.xml
index 4e2f1e8..553f5a4 100644
--- a/pulsar-functions/secrets/pom.xml
+++ b/pulsar-functions/secrets/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-functions</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-functions-secrets</artifactId>
diff --git a/pulsar-functions/utils/pom.xml b/pulsar-functions/utils/pom.xml
index 2ef9695..293cdac 100644
--- a/pulsar-functions/utils/pom.xml
+++ b/pulsar-functions/utils/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-functions</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-functions-utils</artifactId>
diff --git a/pulsar-functions/worker/pom.xml b/pulsar-functions/worker/pom.xml
index cc1ce22..c833431 100644
--- a/pulsar-functions/worker/pom.xml
+++ b/pulsar-functions/worker/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-functions</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-io/aerospike/pom.xml b/pulsar-io/aerospike/pom.xml
index c8cc0d7..40032b3 100644
--- a/pulsar-io/aerospike/pom.xml
+++ b/pulsar-io/aerospike/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-io-aerospike</artifactId>
diff --git a/pulsar-io/aws/pom.xml b/pulsar-io/aws/pom.xml
index be880db..10a4071 100644
--- a/pulsar-io/aws/pom.xml
+++ b/pulsar-io/aws/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-io-aws</artifactId>
diff --git a/pulsar-io/batch-data-generator/pom.xml b/pulsar-io/batch-data-generator/pom.xml
index 4591670..993463b 100644
--- a/pulsar-io/batch-data-generator/pom.xml
+++ b/pulsar-io/batch-data-generator/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-io-batch-data-generator</artifactId>
diff --git a/pulsar-io/batch-discovery-triggerers/pom.xml b/pulsar-io/batch-discovery-triggerers/pom.xml
index e72337b..f1aa8ac 100644
--- a/pulsar-io/batch-discovery-triggerers/pom.xml
+++ b/pulsar-io/batch-discovery-triggerers/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-io-batch-discovery-triggerers</artifactId>
diff --git a/pulsar-io/canal/pom.xml b/pulsar-io/canal/pom.xml
index 73eba07..cbf3e7e 100644
--- a/pulsar-io/canal/pom.xml
+++ b/pulsar-io/canal/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<modelVersion>4.0.0</modelVersion>
diff --git a/pulsar-io/cassandra/pom.xml b/pulsar-io/cassandra/pom.xml
index 58c5c05..070b2e2 100644
--- a/pulsar-io/cassandra/pom.xml
+++ b/pulsar-io/cassandra/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-io-cassandra</artifactId>
diff --git a/pulsar-io/common/pom.xml b/pulsar-io/common/pom.xml
index 6088eb3..6e28b03 100644
--- a/pulsar-io/common/pom.xml
+++ b/pulsar-io/common/pom.xml
@@ -27,7 +27,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-io-common</artifactId>
diff --git a/pulsar-io/core/pom.xml b/pulsar-io/core/pom.xml
index e09a5a1..3619208 100644
--- a/pulsar-io/core/pom.xml
+++ b/pulsar-io/core/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-io-core</artifactId>
diff --git a/pulsar-io/data-generator/pom.xml b/pulsar-io/data-generator/pom.xml
index 627d0f2..ef62a08 100644
--- a/pulsar-io/data-generator/pom.xml
+++ b/pulsar-io/data-generator/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-io-data-generator</artifactId>
diff --git a/pulsar-io/debezium/core/pom.xml b/pulsar-io/debezium/core/pom.xml
index 0de37de..6213542 100644
--- a/pulsar-io/debezium/core/pom.xml
+++ b/pulsar-io/debezium/core/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io-debezium</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-io-debezium-core</artifactId>
diff --git a/pulsar-io/debezium/mongodb/pom.xml b/pulsar-io/debezium/mongodb/pom.xml
index 1e15c5b..535b8d5 100644
--- a/pulsar-io/debezium/mongodb/pom.xml
+++ b/pulsar-io/debezium/mongodb/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io-debezium</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-io-debezium-mongodb</artifactId>
diff --git a/pulsar-io/debezium/mysql/pom.xml b/pulsar-io/debezium/mysql/pom.xml
index 9f4a4f9..edb98f5 100644
--- a/pulsar-io/debezium/mysql/pom.xml
+++ b/pulsar-io/debezium/mysql/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io-debezium</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-io-debezium-mysql</artifactId>
diff --git a/pulsar-io/debezium/pom.xml b/pulsar-io/debezium/pom.xml
index efe25c0..fcdcfb3 100644
--- a/pulsar-io/debezium/pom.xml
+++ b/pulsar-io/debezium/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-io-debezium</artifactId>
diff --git a/pulsar-io/debezium/postgres/pom.xml b/pulsar-io/debezium/postgres/pom.xml
index 8a93f43..1662601 100644
--- a/pulsar-io/debezium/postgres/pom.xml
+++ b/pulsar-io/debezium/postgres/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io-debezium</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-io-debezium-postgres</artifactId>
diff --git a/pulsar-io/docs/pom.xml b/pulsar-io/docs/pom.xml
index 215784ca..4ba8b77 100644
--- a/pulsar-io/docs/pom.xml
+++ b/pulsar-io/docs/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-io-docs</artifactId>
diff --git a/pulsar-io/dynamodb/pom.xml b/pulsar-io/dynamodb/pom.xml
index 3e2b36d..6808d65 100644
--- a/pulsar-io/dynamodb/pom.xml
+++ b/pulsar-io/dynamodb/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-io-dynamodb</artifactId>
diff --git a/pulsar-io/elastic-search/pom.xml b/pulsar-io/elastic-search/pom.xml
index 22f6be2..fca5fe0 100644
--- a/pulsar-io/elastic-search/pom.xml
+++ b/pulsar-io/elastic-search/pom.xml
@@ -23,7 +23,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-io-elastic-search</artifactId>
<name>Pulsar IO :: ElasticSearch</name>
diff --git a/pulsar-io/file/pom.xml b/pulsar-io/file/pom.xml
index a3cf01f..1dd0847 100644
--- a/pulsar-io/file/pom.xml
+++ b/pulsar-io/file/pom.xml
@@ -23,7 +23,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-io-file</artifactId>
diff --git a/pulsar-io/flume/pom.xml b/pulsar-io/flume/pom.xml
index af2abce..47f093b 100644
--- a/pulsar-io/flume/pom.xml
+++ b/pulsar-io/flume/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-io-flume</artifactId>
diff --git a/pulsar-io/hbase/pom.xml b/pulsar-io/hbase/pom.xml
index 230f488..6dd5088 100644
--- a/pulsar-io/hbase/pom.xml
+++ b/pulsar-io/hbase/pom.xml
@@ -25,7 +25,7 @@
<parent>
<artifactId>pulsar-io</artifactId>
<groupId>org.apache.pulsar</groupId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-io-hbase</artifactId>
<name>Pulsar IO :: Hbase</name>
diff --git a/pulsar-io/hdfs2/pom.xml b/pulsar-io/hdfs2/pom.xml
index ef442a8..7c01777 100644
--- a/pulsar-io/hdfs2/pom.xml
+++ b/pulsar-io/hdfs2/pom.xml
@@ -23,7 +23,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-io-hdfs2</artifactId>
<name>Pulsar IO :: Hdfs2</name>
diff --git a/pulsar-io/hdfs3/pom.xml b/pulsar-io/hdfs3/pom.xml
index 8300fd4..af77bc1 100644
--- a/pulsar-io/hdfs3/pom.xml
+++ b/pulsar-io/hdfs3/pom.xml
@@ -23,7 +23,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-io-hdfs3</artifactId>
<name>Pulsar IO :: Hdfs3</name>
diff --git a/pulsar-io/influxdb/pom.xml b/pulsar-io/influxdb/pom.xml
index eec8a76..0141024 100644
--- a/pulsar-io/influxdb/pom.xml
+++ b/pulsar-io/influxdb/pom.xml
@@ -25,7 +25,7 @@
<parent>
<artifactId>pulsar-io</artifactId>
<groupId>org.apache.pulsar</groupId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-io-influxdb</artifactId>
diff --git a/pulsar-io/jdbc/clickhouse/pom.xml b/pulsar-io/jdbc/clickhouse/pom.xml
index 660cf26..cd8b345 100644
--- a/pulsar-io/jdbc/clickhouse/pom.xml
+++ b/pulsar-io/jdbc/clickhouse/pom.xml
@@ -24,7 +24,7 @@
<parent>
<artifactId>pulsar-io-jdbc</artifactId>
<groupId>org.apache.pulsar</groupId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<modelVersion>4.0.0</modelVersion>
diff --git a/pulsar-io/jdbc/core/pom.xml b/pulsar-io/jdbc/core/pom.xml
index 0625bee..2c39e76 100644
--- a/pulsar-io/jdbc/core/pom.xml
+++ b/pulsar-io/jdbc/core/pom.xml
@@ -24,7 +24,7 @@
<parent>
<artifactId>pulsar-io-jdbc</artifactId>
<groupId>org.apache.pulsar</groupId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<modelVersion>4.0.0</modelVersion>
diff --git a/pulsar-io/jdbc/mariadb/pom.xml b/pulsar-io/jdbc/mariadb/pom.xml
index 48dc020..f377bfc 100644
--- a/pulsar-io/jdbc/mariadb/pom.xml
+++ b/pulsar-io/jdbc/mariadb/pom.xml
@@ -24,7 +24,7 @@
<parent>
<artifactId>pulsar-io-jdbc</artifactId>
<groupId>org.apache.pulsar</groupId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<modelVersion>4.0.0</modelVersion>
diff --git a/pulsar-io/jdbc/pom.xml b/pulsar-io/jdbc/pom.xml
index 0f38301..5988e5f 100644
--- a/pulsar-io/jdbc/pom.xml
+++ b/pulsar-io/jdbc/pom.xml
@@ -32,7 +32,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-io-jdbc</artifactId>
diff --git a/pulsar-io/jdbc/postgres/pom.xml b/pulsar-io/jdbc/postgres/pom.xml
index f926acb..3caaeb1 100644
--- a/pulsar-io/jdbc/postgres/pom.xml
+++ b/pulsar-io/jdbc/postgres/pom.xml
@@ -24,7 +24,7 @@
<parent>
<artifactId>pulsar-io-jdbc</artifactId>
<groupId>org.apache.pulsar</groupId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<modelVersion>4.0.0</modelVersion>
diff --git a/pulsar-io/jdbc/sqlite/pom.xml b/pulsar-io/jdbc/sqlite/pom.xml
index 1c83c70..75766d5 100644
--- a/pulsar-io/jdbc/sqlite/pom.xml
+++ b/pulsar-io/jdbc/sqlite/pom.xml
@@ -24,7 +24,7 @@
<parent>
<artifactId>pulsar-io-jdbc</artifactId>
<groupId>org.apache.pulsar</groupId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>pulsar-io-jdbc-sqlite</artifactId>
diff --git a/pulsar-io/kafka-connect-adaptor/pom.xml b/pulsar-io/kafka-connect-adaptor/pom.xml
index 7f5880b..6ca2c14 100644
--- a/pulsar-io/kafka-connect-adaptor/pom.xml
+++ b/pulsar-io/kafka-connect-adaptor/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-io-kafka-connect-adaptor</artifactId>
diff --git a/pulsar-io/kafka/pom.xml b/pulsar-io/kafka/pom.xml
index ba314bf..08ce625 100644
--- a/pulsar-io/kafka/pom.xml
+++ b/pulsar-io/kafka/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-io-kafka</artifactId>
diff --git a/pulsar-io/kinesis/pom.xml b/pulsar-io/kinesis/pom.xml
index 22dc5d4..996b3d0 100644
--- a/pulsar-io/kinesis/pom.xml
+++ b/pulsar-io/kinesis/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-io-kinesis</artifactId>
diff --git a/pulsar-io/mongo/pom.xml b/pulsar-io/mongo/pom.xml
index bc1c05d..6cd9383 100644
--- a/pulsar-io/mongo/pom.xml
+++ b/pulsar-io/mongo/pom.xml
@@ -26,7 +26,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-io-mongo</artifactId>
diff --git a/pulsar-io/netty/pom.xml b/pulsar-io/netty/pom.xml
index 7c1234e..09b2b9f 100644
--- a/pulsar-io/netty/pom.xml
+++ b/pulsar-io/netty/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-io-netty</artifactId>
diff --git a/pulsar-io/nsq/pom.xml b/pulsar-io/nsq/pom.xml
index 34d7607..8b1eab8 100644
--- a/pulsar-io/nsq/pom.xml
+++ b/pulsar-io/nsq/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-io-nsq</artifactId>
diff --git a/pulsar-io/pom.xml b/pulsar-io/pom.xml
index 0582b53..2f90671 100644
--- a/pulsar-io/pom.xml
+++ b/pulsar-io/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-io</artifactId>
diff --git a/pulsar-io/rabbitmq/pom.xml b/pulsar-io/rabbitmq/pom.xml
index 15a123a..5338b12 100644
--- a/pulsar-io/rabbitmq/pom.xml
+++ b/pulsar-io/rabbitmq/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-io-rabbitmq</artifactId>
diff --git a/pulsar-io/redis/pom.xml b/pulsar-io/redis/pom.xml
index 95d3d6c..f12fdf5 100644
--- a/pulsar-io/redis/pom.xml
+++ b/pulsar-io/redis/pom.xml
@@ -25,7 +25,7 @@
<parent>
<artifactId>pulsar-io</artifactId>
<groupId>org.apache.pulsar</groupId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-io-redis</artifactId>
diff --git a/pulsar-io/solr/pom.xml b/pulsar-io/solr/pom.xml
index f330dd5..318775f 100644
--- a/pulsar-io/solr/pom.xml
+++ b/pulsar-io/solr/pom.xml
@@ -25,7 +25,7 @@
<parent>
<artifactId>pulsar-io</artifactId>
<groupId>org.apache.pulsar</groupId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<properties>
diff --git a/pulsar-io/twitter/pom.xml b/pulsar-io/twitter/pom.xml
index 6154f17..831aeaa 100644
--- a/pulsar-io/twitter/pom.xml
+++ b/pulsar-io/twitter/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-io</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-io-twitter</artifactId>
diff --git a/pulsar-metadata/pom.xml b/pulsar-metadata/pom.xml
index 21320d4..19872a5 100644
--- a/pulsar-metadata/pom.xml
+++ b/pulsar-metadata/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-proxy/pom.xml b/pulsar-proxy/pom.xml
index 4844431..887425c 100644
--- a/pulsar-proxy/pom.xml
+++ b/pulsar-proxy/pom.xml
@@ -24,7 +24,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-proxy</artifactId>
diff --git a/pulsar-sql/pom.xml b/pulsar-sql/pom.xml
index 21e96db..cc04d70 100644
--- a/pulsar-sql/pom.xml
+++ b/pulsar-sql/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-sql</artifactId>
diff --git a/pulsar-sql/presto-distribution/pom.xml b/pulsar-sql/presto-distribution/pom.xml
index 649e672..f02eba1 100644
--- a/pulsar-sql/presto-distribution/pom.xml
+++ b/pulsar-sql/presto-distribution/pom.xml
@@ -31,7 +31,7 @@
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-presto-distribution</artifactId>
<name>Pulsar SQL :: Pulsar Presto Distribution</name>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<properties>
<jersey.version>2.31</jersey.version>
diff --git a/pulsar-sql/presto-pulsar-plugin/pom.xml b/pulsar-sql/presto-pulsar-plugin/pom.xml
index 512e0e9..1d8ad55 100644
--- a/pulsar-sql/presto-pulsar-plugin/pom.xml
+++ b/pulsar-sql/presto-pulsar-plugin/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-sql</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-presto-connector</artifactId>
diff --git a/pulsar-sql/presto-pulsar/pom.xml b/pulsar-sql/presto-pulsar/pom.xml
index 7478aac..67ddea0 100644
--- a/pulsar-sql/presto-pulsar/pom.xml
+++ b/pulsar-sql/presto-pulsar/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-sql</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-presto-connector-original</artifactId>
diff --git a/pulsar-testclient/pom.xml b/pulsar-testclient/pom.xml
index 9ef82af..354304f 100644
--- a/pulsar-testclient/pom.xml
+++ b/pulsar-testclient/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-transaction/common/pom.xml b/pulsar-transaction/common/pom.xml
index 846df98..82479f7 100644
--- a/pulsar-transaction/common/pom.xml
+++ b/pulsar-transaction/common/pom.xml
@@ -27,7 +27,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-transaction-parent</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-transaction-common</artifactId>
diff --git a/pulsar-transaction/coordinator/pom.xml b/pulsar-transaction/coordinator/pom.xml
index 7eca8ea..494fc80 100644
--- a/pulsar-transaction/coordinator/pom.xml
+++ b/pulsar-transaction/coordinator/pom.xml
@@ -27,7 +27,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-transaction-parent</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-transaction-coordinator</artifactId>
diff --git a/pulsar-transaction/pom.xml b/pulsar-transaction/pom.xml
index 3d7c06d..748e96e 100644
--- a/pulsar-transaction/pom.xml
+++ b/pulsar-transaction/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-transaction-parent</artifactId>
diff --git a/pulsar-websocket/pom.xml b/pulsar-websocket/pom.xml
index 764ccff..adf68c7 100644
--- a/pulsar-websocket/pom.xml
+++ b/pulsar-websocket/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-zookeeper-utils/pom.xml b/pulsar-zookeeper-utils/pom.xml
index 647934f..4b593d5 100644
--- a/pulsar-zookeeper-utils/pom.xml
+++ b/pulsar-zookeeper-utils/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/pulsar-zookeeper/pom.xml b/pulsar-zookeeper/pom.xml
index 927e587..a6f6529 100644
--- a/pulsar-zookeeper/pom.xml
+++ b/pulsar-zookeeper/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/testmocks/pom.xml b/testmocks/pom.xml
index 6969e2b..9c78d16 100644
--- a/testmocks/pom.xml
+++ b/testmocks/pom.xml
@@ -25,7 +25,7 @@
<parent>
<artifactId>pulsar</artifactId>
<groupId>org.apache.pulsar</groupId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>testmocks</artifactId>
diff --git a/tests/bc_2_0_0/pom.xml b/tests/bc_2_0_0/pom.xml
index e189cb6..4924abc 100644
--- a/tests/bc_2_0_0/pom.xml
+++ b/tests/bc_2_0_0/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar.tests</groupId>
<artifactId>tests-parent</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>bc_2_0_0</artifactId>
diff --git a/tests/bc_2_0_1/pom.xml b/tests/bc_2_0_1/pom.xml
index ac4f820..b770a29 100644
--- a/tests/bc_2_0_1/pom.xml
+++ b/tests/bc_2_0_1/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar.tests</groupId>
<artifactId>tests-parent</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>bc_2_0_1</artifactId>
diff --git a/tests/docker-images/java-test-functions/pom.xml b/tests/docker-images/java-test-functions/pom.xml
index 09f53f9..748d53b 100644
--- a/tests/docker-images/java-test-functions/pom.xml
+++ b/tests/docker-images/java-test-functions/pom.xml
@@ -23,7 +23,7 @@
<parent>
<groupId>org.apache.pulsar.tests</groupId>
<artifactId>docker-images</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>java-test-functions</artifactId>
diff --git a/tests/docker-images/latest-version-image/pom.xml b/tests/docker-images/latest-version-image/pom.xml
index ec429c0..bfb6e6b 100644
--- a/tests/docker-images/latest-version-image/pom.xml
+++ b/tests/docker-images/latest-version-image/pom.xml
@@ -23,7 +23,7 @@
<parent>
<groupId>org.apache.pulsar.tests</groupId>
<artifactId>docker-images</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>latest-version-image</artifactId>
diff --git a/tests/docker-images/pom.xml b/tests/docker-images/pom.xml
index 67105ab..5ee8f20 100644
--- a/tests/docker-images/pom.xml
+++ b/tests/docker-images/pom.xml
@@ -26,7 +26,7 @@
<parent>
<groupId>org.apache.pulsar.tests</groupId>
<artifactId>tests-parent</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>docker-images</artifactId>
<name>Apache Pulsar :: Tests :: Docker Images</name>
diff --git a/tests/integration/pom.xml b/tests/integration/pom.xml
index d9f8234..b147b46 100644
--- a/tests/integration/pom.xml
+++ b/tests/integration/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar.tests</groupId>
<artifactId>tests-parent</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>integration</artifactId>
diff --git a/tests/pom.xml b/tests/pom.xml
index 528abc7..59fc9c2 100644
--- a/tests/pom.xml
+++ b/tests/pom.xml
@@ -26,7 +26,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<groupId>org.apache.pulsar.tests</groupId>
<artifactId>tests-parent</artifactId>
diff --git a/tests/pulsar-client-admin-shade-test/pom.xml b/tests/pulsar-client-admin-shade-test/pom.xml
index 6a0226d..9c05735 100644
--- a/tests/pulsar-client-admin-shade-test/pom.xml
+++ b/tests/pulsar-client-admin-shade-test/pom.xml
@@ -26,7 +26,7 @@
<parent>
<groupId>org.apache.pulsar.tests</groupId>
<artifactId>tests-parent</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-client-admin-shade-test</artifactId>
diff --git a/tests/pulsar-client-all-shade-test/pom.xml b/tests/pulsar-client-all-shade-test/pom.xml
index 102c471..b2d045e 100644
--- a/tests/pulsar-client-all-shade-test/pom.xml
+++ b/tests/pulsar-client-all-shade-test/pom.xml
@@ -26,7 +26,7 @@
<parent>
<groupId>org.apache.pulsar.tests</groupId>
<artifactId>tests-parent</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-client-all-shade-test</artifactId>
diff --git a/tests/pulsar-client-shade-test/pom.xml b/tests/pulsar-client-shade-test/pom.xml
index 86f7a05..b7d04b6 100644
--- a/tests/pulsar-client-shade-test/pom.xml
+++ b/tests/pulsar-client-shade-test/pom.xml
@@ -27,7 +27,7 @@
<parent>
<groupId>org.apache.pulsar.tests</groupId>
<artifactId>tests-parent</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
</parent>
<artifactId>pulsar-client-shade-test</artifactId>
diff --git a/tiered-storage/file-system/pom.xml b/tiered-storage/file-system/pom.xml
index 6bb625a..fc5657b 100644
--- a/tiered-storage/file-system/pom.xml
+++ b/tiered-storage/file-system/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>tiered-storage-parent</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/tiered-storage/jcloud/pom.xml b/tiered-storage/jcloud/pom.xml
index 3a36d40..462e706 100644
--- a/tiered-storage/jcloud/pom.xml
+++ b/tiered-storage/jcloud/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>tiered-storage-parent</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<relativePath>..</relativePath>
</parent>
diff --git a/tiered-storage/pom.xml b/tiered-storage/pom.xml
index 4856b29..72d426b 100644
--- a/tiered-storage/pom.xml
+++ b/tiered-storage/pom.xml
@@ -25,7 +25,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar</artifactId>
- <version>2.7.2</version>
+ <version>2.7.2_1.0.0</version>
<relativePath>..</relativePath>
</parent>
[pulsar] 19/46: [Branch-2.7] Fix deadlock on Monitoring thread
blocked by LeaderService.isLeader() (#10512)
Posted by eo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 56e127d30d90906e866ceed544dc882aed96e042
Author: Yong Zhang <zh...@gmail.com>
AuthorDate: Sun May 9 08:37:33 2021 +0800
[Branch-2.7] Fix deadlock on Monitoring thread blocked by LeaderService.isLeader() (#10512)
---
Fixes #10235
According to #10235, when `LeaderService` is changing leadership status
(like losing leadership, or becoming a leader), the `LeaderService` will
be locked with `synchronized` block. Which will block other threads if
calling `LeaderService.isLeader()`. This PR changes `ClusterServiceCoordinator`
and `WorkerStatsManager` to check if is leader from `MembershipManager`,
which will not block other threads if `LeaderService` is at `synchronized` block.
Also, this PR will not resolve the root cause of #10235, since there is
lack of context about blocked reader for the `FunctionAssignmentTailer`.
- [ ] Make sure that the change passes the CI checks.
---
Original PR is https://github.com/apache/pulsar/pull/10502
---
.../apache/pulsar/functions/worker/ClusterServiceCoordinator.java | 8 +++++---
.../java/org/apache/pulsar/functions/worker/WorkerService.java | 7 ++++++-
.../org/apache/pulsar/functions/worker/WorkerStatsManager.java | 6 +++++-
.../pulsar/functions/worker/ClusterServiceCoordinatorTest.java | 6 +++++-
4 files changed, 21 insertions(+), 6 deletions(-)
diff --git a/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/ClusterServiceCoordinator.java b/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/ClusterServiceCoordinator.java
index c2bde9d..c1682a8 100644
--- a/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/ClusterServiceCoordinator.java
+++ b/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/ClusterServiceCoordinator.java
@@ -29,6 +29,7 @@ import lombok.extern.slf4j.Slf4j;
import java.util.HashMap;
import java.util.Map;
+import java.util.function.Supplier;
@Slf4j
public class ClusterServiceCoordinator implements AutoCloseable {
@@ -49,10 +50,12 @@ public class ClusterServiceCoordinator implements AutoCloseable {
private final Map<String, TimerTaskInfo> tasks = new HashMap<>();
private final ScheduledExecutorService executor;
private final LeaderService leaderService;
+ private final Supplier<Boolean> isLeader;
- public ClusterServiceCoordinator(String workerId, LeaderService leaderService) {
+ public ClusterServiceCoordinator(String workerId, LeaderService leaderService, Supplier<Boolean> isLeader) {
this.workerId = workerId;
this.leaderService = leaderService;
+ this.isLeader = isLeader;
this.executor = Executors.newSingleThreadScheduledExecutor(
new ThreadFactoryBuilder().setNameFormat("cluster-service-coordinator-timer").build());
}
@@ -67,8 +70,7 @@ public class ClusterServiceCoordinator implements AutoCloseable {
TimerTaskInfo timerTaskInfo = entry.getValue();
String taskName = entry.getKey();
this.executor.scheduleAtFixedRate(() -> {
- boolean isLeader = leaderService.isLeader();
- if (isLeader) {
+ if (isLeader.get()) {
try {
timerTaskInfo.getTask().run();
} catch (Exception e) {
diff --git a/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/WorkerService.java b/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/WorkerService.java
index 4af8aad..d0218b4 100644
--- a/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/WorkerService.java
+++ b/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/WorkerService.java
@@ -41,6 +41,7 @@ import org.apache.pulsar.client.api.PulsarClientException;
import java.net.URI;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
+import java.util.function.Supplier;
/**
* A service component contains everything to run a worker except rest server.
@@ -228,6 +229,8 @@ public class WorkerService {
log.info("/** Initializing Runtime Manager **/");
MessageId lastAssignmentMessageId = functionRuntimeManager.initialize();
+ Supplier<Boolean> checkIsStillLeader =
+ () -> membershipManager.getLeader().getWorkerId().equals(workerConfig.getWorkerId());
// Setting references to managers in scheduler
schedulerManager.setFunctionMetaDataManager(functionMetaDataManager);
@@ -250,7 +253,8 @@ public class WorkerService {
// Starting cluster services
this.clusterServiceCoordinator = new ClusterServiceCoordinator(
workerConfig.getWorkerId(),
- leaderService);
+ leaderService,
+ checkIsStillLeader);
clusterServiceCoordinator.addTask("membership-monitor",
workerConfig.getFailureCheckFreqMs(),
@@ -291,6 +295,7 @@ public class WorkerService {
workerStatsManager.setFunctionRuntimeManager(functionRuntimeManager);
workerStatsManager.setFunctionMetaDataManager(functionMetaDataManager);
workerStatsManager.setLeaderService(leaderService);
+ workerStatsManager.setIsLeader(checkIsStillLeader);
workerStatsManager.startupTimeEnd();
} catch (Throwable t) {
log.error("Error Starting up in worker", t);
diff --git a/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/WorkerStatsManager.java b/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/WorkerStatsManager.java
index 703b131..78cfdb1 100644
--- a/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/WorkerStatsManager.java
+++ b/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/WorkerStatsManager.java
@@ -33,6 +33,7 @@ import org.apache.pulsar.functions.proto.Function;
import java.io.IOException;
import java.io.StringWriter;
import java.util.List;
+import java.util.function.Supplier;
public class WorkerStatsManager {
@@ -66,6 +67,9 @@ public class WorkerStatsManager {
@Setter
private LeaderService leaderService;
+ @Setter
+ private Supplier<Boolean> isLeader;
+
private CollectorRegistry collectorRegistry = new CollectorRegistry();
private final Summary statWorkerStartupTime;
@@ -279,7 +283,7 @@ public class WorkerStatsManager {
}
private void generateLeaderMetrics(StringWriter stream) {
- if (leaderService.isLeader()) {
+ if (isLeader.get()) {
List<Function.FunctionMetaData> metadata = functionMetaDataManager.getAllFunctionMetaData();
// get total number functions
diff --git a/pulsar-functions/worker/src/test/java/org/apache/pulsar/functions/worker/ClusterServiceCoordinatorTest.java b/pulsar-functions/worker/src/test/java/org/apache/pulsar/functions/worker/ClusterServiceCoordinatorTest.java
index a9e33da..63246e5 100644
--- a/pulsar-functions/worker/src/test/java/org/apache/pulsar/functions/worker/ClusterServiceCoordinatorTest.java
+++ b/pulsar-functions/worker/src/test/java/org/apache/pulsar/functions/worker/ClusterServiceCoordinatorTest.java
@@ -30,6 +30,8 @@ import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.ThreadFactory;
import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+
import org.apache.pulsar.functions.worker.executor.MockExecutorController;
import org.powermock.api.mockito.PowerMockito;
import org.powermock.core.classloader.annotations.PowerMockIgnore;
@@ -56,6 +58,7 @@ public class ClusterServiceCoordinatorTest {
private ClusterServiceCoordinator coordinator;
private ScheduledExecutorService mockExecutor;
private MockExecutorController mockExecutorController;
+ private Supplier<Boolean> checkIsStillLeader;
@BeforeMethod
public void setup() throws Exception {
@@ -71,7 +74,8 @@ public class ClusterServiceCoordinatorTest {
).thenReturn(mockExecutor);
this.leaderService = mock(LeaderService.class);
- this.coordinator = new ClusterServiceCoordinator("test-coordinator", leaderService);
+ this.checkIsStillLeader = () -> leaderService.isLeader();
+ this.coordinator = new ClusterServiceCoordinator("test-coordinator", leaderService, checkIsStillLeader);
}
[pulsar] 24/46: Implement GenericObject - Allow GenericRecord to
wrap any Java Object (#10057)
Posted by eo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 23b2859389910698779c0a6d726c496429fc05f9
Author: Enrico Olivelli <eo...@gmail.com>
AuthorDate: Thu Apr 8 03:14:46 2021 +0200
Implement GenericObject - Allow GenericRecord to wrap any Java Object (#10057)
Contents:
- introduce new high level interface GenericObject, that represents every logical value on the Pulsar topic
- allow AutoConsumeSchema to deal with every Schema type
- handle schema less topics with AutoConsumeSchema (return Bytes schema and byte[] payload)
- rename `getNativeRecord` to `getNativeObject`
How it works:
- in case of non struct schema, like for primitives and for KeyValue, wrap the result into a PrimitiveRecord class, that is an implementation of GenericRecord that wraps a Object and a SchemaType, it returns an empty list of "fields"
Intended audience:
- allow users to implement Sink<GenericObject>, that is to write Sinks that work with every Schema type (the patch for Pulsar IO will follow up as a separate PR)
```
interface GenericObject {
Object getNativeObject();
SchemaType getSchemaType();
}
interface GenericRecord extends GenericObject
```
---
.../client/api/SimpleProducerConsumerTest.java | 49 ++++++++++
.../java/org/apache/pulsar/schema/SchemaTest.java | 101 +++++++++++++++++++++
.../java/org/apache/pulsar/client/api/Schema.java | 6 +-
.../pulsar/client/api/schema/GenericObject.java | 50 ++++++++++
.../pulsar/client/api/schema/GenericRecord.java | 28 +++++-
.../client/impl/schema/AutoConsumeSchema.java | 92 +++++++++++++------
.../client/impl/schema/GenericObjectWrapper.java | 98 ++++++++++++++++++++
.../impl/schema/generic/GenericAvroRecord.java | 9 ++
.../impl/schema/generic/GenericJsonRecord.java | 10 ++
.../generic/GenericProtobufNativeRecord.java | 10 ++
.../impl/schema/generic/GenericJsonRecordTest.java | 51 ++++++++++-
.../generic/GenericProtobufNativeReaderTest.java | 13 +++
12 files changed, 483 insertions(+), 34 deletions(-)
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleProducerConsumerTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleProducerConsumerTest.java
index 4db81d3..e522908 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleProducerConsumerTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleProducerConsumerTest.java
@@ -3883,6 +3883,55 @@ public class SimpleProducerConsumerTest extends ProducerConsumerBase {
GenericRecord res = consumer.receive().getValue();
consumer.close();
+ assertEquals(SchemaType.AVRO, res.getSchemaType());
+ org.apache.avro.generic.GenericRecord nativeRecord = (org.apache.avro.generic.GenericRecord) res.getNativeObject();
+ org.apache.avro.Schema schema = nativeRecord.getSchema();
+ for (org.apache.pulsar.client.api.schema.Field f : res.getFields()) {
+ log.info("field {} {}", f.getName(), res.getField(f));
+ assertEquals("field", f.getName());
+ assertEquals("aaaaaaaaaaaaaaaaaaaaaaaaa", res.getField(f));
+ assertEquals("aaaaaaaaaaaaaaaaaaaaaaaaa", nativeRecord.get(f.getName()).toString());
+ }
+
+ assertEquals(1, res.getFields().size());
+ }
+
+ @DataProvider(name = "avroSchemaProvider")
+ public static Object[] avroSchemaProvider() {
+ return new Object[]{Schema.AVRO(MyBean.class), Schema.JSON(MyBean.class)};
+ }
+
+ @Test(dataProvider = "avroSchemaProvider")
+ public void testAccessAvroSchemaMetadata(Schema<MyBean> schema) throws Exception {
+ log.info("-- Starting {} test --", methodName);
+
+ final String topic = "persistent://my-property/my-ns/accessSchema";
+ Consumer<GenericRecord> consumer = pulsarClient.newConsumer(Schema.AUTO_CONSUME())
+ .topic(topic)
+ .subscriptionName("testsub")
+ .subscribe();
+
+ Producer<MyBean> producer = pulsarClient
+ .newProducer(schema)
+ .topic(topic)
+ .create();
+ MyBean payload = new MyBean();
+ payload.setField("aaaaaaaaaaaaaaaaaaaaaaaaa");
+ producer.send(payload);
+ producer.close();
+
+ GenericRecord res = consumer.receive().getValue();
+ consumer.close();
+ assertEquals(schema.getSchemaInfo().getType(), res.getSchemaType());
+ org.apache.avro.generic.GenericRecord nativeAvroRecord = null;
+ JsonNode nativeJsonRecord = null;
+ if (schema.getSchemaInfo().getType() == SchemaType.AVRO) {
+ nativeAvroRecord = (org.apache.avro.generic.GenericRecord) res.getNativeObject();
+ assertNotNull(nativeAvroRecord);
+ } else {
+ nativeJsonRecord = (JsonNode) res.getNativeObject();
+ assertNotNull(nativeJsonRecord);
+ }
for (org.apache.pulsar.client.api.schema.Field f : res.getFields()) {
log.info("field {} {}", f.getName(), res.getField(f));
assertEquals("field", f.getName());
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/schema/SchemaTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/schema/SchemaTest.java
index d72cbf1..fba8108 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/schema/SchemaTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/schema/SchemaTest.java
@@ -26,6 +26,7 @@ import static org.testng.Assert.assertNotNull;
import static org.testng.Assert.assertTrue;
import static org.testng.Assert.fail;
import com.google.common.collect.Sets;
+import java.nio.charset.StandardCharsets;
import java.util.Collections;
import java.util.List;
import java.util.concurrent.CompletableFuture;
@@ -204,6 +205,106 @@ public class SchemaTest extends MockedPulsarServiceBaseTest {
}
@Test
+ public void testStringSchema() throws Exception {
+ final String tenant = PUBLIC_TENANT;
+ final String namespace = "test-namespace-" + randomName(16);
+ final String topicName = "test-string-schema";
+
+ final String topic = TopicName.get(
+ TopicDomain.persistent.value(),
+ tenant,
+ namespace,
+ topicName).toString();
+
+ admin.namespaces().createNamespace(
+ tenant + "/" + namespace,
+ Sets.newHashSet(CLUSTER_NAME));
+
+ admin.topics().createPartitionedTopic(topic, 2);
+
+ Producer<String> producer = pulsarClient
+ .newProducer(Schema.STRING)
+ .topic(topic)
+ .create();
+
+ Consumer<String> consumer = pulsarClient.newConsumer(Schema.STRING)
+ .subscriptionName("test-sub")
+ .topic(topic)
+ .subscribe();
+
+ // use GenericRecord even for primitive types
+ // it will be a PrimitiveRecord
+ Consumer<GenericRecord> consumer2 = pulsarClient.newConsumer(Schema.AUTO_CONSUME())
+ .subscriptionName("test-sub3")
+ .topic(topic)
+ .subscribe();
+
+ producer.send("foo");
+
+ Message<String> message = consumer.receive();
+ Message<GenericRecord> message3 = consumer2.receive();
+
+ assertEquals("foo", message.getValue());
+ assertEquals(message3.getValue().getClass().getName(), "org.apache.pulsar.client.impl.schema.GenericObjectWrapper");
+ assertEquals(SchemaType.STRING, message3.getValue().getSchemaType());
+ assertEquals("foo", message3.getValue().getNativeObject());
+
+ producer.close();
+ consumer.close();
+ consumer2.close();
+ }
+
+ @Test
+ public void testUseAutoConsumeWithSchemalessTopic() throws Exception {
+ final String tenant = PUBLIC_TENANT;
+ final String namespace = "test-namespace-" + randomName(16);
+ final String topicName = "test-schemaless";
+
+ final String topic = TopicName.get(
+ TopicDomain.persistent.value(),
+ tenant,
+ namespace,
+ topicName).toString();
+
+ admin.namespaces().createNamespace(
+ tenant + "/" + namespace,
+ Sets.newHashSet(CLUSTER_NAME));
+
+ admin.topics().createPartitionedTopic(topic, 2);
+
+ Producer<byte[]> producer = pulsarClient
+ .newProducer()
+ .topic(topic)
+ .create();
+
+ Consumer<byte[]> consumer = pulsarClient.newConsumer()
+ .subscriptionName("test-sub")
+ .topic(topic)
+ .subscribe();
+
+ // use GenericRecord even for primitive types
+ // it will be a PrimitiveRecord
+ Consumer<GenericRecord> consumer2 = pulsarClient.newConsumer(Schema.AUTO_CONSUME())
+ .subscriptionName("test-sub3")
+ .topic(topic)
+ .subscribe();
+
+ producer.send("foo".getBytes(StandardCharsets.UTF_8));
+
+ Message<byte[]> message = consumer.receive();
+ Message<GenericRecord> message3 = consumer2.receive();
+
+ assertEquals("foo".getBytes(StandardCharsets.UTF_8), message.getValue());
+ assertEquals(message3.getValue().getClass().getName(), "org.apache.pulsar.client.impl.schema.GenericObjectWrapper");
+ assertEquals(SchemaType.BYTES, message3.getValue().getSchemaType());
+ assertEquals("foo".getBytes(StandardCharsets.UTF_8), message3.getValue().getNativeObject());
+
+ producer.close();
+ consumer.close();
+ consumer2.close();
+ }
+
+ @Test
public void testIsUsingAvroSchemaParser() {
for (SchemaType value : SchemaType.values()) {
if (value == SchemaType.AVRO || value == SchemaType.JSON || value == SchemaType.PROTOBUF) {
diff --git a/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Schema.java b/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Schema.java
index b5bea77..9b41836 100644
--- a/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Schema.java
+++ b/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Schema.java
@@ -28,6 +28,7 @@ import java.time.LocalTime;
import java.util.Date;
import org.apache.pulsar.client.api.schema.GenericRecord;
import org.apache.pulsar.client.api.schema.GenericSchema;
+import org.apache.pulsar.client.api.schema.GenericObject;
import org.apache.pulsar.client.api.schema.SchemaDefinition;
import org.apache.pulsar.client.api.schema.SchemaInfoProvider;
import org.apache.pulsar.client.internal.DefaultImplementation;
@@ -361,9 +362,8 @@ public interface Schema<T> extends Cloneable{
* Create a schema instance that automatically deserialize messages
* based on the current topic schema.
*
- * <p>The messages values are deserialized into a {@link GenericRecord} object.
- *
- * <p>Currently this is only supported with Avro and JSON schema types.
+ * <p>The messages values are deserialized into a {@link GenericRecord} object,
+ * that extends the {@link GenericObject} interface.
*
* @return the auto schema instance
*/
diff --git a/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/schema/GenericObject.java b/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/schema/GenericObject.java
new file mode 100644
index 0000000..db6d9e8
--- /dev/null
+++ b/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/schema/GenericObject.java
@@ -0,0 +1,50 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.api.schema;
+
+import org.apache.pulsar.common.schema.SchemaType;
+
+/**
+ * This is an abstraction over the logical value that is store into a Message.
+ * Pulsar decodes the payload of the Message using the Schema that is configured for the topic.
+ */
+public interface GenericObject {
+
+ /**
+ * Return the schema tyoe.
+ *
+ * @return the schema type
+ * @throws UnsupportedOperationException if this feature is not implemented
+ * @see SchemaType#BYTES when the topic has no schema information
+ * @see SchemaType#STRING
+ * @see SchemaType#AVRO
+ * @see SchemaType#PROTOBUF_NATIVE
+ * @see SchemaType#JSON
+ */
+ SchemaType getSchemaType();
+
+ /**
+ * Return the internal native representation of the Object,
+ * like a AVRO GenericRecord, a String or a byte[].
+ *
+ * @return the decoded object
+ * @throws UnsupportedOperationException if the operation is not supported
+ */
+ Object getNativeObject();
+}
diff --git a/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/schema/GenericRecord.java b/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/schema/GenericRecord.java
index d380b3c..1dda99f 100644
--- a/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/schema/GenericRecord.java
+++ b/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/schema/GenericRecord.java
@@ -27,7 +27,7 @@ import org.apache.pulsar.common.classification.InterfaceStability;
*/
@InterfaceAudience.Public
@InterfaceStability.Stable
-public interface GenericRecord {
+public interface GenericRecord extends GenericObject {
/**
* Return schema version.
@@ -61,4 +61,30 @@ public interface GenericRecord {
*/
Object getField(String fieldName);
+ /**
+ * Return the schema tyoe.
+ *
+ * @return the schema type
+ * @throws UnsupportedOperationException if this feature is not implemented
+ * @see SchemaType#AVRO
+ * @see SchemaType#PROTOBUF_NATIVE
+ * @see SchemaType#JSON
+ */
+ @Override
+ default SchemaType getSchemaType() {
+ throw new UnsupportedOperationException();
+ }
+
+ /**
+ * Return the internal native representation of the Record,
+ * like a AVRO GenericRecord.
+ *
+ * @return the internal representation of the record
+ * @throws UnsupportedOperationException if the operation is not supported
+ */
+ @Override
+ default Object getNativeObject() {
+ throw new UnsupportedOperationException();
+ }
+
}
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java
index 405b9cc..6ab92ad 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java
@@ -22,11 +22,9 @@ import lombok.extern.slf4j.Slf4j;
import org.apache.pulsar.client.api.Schema;
import org.apache.pulsar.client.api.SchemaSerializationException;
import org.apache.pulsar.client.api.schema.GenericRecord;
-import org.apache.pulsar.client.api.schema.GenericSchema;
import org.apache.pulsar.client.api.schema.SchemaInfoProvider;
-import org.apache.pulsar.client.impl.schema.generic.GenericAvroSchema;
-import org.apache.pulsar.client.impl.schema.generic.GenericJsonSchema;
import org.apache.pulsar.client.impl.schema.generic.GenericProtobufNativeSchema;
+import org.apache.pulsar.client.impl.schema.generic.GenericSchemaImpl;
import org.apache.pulsar.common.schema.KeyValue;
import org.apache.pulsar.common.schema.SchemaInfo;
@@ -35,12 +33,12 @@ import java.util.concurrent.ExecutionException;
import static com.google.common.base.Preconditions.checkState;
/**
- * Auto detect schema.
+ * Auto detect schema, returns only GenericRecord instances.
*/
@Slf4j
public class AutoConsumeSchema implements Schema<GenericRecord> {
- private Schema<GenericRecord> schema;
+ private Schema<?> schema;
private String topicName;
@@ -48,7 +46,7 @@ public class AutoConsumeSchema implements Schema<GenericRecord> {
private SchemaInfoProvider schemaInfoProvider;
- public void setSchema(Schema<GenericRecord> schema) {
+ public void setSchema(Schema<?> schema) {
this.schema = schema;
}
@@ -64,15 +62,15 @@ public class AutoConsumeSchema implements Schema<GenericRecord> {
}
@Override
- public boolean supportSchemaVersioning() {
- return true;
- }
-
- @Override
public byte[] encode(GenericRecord message) {
ensureSchemaInitialized();
- return schema.encode(message);
+ throw new UnsupportedOperationException("AutoConsumeSchema is not intended to be used for encoding");
+ }
+
+ @Override
+ public boolean supportSchemaVersioning() {
+ return schema == null || schema.supportSchemaVersioning();
}
@Override
@@ -81,20 +79,25 @@ public class AutoConsumeSchema implements Schema<GenericRecord> {
SchemaInfo schemaInfo = null;
try {
schemaInfo = schemaInfoProvider.getLatestSchema().get();
+ if (schemaInfo == null) {
+ // schemaless topic
+ schemaInfo = BytesSchema.of().getSchemaInfo();
+ }
} catch (InterruptedException | ExecutionException e ) {
if (e instanceof InterruptedException) {
Thread.currentThread().interrupt();
}
- log.error("Con't get last schema for topic {} use AutoConsumeSchema", topicName);
+ log.error("Can't get last schema for topic {} use AutoConsumeSchema", topicName);
throw new SchemaSerializationException(e.getCause());
}
+ // schemaInfo null means that there is no schema attached to the topic.
schema = generateSchema(schemaInfo);
schema.setSchemaInfoProvider(schemaInfoProvider);
log.info("Configure {} schema for topic {} : {}",
componentName, topicName, schemaInfo.getSchemaDefinition());
}
ensureSchemaInitialized();
- return schema.decode(bytes, schemaVersion);
+ return adapt(schema.decode(bytes, schemaVersion), schemaVersion);
}
@Override
@@ -115,18 +118,13 @@ public class AutoConsumeSchema implements Schema<GenericRecord> {
}
@Override
- public boolean requireFetchingSchemaInfo() {
- return true;
- }
-
- @Override
public void configureSchemaInfo(String topicName,
String componentName,
SchemaInfo schemaInfo) {
this.topicName = topicName;
this.componentName = componentName;
if (schemaInfo != null) {
- GenericSchema genericSchema = generateSchema(schemaInfo);
+ Schema<?> genericSchema = generateSchema(schemaInfo);
setSchema(genericSchema);
log.info("Configure {} schema for topic {} : {}",
componentName, topicName, schemaInfo.getSchemaDefinition());
@@ -134,6 +132,7 @@ public class AutoConsumeSchema implements Schema<GenericRecord> {
}
@Override
+<<<<<<< HEAD
public Schema<GenericRecord> clone() {
Schema<GenericRecord> schema = Schema.AUTO_CONSUME();
if (this.schema != null) {
@@ -147,20 +146,27 @@ public class AutoConsumeSchema implements Schema<GenericRecord> {
return schema;
}
- private GenericSchema generateSchema(SchemaInfo schemaInfo) {
+ public Optional<Object> getNativeSchema() {
+ ensureSchemaInitialized();
+ if (schema == null) {
+ return Optional.empty();
+ } else {
+ return schema.getNativeSchema();
+ }
+ }
+
+ private Schema<?> generateSchema(SchemaInfo schemaInfo) {
// when using `AutoConsumeSchema`, we use the schema associated with the messages as schema reader
// to decode the messages.
final boolean useProvidedSchemaAsReaderSchema = false;
switch (schemaInfo.getType()) {
case JSON:
- return GenericJsonSchema.of(schemaInfo,useProvidedSchemaAsReaderSchema);
case AVRO:
- return GenericAvroSchema.of(schemaInfo,useProvidedSchemaAsReaderSchema);
+ return GenericSchemaImpl.of(schemaInfo,useProvidedSchemaAsReaderSchema);
case PROTOBUF_NATIVE:
return GenericProtobufNativeSchema.of(schemaInfo, useProvidedSchemaAsReaderSchema);
default:
- throw new IllegalArgumentException("Currently auto consume works for type '"
- + schemaInfo.getType() + "' is not supported yet");
+ return getSchema(schemaInfo);
}
}
@@ -199,21 +205,49 @@ public class AutoConsumeSchema implements Schema<GenericRecord> {
case LOCAL_DATE_TIME:
return LocalDateTimeSchema.of();
case JSON:
- return GenericJsonSchema.of(schemaInfo);
case AVRO:
- return GenericAvroSchema.of(schemaInfo);
+ return GenericSchemaImpl.of(schemaInfo);
case PROTOBUF_NATIVE:
return GenericProtobufNativeSchema.of(schemaInfo);
case KEY_VALUE:
KeyValue<SchemaInfo, SchemaInfo> kvSchemaInfo =
- KeyValueSchemaInfo.decodeKeyValueSchemaInfo(schemaInfo);
+ KeyValueSchemaInfo.decodeKeyValueSchemaInfo(schemaInfo);
Schema<?> keySchema = getSchema(kvSchemaInfo.getKey());
Schema<?> valueSchema = getSchema(kvSchemaInfo.getValue());
return KeyValueSchema.of(keySchema, valueSchema,
KeyValueSchemaInfo.decodeKeyValueEncodingType(schemaInfo));
default:
throw new IllegalArgumentException("Retrieve schema instance from schema info for type '"
- + schemaInfo.getType() + "' is not supported yet");
+ + schemaInfo.getType() + "' is not supported yet");
+ }
+ }
+
+ public Schema<GenericRecord> clone() {
+ Schema<GenericRecord> schema = new AutoConsumeSchema();
+ if (this.schema != null) {
+ schema.configureSchemaInfo(topicName, componentName, this.schema.getSchemaInfo());
+ } else {
+ schema.configureSchemaInfo(topicName, componentName, null);
+ }
+ if (schemaInfoProvider != null) {
+ schema.setSchemaInfoProvider(schemaInfoProvider);
+ }
+ return schema;
+ }
+
+ @Override
+ public boolean requireFetchingSchemaInfo() {
+ return true;
+ }
+
+ protected GenericRecord adapt(Object value, byte[] schemaVersion) {
+ if (value instanceof GenericRecord) {
+ return (GenericRecord) value;
+ }
+ if (this.schema == null) {
+ throw new IllegalStateException("Cannot decode a message without schema");
}
+ return GenericObjectWrapper.of(value,
+ this.schema.getSchemaInfo().getType(), schemaVersion);
}
}
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/GenericObjectWrapper.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/GenericObjectWrapper.java
new file mode 100644
index 0000000..38e515c
--- /dev/null
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/GenericObjectWrapper.java
@@ -0,0 +1,98 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.impl.schema;
+
+import org.apache.pulsar.client.api.schema.Field;
+import org.apache.pulsar.client.api.schema.GenericRecord;
+import org.apache.pulsar.common.classification.InterfaceAudience;
+import org.apache.pulsar.common.classification.InterfaceStability;
+import org.apache.pulsar.common.schema.SchemaType;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+
+/**
+ * Implementation of GenericRecord that wraps objects of non Struct types.
+ */
+@InterfaceAudience.Private
+@InterfaceStability.Evolving
+class GenericObjectWrapper implements GenericRecord {
+
+ private final Object nativeObject;
+ private final SchemaType schemaType;
+ private final byte[] schemaVersion;
+
+ static GenericObjectWrapper of(Object nativeObject, SchemaType schemaType, byte[] schemaVersion) {
+ return new GenericObjectWrapper(nativeObject, schemaType, schemaVersion);
+ }
+
+ private GenericObjectWrapper(Object nativeObject, SchemaType schemaType, byte[] schemaVersion) {
+ this.nativeObject = nativeObject;
+ this.schemaType = Objects.requireNonNull(schemaType, "SchemaType is required");
+ this.schemaVersion = schemaVersion;
+ }
+
+ @Override
+ public byte[] getSchemaVersion() {
+ return schemaVersion;
+ }
+
+ @Override
+ public List<Field> getFields() {
+ return Collections.emptyList();
+ }
+
+ @Override
+ public Object getField(String fieldName) {
+ return null;
+ }
+
+ @Override
+ public SchemaType getSchemaType() {
+ return schemaType;
+ }
+
+ @Override
+ public Object getNativeObject() {
+ return nativeObject;
+ }
+
+ @Override
+ public String toString() {
+ return Objects.toString(nativeObject);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hashCode(nativeObject);
+ }
+
+ @Override
+ public boolean equals(Object other) {
+ if (!(other instanceof GenericObjectWrapper)) {
+ return false;
+ }
+ GenericObjectWrapper gw = (GenericObjectWrapper) other;
+ return this.schemaType == gw.schemaType &&
+ Objects.equals(nativeObject, gw.nativeObject) &&
+ Arrays.equals(schemaVersion, gw.schemaVersion);
+ }
+}
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericAvroRecord.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericAvroRecord.java
index 721c06f..767dcdb 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericAvroRecord.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericAvroRecord.java
@@ -65,4 +65,13 @@ public class GenericAvroRecord extends VersionedGenericRecord {
return record;
}
+ @Override
+ public Object getNativeObject() {
+ return record;
+ }
+
+ @Override
+ public SchemaType getSchemaType() {
+ return SchemaType.AVRO;
+ }
}
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericJsonRecord.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericJsonRecord.java
index 85d3e8e..7b4a530 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericJsonRecord.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericJsonRecord.java
@@ -129,4 +129,14 @@ public class GenericJsonRecord extends VersionedGenericRecord {
parser.setValidateDefaults(false);
return parser.parse(schemaJson);
}
+
+ @Override
+ public Object getNativeObject() {
+ return jn;
+ }
+
+ @Override
+ public SchemaType getSchemaType() {
+ return SchemaType.JSON;
+ }
}
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericProtobufNativeRecord.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericProtobufNativeRecord.java
index 51b4cb0..1f94bb1 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericProtobufNativeRecord.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericProtobufNativeRecord.java
@@ -43,4 +43,14 @@ public class GenericProtobufNativeRecord<T extends DynamicMessage> extends Versi
public DynamicMessage getProtobufRecord() {
return record;
}
+
+ @Override
+ public Object getNativeObject() {
+ return record;
+ }
+
+ @Override
+ public SchemaType getSchemaType() {
+ return SchemaType.PROTOBUF_NATIVE;
+ }
}
diff --git a/pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/generic/GenericJsonRecordTest.java b/pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/generic/GenericJsonRecordTest.java
index 6486a24..4a9bdae 100644
--- a/pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/generic/GenericJsonRecordTest.java
+++ b/pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/generic/GenericJsonRecordTest.java
@@ -67,4 +67,53 @@ public class GenericJsonRecordTest {
Object boolValue = record.getField("on");
assertTrue((boolean)boolValue);
}
-}
\ No newline at end of file
+
+ @Data
+ @AllArgsConstructor
+ @NoArgsConstructor
+ private static class Seller {
+ public String state;
+ public String street;
+ public long zipCode;
+ }
+
+ @Data
+ @AllArgsConstructor
+ @NoArgsConstructor
+ private static class PC {
+ public String brand;
+ public String model;
+ public int year;
+ public GPU gpu;
+ public Seller seller;
+ }
+
+ private enum GPU {
+ AMD, NVIDIA
+ }
+
+ @Test
+ public void testEncodeAndDecodeObject() throws JsonProcessingException {
+ // test case from issue https://github.com/apache/pulsar/issues/9605
+ JSONSchema<PC> jsonSchema = JSONSchema.of(SchemaDefinition.<PC>builder().withPojo(PC.class).build());
+ GenericSchema genericJsonSchema = GenericJsonSchema.of(jsonSchema.getSchemaInfo());
+ PC pc = new PC("dell", "alienware", 2021, GPU.AMD,
+ new Seller("WA", "street", 98004));
+ JsonNode jsonNode = ObjectMapperFactory.getThreadLocal().valueToTree(pc);
+ GenericJsonRecord genericJsonRecord =
+ new GenericJsonRecord(null, null, jsonNode, genericJsonSchema.getSchemaInfo());
+ byte[] encoded = genericJsonSchema.encode(genericJsonRecord);
+ PC roundtrippedPc = jsonSchema.decode(encoded);
+ assertEquals(roundtrippedPc, pc);
+ }
+
+ @Test
+ public void testGetNativeRecord() throws Exception{
+ byte[] json = "{\"somefield\":null}".getBytes(UTF_8);
+ GenericJsonRecord record
+ = new GenericJsonReader(Collections.singletonList(new Field("somefield", 0)))
+ .read(json, 0, json.length);
+ assertEquals(SchemaType.JSON, record.getSchemaType());
+ assertSame(record.getNativeObject(), record.getJsonNode());
+ }
+}
diff --git a/pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/generic/GenericProtobufNativeReaderTest.java b/pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/generic/GenericProtobufNativeReaderTest.java
index 985f37d..3a57688 100644
--- a/pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/generic/GenericProtobufNativeReaderTest.java
+++ b/pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/generic/GenericProtobufNativeReaderTest.java
@@ -65,6 +65,19 @@ public class GenericProtobufNativeReaderTest {
}
+ @Test
+ public void testGetNativeRecord() {
+ message = TestMessage.newBuilder().setStringField(STRING_FIELD_VLUE).setDoubleField(DOUBLE_FIELD_VLUE).build();
+ GenericProtobufNativeReader genericProtobufNativeReader = new GenericProtobufNativeReader(genericProtobufNativeSchema.getProtobufNativeSchema());
+ GenericRecord record = genericProtobufNativeReader.read(message.toByteArray());
+ assertEquals(record.getField("stringField"), STRING_FIELD_VLUE);
+ assertEquals(record.getField("doubleField"), DOUBLE_FIELD_VLUE);
+ assertEquals(SchemaType.PROTOBUF_NATIVE, record.getSchemaType());
+ DynamicMessage nativeRecord = (DynamicMessage) record.getNativeObject();
+ assertEquals(nativeRecord.getField(nativeRecord.getDescriptorForType().findFieldByName("stringField")), STRING_FIELD_VLUE);
+ assertEquals(nativeRecord.getField(nativeRecord.getDescriptorForType().findFieldByName("doubleField")), DOUBLE_FIELD_VLUE);
+ }
+
private final static String STRING_FIELD_VLUE = "stringFieldValue";
private final static double DOUBLE_FIELD_VLUE = 0.2D;
[pulsar] 28/46: Sink unwrap internal
AutoConsumeSchema and allow to handle topics with KeyValue schema (#10211)
Posted by eo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit ec3ad85d1b88dae36423c38a1a86651e6fd63c28
Author: Enrico Olivelli <eo...@gmail.com>
AuthorDate: Sat Apr 17 10:14:21 2021 +0200
Sink<GenericObject> unwrap internal AutoConsumeSchema and allow to handle topics with KeyValue schema (#10211)
---
.../pulsar/client/impl/TopicMessageImpl.java | 9 +++
.../pulsar/functions/instance/SinkRecord.java | 8 ++-
.../pulsar/functions/source/PulsarSource.java | 4 ++
.../integration/io/TestGenericObjectSink.java | 29 +++++++--
.../io/PulsarGenericObjectSinkTest.java | 70 ++++++++++++++++++----
5 files changed, 102 insertions(+), 18 deletions(-)
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/TopicMessageImpl.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/TopicMessageImpl.java
index fb62ba3..00869f1 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/TopicMessageImpl.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/TopicMessageImpl.java
@@ -23,6 +23,7 @@ import java.util.Map;
import java.util.Optional;
import org.apache.pulsar.client.api.Message;
import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.Schema;
import org.apache.pulsar.common.api.EncryptionContext;
public class TopicMessageImpl<T> implements Message<T> {
@@ -171,4 +172,12 @@ public class TopicMessageImpl<T> implements Message<T> {
public Message<T> getMessage() {
return msg;
}
+
+ public Schema<T> getSchema() {
+ if (this.msg instanceof MessageImpl) {
+ MessageImpl message = (MessageImpl) this.msg;
+ return message.getSchema();
+ }
+ return null;
+ }
}
diff --git a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/SinkRecord.java b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/SinkRecord.java
index dc1bfd1..38c7036 100644
--- a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/SinkRecord.java
+++ b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/SinkRecord.java
@@ -27,6 +27,7 @@ import lombok.Data;
import lombok.extern.slf4j.Slf4j;
import org.apache.pulsar.client.api.Message;
import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.impl.schema.AutoConsumeSchema;
import org.apache.pulsar.client.impl.schema.KeyValueSchema;
import org.apache.pulsar.functions.api.KVRecord;
import org.apache.pulsar.functions.api.Record;
@@ -95,7 +96,12 @@ public class SinkRecord<T> implements Record<T> {
}
if (sourceRecord.getSchema() != null) {
- return sourceRecord.getSchema();
+ // unwrap actual schema
+ Schema<T> schema = sourceRecord.getSchema();
+ if (schema instanceof AutoConsumeSchema) {
+ schema = (Schema<T>) ((AutoConsumeSchema) schema).getInternalSchema();
+ }
+ return schema;
}
if (sourceRecord instanceof KVRecord) {
diff --git a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/source/PulsarSource.java b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/source/PulsarSource.java
index 235c6ca..e1ec9f6 100644
--- a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/source/PulsarSource.java
+++ b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/source/PulsarSource.java
@@ -34,6 +34,7 @@ import lombok.extern.slf4j.Slf4j;
import org.apache.pulsar.client.api.*;
import org.apache.pulsar.client.impl.MessageImpl;
import org.apache.pulsar.client.impl.MultiTopicsConsumerImpl;
+import org.apache.pulsar.client.impl.TopicMessageImpl;
import org.apache.pulsar.functions.api.Record;
import org.apache.pulsar.common.functions.FunctionConfig;
import org.apache.pulsar.common.util.Reflections;
@@ -132,6 +133,9 @@ public class PulsarSource<T> extends PushSource<T> implements MessageListener<T>
if (message instanceof MessageImpl) {
MessageImpl impl = (MessageImpl) message;
schema = impl.getSchema();
+ } else if (message instanceof TopicMessageImpl) {
+ TopicMessageImpl impl = (TopicMessageImpl) message;
+ schema = impl.getSchema();
}
Record<T> record = PulsarRecord.<T>builder()
.message(message)
diff --git a/tests/docker-images/java-test-functions/src/main/java/org/apache/pulsar/tests/integration/io/TestGenericObjectSink.java b/tests/docker-images/java-test-functions/src/main/java/org/apache/pulsar/tests/integration/io/TestGenericObjectSink.java
index 6b0da0b..c0c4ac2 100644
--- a/tests/docker-images/java-test-functions/src/main/java/org/apache/pulsar/tests/integration/io/TestGenericObjectSink.java
+++ b/tests/docker-images/java-test-functions/src/main/java/org/apache/pulsar/tests/integration/io/TestGenericObjectSink.java
@@ -20,18 +20,13 @@ package org.apache.pulsar.tests.integration.io;
import lombok.extern.slf4j.Slf4j;
import org.apache.pulsar.client.api.schema.GenericObject;
+import org.apache.pulsar.client.impl.schema.KeyValueSchema;
import org.apache.pulsar.common.schema.KeyValue;
import org.apache.pulsar.common.schema.SchemaType;
-import org.apache.pulsar.functions.api.KVRecord;
import org.apache.pulsar.functions.api.Record;
import org.apache.pulsar.io.core.Sink;
import org.apache.pulsar.io.core.SinkContext;
-import org.apache.pulsar.io.core.Source;
-import org.apache.pulsar.io.core.SourceContext;
-
-import java.util.HashMap;
import java.util.Map;
-import java.util.Optional;
@Slf4j
public class TestGenericObjectSink implements Sink<GenericObject> {
@@ -41,10 +36,32 @@ public class TestGenericObjectSink implements Sink<GenericObject> {
}
public void write(Record<GenericObject> record) {
+
+ log.info("properties {}", record.getProperties());
log.info("received record {} {}", record, record.getClass());
log.info("schema {}", record.getSchema());
log.info("native schema {}", record.getSchema().getNativeSchema().orElse(null));
+ String expectedRecordType = record.getProperties().getOrDefault("expectedType", "MISSING");
+ if (!expectedRecordType.equals(record.getSchema().getSchemaInfo().getType().name())) {
+ throw new RuntimeException("Unexpected record type "+record.getSchema().getSchemaInfo().getType().name() +" is not "+expectedRecordType);
+ }
+
+ log.info("value {}", record.getValue());
+ log.info("value schema type {}", record.getValue().getSchemaType());
+ log.info("value native object {}", record.getValue().getNativeObject());
+
+ if (record.getSchema().getSchemaInfo().getType() == SchemaType.KEY_VALUE) {
+ // assert that we are able to access the schema (leads to ClassCastException if there is a problem)
+ KeyValueSchema kvSchema = (KeyValueSchema) record.getSchema();
+ log.info("key schema type {}", kvSchema.getKeySchema());
+ log.info("value schema type {}", kvSchema.getValueSchema());
+ log.info("key encoding {}", kvSchema.getKeyValueEncodingType());
+
+ KeyValue keyValue = (KeyValue) record.getValue().getNativeObject();
+ log.info("kvkey {}", keyValue.getKey());
+ log.info("kvvalue {}", keyValue.getValue());
+ }
log.info("value {}", record.getValue());
log.info("value schema type {}", record.getValue().getSchemaType());
log.info("value native object {}", record.getValue().getNativeObject());
diff --git a/tests/integration/src/test/java/org/apache/pulsar/tests/integration/io/PulsarGenericObjectSinkTest.java b/tests/integration/src/test/java/org/apache/pulsar/tests/integration/io/PulsarGenericObjectSinkTest.java
index 25f1776..1e5fb74 100644
--- a/tests/integration/src/test/java/org/apache/pulsar/tests/integration/io/PulsarGenericObjectSinkTest.java
+++ b/tests/integration/src/test/java/org/apache/pulsar/tests/integration/io/PulsarGenericObjectSinkTest.java
@@ -22,8 +22,10 @@ import lombok.Builder;
import lombok.Cleanup;
import lombok.Data;
import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.io.IOUtils;
import org.apache.commons.lang3.StringUtils;
import org.apache.pulsar.client.admin.PulsarAdmin;
+import org.apache.pulsar.client.api.MessageId;
import org.apache.pulsar.client.api.Producer;
import org.apache.pulsar.client.api.PulsarClient;
import org.apache.pulsar.client.api.Schema;
@@ -36,8 +38,11 @@ import org.apache.pulsar.tests.integration.topologies.PulsarCluster;
import org.awaitility.Awaitility;
import org.testng.annotations.Test;
+import java.io.ByteArrayOutputStream;
+import java.nio.charset.StandardCharsets;
import java.util.Arrays;
import java.util.List;
+import java.util.concurrent.TimeUnit;
import static org.apache.pulsar.tests.integration.functions.utils.CommandGenerator.JAVAJAR;
import static org.testng.Assert.assertEquals;
@@ -71,16 +76,29 @@ public class PulsarGenericObjectSinkTest extends PulsarStandaloneTestSuite {
private int field2;
}
+ @Data
+ @Builder
+ public static final class PojoKey {
+ private String field1;
+ }
+
@Test(groups = {"sink"})
public void testGenericObjectSink() throws Exception {
+
+ @Cleanup PulsarClient client = PulsarClient.builder()
+ .serviceUrl(container.getPlainTextServiceUrl())
+ .build();
+
// we are not using a parametrized test in order to save resources
// we create N sinks, send the records and verify each sink
// sinks execution happens in parallel
List<SinkSpec> specs = Arrays.asList(
new SinkSpec("test-kv-sink-input-string-" + randomName(8), "test-kv-sink-string-" + randomName(8), Schema.STRING, "foo"),
- new SinkSpec("test-kv-sink-input-int-" + randomName(8), "test-kv-sink-int-" + randomName(8), Schema.INT32, 123),
new SinkSpec("test-kv-sink-input-avro-" + randomName(8), "test-kv-sink-avro-" + randomName(8), Schema.AVRO(Pojo.class), Pojo.builder().field1("a").field2(2).build()),
- new SinkSpec("test-kv-sink-input-json-" + randomName(8), "test-kv-sink-json-" + randomName(8), Schema.JSON(Pojo.class), Pojo.builder().field1("a").field2(2).build())
+ new SinkSpec("test-kv-sink-input-kv-string-int-" + randomName(8), "test-kv-sink-input-kv-string-int-" + randomName(8),
+ Schema.KeyValue(Schema.STRING, Schema.INT32), new KeyValue<>("foo", 123)),
+ new SinkSpec("test-kv-sink-input-kv-avro-json-" + randomName(8), "test-kv-sink-input-kv-string-int-" + randomName(8),
+ Schema.KeyValue(Schema.AVRO(PojoKey.class), Schema.JSON(Pojo.class)), new KeyValue<>(PojoKey.builder().field1("a").build(), Pojo.builder().field1("a").field2(2).build()))
);
// submit all sinks
for (SinkSpec spec : specs) {
@@ -93,10 +111,6 @@ public class PulsarGenericObjectSinkTest extends PulsarStandaloneTestSuite {
getSinkStatus(spec.sinkName);
}
- @Cleanup PulsarClient client = PulsarClient.builder()
- .serviceUrl(container.getPlainTextServiceUrl())
- .build();
-
final int numRecords = 10;
for (SinkSpec spec : specs) {
@@ -104,21 +118,41 @@ public class PulsarGenericObjectSinkTest extends PulsarStandaloneTestSuite {
.topic(spec.outputTopicName)
.create();
for (int i = 0; i < numRecords; i++) {
- producer.send(spec.testValue);
+ MessageId messageId = producer.newMessage()
+ .value(spec.testValue)
+ .property("expectedType", spec.schema.getSchemaInfo().getType().toString())
+ .send();
+ log.info("sent message {} {} with ID {}", spec.testValue, spec.schema.getSchemaInfo().getType().toString(), messageId);
}
}
// wait that all sinks processed all records without errors
try (PulsarAdmin admin = PulsarAdmin.builder().serviceHttpUrl(container.getHttpServiceUrl()).build()) {
-
for (SinkSpec spec : specs) {
- Awaitility.await().ignoreExceptions().untilAsserted(() -> {
+ try {
+ log.info("waiting for sink {}", spec.sinkName);
+ for (int i = 0; i < 120; i++) {
+ SinkStatus status = admin.sinks().getSinkStatus("public", "default", spec.sinkName);
+ log.info("sink {} status {}", spec.sinkName, status);
+ assertEquals(status.getInstances().size(), 1);
+ SinkStatus.SinkInstanceStatus instance = status.getInstances().get(0);
+ if (instance.getStatus().numWrittenToSink >= numRecords) {
+ break;
+ }
+ assertTrue(instance.getStatus().numRestarts > 1, "Sink was restarted, probably an error occurred");
+ Thread.sleep(1000);
+ }
+
SinkStatus status = admin.sinks().getSinkStatus("public", "default", spec.sinkName);
+ log.info("sink {} status {}", spec.sinkName, status);
assertEquals(status.getInstances().size(), 1);
- assertTrue(status.getInstances().get(0).getStatus().numReadFromPulsar >= numRecords);
+ assertTrue(status.getInstances().get(0).getStatus().numWrittenToSink >= numRecords);
assertTrue(status.getInstances().get(0).getStatus().numSinkExceptions == 0);
assertTrue(status.getInstances().get(0).getStatus().numSystemExceptions == 0);
- });
+ log.info("sink {} is okay", spec.sinkName);
+ } finally {
+ dumpSinkLogs(spec);
+ }
}
}
@@ -129,6 +163,18 @@ public class PulsarGenericObjectSinkTest extends PulsarStandaloneTestSuite {
}
}
+ private void dumpSinkLogs(SinkSpec spec) {
+ try {
+ String logFile = "/pulsar/logs/functions/public/default/" + spec.sinkName + "/" + spec.sinkName + "-0.log";
+ String logs = container.<String>copyFileFromContainer(logFile, (inputStream) -> {
+ return IOUtils.toString(inputStream, "utf-8");
+ });
+ log.info("Sink {} logs {}", spec.sinkName, logs);
+ } catch (Throwable err) {
+ log.info("Cannot download sink {} logs", spec.sinkName, err);
+ }
+ }
+
private void submitSinkConnector(String sinkName,
String inputTopicName,
String className,
@@ -169,6 +215,8 @@ public class PulsarGenericObjectSinkTest extends PulsarStandaloneTestSuite {
"--namespace", "default",
"--name", sinkName
);
+ log.info(result.getStdout());
+ log.info(result.getStderr());
assertTrue(result.getStdout().contains("\"running\" : true"));
}
[pulsar] 06/46: suppress nashorn warning in JDK11 runtime
Posted by eo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 65f744f5ba3fe25c97c1d327394fddf70e8eb015
Author: ming luo <it...@gmail.com>
AuthorDate: Tue Jan 12 23:10:56 2021 +0000
suppress nashorn warning in JDK11 runtime
(cherry picked from commit cebcc97f33a6c9b2c76b5708d400229ca08e941d)
---
bin/pulsar-admin-common.sh | 1 +
1 file changed, 1 insertion(+)
diff --git a/bin/pulsar-admin-common.sh b/bin/pulsar-admin-common.sh
index 0ccfc70..8f7256f 100755
--- a/bin/pulsar-admin-common.sh
+++ b/bin/pulsar-admin-common.sh
@@ -87,6 +87,7 @@ PULSAR_CLASSPATH="$PULSAR_JAR:$PULSAR_CLASSPATH:$PULSAR_EXTRA_CLASSPATH"
PULSAR_CLASSPATH="`dirname $PULSAR_LOG_CONF`:$PULSAR_CLASSPATH"
OPTS="$OPTS -Dlog4j.configurationFile=`basename $PULSAR_LOG_CONF`"
OPTS="$OPTS -Djava.net.preferIPv4Stack=true"
+OPTS="$OPTS -Dnashorn.args=--no-deprecation-warning"
OPTS="-cp $PULSAR_CLASSPATH $OPTS"
[pulsar] 23/46: Add presto password authenticators plugin to enable
password auth in Pulsar SQL
Posted by eo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 6fe93f50a8c1b6d63078e2a774d6ece77aec013f
Author: Chris Bartholomew <c_...@yahoo.com>
AuthorDate: Mon Feb 1 16:58:43 2021 -0500
Add presto password authenticators plugin to enable password auth in Pulsar SQL
---
pulsar-sql/pom.xml | 1 +
pulsar-sql/presto-distribution/pom.xml | 8 +++
.../presto-distribution/src/assembly/assembly.xml | 11 ++++
.../presto-password-authenticators-plugin/pom.xml | 75 ++++++++++++++++++++++
.../src/assembly/assembly.xml | 30 +--------
.../META-INF/services/io.prestosql.spi.Plugin | 1 +
6 files changed, 99 insertions(+), 27 deletions(-)
diff --git a/pulsar-sql/pom.xml b/pulsar-sql/pom.xml
index 22276b7..4ad5cf3 100644
--- a/pulsar-sql/pom.xml
+++ b/pulsar-sql/pom.xml
@@ -34,6 +34,7 @@
<modules>
<module>presto-pulsar</module>
<module>presto-pulsar-plugin</module>
+ <module>presto-password-authenticators-plugin</module>
<module>presto-distribution</module>
</modules>
diff --git a/pulsar-sql/presto-distribution/pom.xml b/pulsar-sql/presto-distribution/pom.xml
index 710eda8..9e2a86e 100644
--- a/pulsar-sql/presto-distribution/pom.xml
+++ b/pulsar-sql/presto-distribution/pom.xml
@@ -166,6 +166,14 @@
</dependency>
<dependency>
+ <groupId>org.apache.pulsar</groupId>
+ <artifactId>presto-password-authenticators-plugin</artifactId>
+ <version>${project.version}</version>
+ <type>tar.gz</type>
+ <scope>provided</scope>
+ </dependency>
+
+ <dependency>
<groupId>org.objenesis</groupId>
<artifactId>objenesis</artifactId>
<version>${objenesis.version}</version>
diff --git a/pulsar-sql/presto-distribution/src/assembly/assembly.xml b/pulsar-sql/presto-distribution/src/assembly/assembly.xml
index 54961af..58cbb95 100644
--- a/pulsar-sql/presto-distribution/src/assembly/assembly.xml
+++ b/pulsar-sql/presto-distribution/src/assembly/assembly.xml
@@ -40,6 +40,17 @@
<directory>${basedir}/../presto-pulsar-plugin/target/pulsar-presto-connector/</directory>
<outputDirectory>plugin/</outputDirectory>
</fileSet>
+ <fileSet>
+ <directory>${basedir}/../presto-password-authenticators-plugin/target/presto-password-authenticators-plugin</directory>
+ <outputDirectory>plugin/</outputDirectory>
+ </fileSet>
+ <fileSet>
+ <directory>${basedir}/../presto-password-authenticators-plugin/target/</directory>
+ <outputDirectory>plugin/presto-password-authenticators-plugin</outputDirectory>
+ <includes>
+ <include>*.jar</include>
+ </includes>
+ </fileSet>
</fileSets>
<dependencySets>
<dependencySet>
diff --git a/pulsar-sql/presto-password-authenticators-plugin/pom.xml b/pulsar-sql/presto-password-authenticators-plugin/pom.xml
new file mode 100644
index 0000000..26d96d5
--- /dev/null
+++ b/pulsar-sql/presto-password-authenticators-plugin/pom.xml
@@ -0,0 +1,75 @@
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.pulsar</groupId>
+ <artifactId>pulsar-sql</artifactId>
+ <version>2.7.2.1.0.0</version>
+ </parent>
+
+ <artifactId>presto-password-authenticators-plugin</artifactId>
+ <name>Pulsar SQL :: Presto Password Authenticators</name>
+
+ <dependencies>
+
+ <dependency>
+ <groupId>io.prestosql</groupId>
+ <artifactId>presto-password-authenticators</artifactId>
+ <version>${presto.version}</version>
+ </dependency>
+
+
+ </dependencies>
+
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-install-plugin</artifactId>
+ </plugin>
+ <plugin>
+ <artifactId>maven-assembly-plugin</artifactId>
+ <version>3.3.0</version>
+ <configuration>
+ <appendAssemblyId>false</appendAssemblyId>
+ <attach>true</attach>
+ <tarLongFileMode>posix</tarLongFileMode>
+ <descriptors>
+ <descriptor>src/assembly/assembly.xml</descriptor>
+ </descriptors>
+ </configuration>
+ <executions>
+ <execution>
+ <id>package</id>
+ <phase>package</phase>
+ <goals>
+ <goal>single</goal>
+ </goals>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+
+</project>
diff --git a/pulsar-sql/presto-distribution/src/assembly/assembly.xml b/pulsar-sql/presto-password-authenticators-plugin/src/assembly/assembly.xml
similarity index 54%
copy from pulsar-sql/presto-distribution/src/assembly/assembly.xml
copy to pulsar-sql/presto-password-authenticators-plugin/src/assembly/assembly.xml
index 54961af..42200aa 100644
--- a/pulsar-sql/presto-distribution/src/assembly/assembly.xml
+++ b/pulsar-sql/presto-password-authenticators-plugin/src/assembly/assembly.xml
@@ -26,38 +26,14 @@
<format>tar.gz</format>
<format>dir</format>
</formats>
- <includeBaseDirectory>false</includeBaseDirectory>
- <files>
- <file>
- <source>${basedir}/LICENSE</source>
- <destName>LICENSE</destName>
- <outputDirectory>.</outputDirectory>
- <fileMode>644</fileMode>
- </file>
- </files>
- <fileSets>
- <fileSet>
- <directory>${basedir}/../presto-pulsar-plugin/target/pulsar-presto-connector/</directory>
- <outputDirectory>plugin/</outputDirectory>
- </fileSet>
- </fileSets>
<dependencySets>
<dependencySet>
- <outputDirectory>lib/</outputDirectory>
- <useProjectArtifact>true</useProjectArtifact>
+ <outputDirectory>/</outputDirectory>
+ <useProjectArtifact>false</useProjectArtifact>
<scope>runtime</scope>
<excludes>
- <exclude>io.airlift:launcher:tar.gz:bin:${airlift.version}</exclude>
- <exclude>io.airlift:launcher:tar.gz:properties:${airlift.version}</exclude>
+ <exclude>jakarta.ws.rs:jakarta.ws.rs-api</exclude>
</excludes>
</dependencySet>
- <dependencySet>
- <outputDirectory></outputDirectory>
- <includes>
- <include>io.airlift:launcher:tar.gz:bin:${airlift.version}</include>
- <include>io.airlift:launcher:tar.gz:properties:${airlift.version}</include>
- </includes>
- <unpack>true</unpack>
- </dependencySet>
</dependencySets>
</assembly>
\ No newline at end of file
diff --git a/pulsar-sql/presto-password-authenticators-plugin/src/main/resources/META-INF/services/io.prestosql.spi.Plugin b/pulsar-sql/presto-password-authenticators-plugin/src/main/resources/META-INF/services/io.prestosql.spi.Plugin
new file mode 100644
index 0000000..f6dfd25
--- /dev/null
+++ b/pulsar-sql/presto-password-authenticators-plugin/src/main/resources/META-INF/services/io.prestosql.spi.Plugin
@@ -0,0 +1 @@
+io.prestosql.plugin.password.PasswordAuthenticatorPlugin
[pulsar] 14/46: [pulsar-functions] enhance kubernetes manifest
customizer with default options (#9445)
Posted by eo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 421fd828a02b04a0a3d21f8288c421b285154a0a
Author: Rui Fu <fr...@users.noreply.github.com>
AuthorDate: Fri Feb 5 18:56:53 2021 +0800
[pulsar-functions] enhance kubernetes manifest customizer with default options (#9445)
The KubernetesManifestCustomizer was introduced by customizing the stateful set of running Pulsar Functions. but no default value was loaded from `functions_worker.yaml`.
Add load default runtime options in `BasicKubernetesManifestCustomizer`
Add unit tests
- [ ] Make sure that the change passes the CI checks.
---
conf/functions_worker.yml | 14 +++
pom.xml | 4 -
.../BasicKubernetesManifestCustomizer.java | 117 +++++++++++++++++--
.../BasicKubernetesManifestCustomizerTest.java | 96 ++++++++++++++++
.../kubernetes/KubernetesRuntimeFactoryTest.java | 3 +-
.../runtime/kubernetes/KubernetesRuntimeTest.java | 126 +++++++++++++++------
6 files changed, 309 insertions(+), 51 deletions(-)
diff --git a/conf/functions_worker.yml b/conf/functions_worker.yml
index 4dfcf58..70bcec1 100644
--- a/conf/functions_worker.yml
+++ b/conf/functions_worker.yml
@@ -193,6 +193,20 @@ functionRuntimeFactoryConfigs:
## This class receives the customRuntimeOptions string and can customize details of how the runtime operates.
#runtimeCustomizerClassName: "org.apache.pulsar.functions.runtime.kubernetes.KubernetesManifestCustomizer"
+## This config will pass to RuntimeCustomizer's initialize function to do initializing.
+#runtimeCustomizerConfig:
+# extractLabels:
+# extraLabel: value
+# extraAnnotations:
+# extraAnnotation: value
+# nodeSelectorLabels:
+# customLabel: value
+# jobNamespace: namespace
+# tolerations:
+# - key: custom-key
+# value: value
+# effect: NoSchedule
+
## Config admin CLI
#configAdminCLI:
diff --git a/pom.xml b/pom.xml
index d122c8a..c99802d 100644
--- a/pom.xml
+++ b/pom.xml
@@ -483,11 +483,7 @@
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
-<<<<<<< HEAD
<version>${commons-lang3.version}</version>
-=======
- <version>3.9</version>
->>>>>>> da22685e074... upgrade presto version to 334 and JDK11
</dependency>
<dependency>
diff --git a/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/kubernetes/BasicKubernetesManifestCustomizer.java b/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/kubernetes/BasicKubernetesManifestCustomizer.java
index a9a6929..c69d315 100644
--- a/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/kubernetes/BasicKubernetesManifestCustomizer.java
+++ b/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/kubernetes/BasicKubernetesManifestCustomizer.java
@@ -21,12 +21,18 @@ package org.apache.pulsar.functions.runtime.kubernetes;
import com.google.gson.Gson;
import io.kubernetes.client.custom.Quantity;
import io.kubernetes.client.openapi.models.*;
+import lombok.AllArgsConstructor;
+import lombok.Builder;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.Setter;
+import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
+import org.apache.pulsar.common.util.ObjectMapperFactory;
import org.apache.pulsar.functions.proto.Function;
+import java.util.ArrayList;
+import java.util.HashMap;
import java.util.List;
import java.util.Map;
@@ -39,6 +45,7 @@ import java.util.Map;
* modify (for example, a service account must have permissions in the specified jobNamespace)
*
*/
+@Slf4j
public class BasicKubernetesManifestCustomizer implements KubernetesManifestCustomizer {
private static final String RESOURCE_CPU = "cpu";
@@ -48,7 +55,9 @@ public class BasicKubernetesManifestCustomizer implements KubernetesManifestCust
@Getter
@Setter
@NoArgsConstructor
- static private class RuntimeOpts {
+ @AllArgsConstructor
+ @Builder(toBuilder = true)
+ static public class RuntimeOpts {
private String jobNamespace;
private String jobName;
private Map<String, String> extraLabels;
@@ -58,13 +67,25 @@ public class BasicKubernetesManifestCustomizer implements KubernetesManifestCust
private List<V1Toleration> tolerations;
}
+ @Getter
+ private RuntimeOpts runtimeOpts = new RuntimeOpts();
+
@Override
public void initialize(Map<String, Object> config) {
+ if (config != null) {
+ RuntimeOpts opts = ObjectMapperFactory.getThreadLocal().convertValue(config, RuntimeOpts.class);
+ if (opts != null) {
+ runtimeOpts = opts.toBuilder().build();
+ }
+ } else {
+ log.warn("initialize with null config");
+ }
}
@Override
public String customizeNamespace(Function.FunctionDetails funcDetails, String currentNamespace) {
RuntimeOpts opts = getOptsFromDetails(funcDetails);
+ opts = mergeRuntimeOpts(runtimeOpts, opts);
if (!StringUtils.isEmpty(opts.getJobNamespace())) {
return opts.getJobNamespace();
} else {
@@ -75,6 +96,7 @@ public class BasicKubernetesManifestCustomizer implements KubernetesManifestCust
@Override
public String customizeName(Function.FunctionDetails funcDetails, String currentName) {
RuntimeOpts opts = getOptsFromDetails(funcDetails);
+ opts = mergeRuntimeOpts(runtimeOpts, opts);
if (!StringUtils.isEmpty(opts.getJobName())) {
return opts.getJobName();
} else {
@@ -85,24 +107,27 @@ public class BasicKubernetesManifestCustomizer implements KubernetesManifestCust
@Override
public V1Service customizeService(Function.FunctionDetails funcDetails, V1Service service) {
RuntimeOpts opts = getOptsFromDetails(funcDetails);
+ opts = mergeRuntimeOpts(runtimeOpts, opts);
service.setMetadata(updateMeta(opts, service.getMetadata()));
return service;
}
@Override
public V1StatefulSet customizeStatefulSet(Function.FunctionDetails funcDetails, V1StatefulSet statefulSet) {
- RuntimeOpts opts = getOptsFromDetails(funcDetails);
+ RuntimeOpts opts = mergeRuntimeOpts(runtimeOpts, getOptsFromDetails(funcDetails));
statefulSet.setMetadata(updateMeta(opts, statefulSet.getMetadata()));
V1PodTemplateSpec pt = statefulSet.getSpec().getTemplate();
pt.setMetadata(updateMeta(opts, pt.getMetadata()));
V1PodSpec ps = pt.getSpec();
- if (opts.getNodeSelectorLabels() != null && opts.getNodeSelectorLabels().size() > 0) {
- opts.getNodeSelectorLabels().forEach(ps::putNodeSelectorItem);
- }
- if (opts.getTolerations() != null && opts.getTolerations().size() > 0) {
- opts.getTolerations().forEach(ps::addTolerationsItem);
+ if (ps != null) {
+ if (opts.getNodeSelectorLabels() != null && opts.getNodeSelectorLabels().size() > 0) {
+ opts.getNodeSelectorLabels().forEach(ps::putNodeSelectorItem);
+ }
+ if (opts.getTolerations() != null && opts.getTolerations().size() > 0) {
+ opts.getTolerations().forEach(ps::addTolerationsItem);
+ }
+ ps.getContainers().forEach(container -> updateContainerResources(container, opts));
}
- ps.getContainers().forEach(container -> updateContainerResources(container, opts));
return statefulSet;
}
@@ -113,10 +138,10 @@ public class BasicKubernetesManifestCustomizer implements KubernetesManifestCust
Map<String, Quantity> limits = resourceRequirements.getLimits();
Map<String, Quantity> requests = resourceRequirements.getRequests();
for (String resource : RESOURCES) {
- if (limits.containsKey(resource)) {
+ if (limits != null && limits.containsKey(resource)) {
containerResources.putLimitsItem(resource, limits.get(resource));
}
- if (requests.containsKey(resource)) {
+ if (requests != null && requests.containsKey(resource)) {
containerResources.putRequestsItem(resource, requests.get(resource));
}
}
@@ -143,4 +168,76 @@ public class BasicKubernetesManifestCustomizer implements KubernetesManifestCust
return meta;
}
+ public static RuntimeOpts mergeRuntimeOpts(RuntimeOpts oriOpts, RuntimeOpts newOpts) {
+ RuntimeOpts mergedOpts = oriOpts.toBuilder().build();
+ if (mergedOpts.getExtraLabels() == null) {
+ mergedOpts.setExtraLabels(new HashMap<>());
+ }
+ if (mergedOpts.getExtraAnnotations() == null) {
+ mergedOpts.setExtraAnnotations(new HashMap<>());
+ }
+ if (mergedOpts.getNodeSelectorLabels() == null) {
+ mergedOpts.setNodeSelectorLabels(new HashMap<>());
+ }
+ if (mergedOpts.getTolerations() == null) {
+ mergedOpts.setTolerations(new ArrayList<>());
+ }
+ if (mergedOpts.getResourceRequirements() == null) {
+ mergedOpts.setResourceRequirements(new V1ResourceRequirements());
+ }
+
+ if (!StringUtils.isEmpty(newOpts.getJobName())) {
+ mergedOpts.setJobName(newOpts.getJobName());
+ }
+ if (!StringUtils.isEmpty(newOpts.getJobNamespace())) {
+ mergedOpts.setJobNamespace(newOpts.getJobNamespace());
+ }
+ if (newOpts.getExtraLabels() != null && !newOpts.getExtraLabels().isEmpty()) {
+ newOpts.getExtraLabels().forEach((key, labelsItem) -> {
+ if (!mergedOpts.getExtraLabels().containsKey(key)) {
+ log.debug("extra label {} has been changed to {}", key, labelsItem);
+ }
+ mergedOpts.getExtraLabels().put(key, labelsItem);
+ });
+ }
+ if (newOpts.getExtraAnnotations() != null && !newOpts.getExtraAnnotations().isEmpty()) {
+ newOpts.getExtraAnnotations().forEach((key, annotationsItem) -> {
+ if (!mergedOpts.getExtraAnnotations().containsKey(key)) {
+ log.debug("extra annotation {} has been changed to {}", key, annotationsItem);
+ }
+ mergedOpts.getExtraAnnotations().put(key, annotationsItem);
+ });
+ }
+ if (newOpts.getNodeSelectorLabels() != null && !newOpts.getNodeSelectorLabels().isEmpty()) {
+ newOpts.getNodeSelectorLabels().forEach((key, nodeSelectorItem) -> {
+ if (!mergedOpts.getNodeSelectorLabels().containsKey(key)) {
+ log.debug("node selector label {} has been changed to {}", key, nodeSelectorItem);
+ }
+ mergedOpts.getNodeSelectorLabels().put(key, nodeSelectorItem);
+ });
+ }
+
+ if (newOpts.getResourceRequirements() != null) {
+ V1ResourceRequirements mergedResourcesRequirements = mergedOpts.getResourceRequirements();
+ V1ResourceRequirements newResourcesRequirements = newOpts.getResourceRequirements();
+
+ Map<String, Quantity> limits = newResourcesRequirements.getLimits();
+ Map<String, Quantity> requests = newResourcesRequirements.getRequests();
+ for (String resource : RESOURCES) {
+ if (limits != null && limits.containsKey(resource)) {
+ mergedResourcesRequirements.putLimitsItem(resource, limits.get(resource));
+ }
+ if (requests != null && requests.containsKey(resource)) {
+ mergedResourcesRequirements.putRequestsItem(resource, requests.get(resource));
+ }
+ }
+ mergedOpts.setResourceRequirements(mergedResourcesRequirements);
+ }
+
+ if (newOpts.getTolerations() != null && !newOpts.getTolerations().isEmpty()) {
+ mergedOpts.getTolerations().addAll(newOpts.getTolerations());
+ }
+ return mergedOpts;
+ }
+
}
diff --git a/pulsar-functions/runtime/src/test/java/org/apache/pulsar/functions/runtime/kubernetes/BasicKubernetesManifestCustomizerTest.java b/pulsar-functions/runtime/src/test/java/org/apache/pulsar/functions/runtime/kubernetes/BasicKubernetesManifestCustomizerTest.java
new file mode 100644
index 0000000..1b38306
--- /dev/null
+++ b/pulsar-functions/runtime/src/test/java/org/apache/pulsar/functions/runtime/kubernetes/BasicKubernetesManifestCustomizerTest.java
@@ -0,0 +1,96 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.functions.runtime.kubernetes;
+
+import com.google.gson.Gson;
+import io.kubernetes.client.custom.Quantity;
+import io.kubernetes.client.openapi.models.V1ResourceRequirements;
+import io.kubernetes.client.openapi.models.V1Toleration;
+import org.testng.annotations.Test;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertNotEquals;
+import static org.testng.Assert.assertNull;
+
+/**
+ * Unit test of {@link BasicKubernetesManifestCustomizerTest}.
+ */
+public class BasicKubernetesManifestCustomizerTest {
+
+ @Test
+ public void TestInitializeWithNullData() {
+ BasicKubernetesManifestCustomizer customizer = new BasicKubernetesManifestCustomizer();
+ customizer.initialize(null);
+ assertNotEquals(customizer.getRuntimeOpts(), null);
+ assertNull(customizer.getRuntimeOpts().getExtraLabels());
+ assertNull(customizer.getRuntimeOpts().getExtraAnnotations());
+ assertNull(customizer.getRuntimeOpts().getNodeSelectorLabels());
+ assertNull(customizer.getRuntimeOpts().getTolerations());
+ assertNull(customizer.getRuntimeOpts().getResourceRequirements());
+ }
+
+ @Test
+ public void TestInitializeWithData() {
+ BasicKubernetesManifestCustomizer customizer = new BasicKubernetesManifestCustomizer();
+ Map<String, Object> confs = new HashMap<>();
+ confs.put("jobNamespace", "custom-ns");
+ confs.put("jobName", "custom-name");
+ customizer.initialize(confs);
+ assertNotEquals(customizer.getRuntimeOpts(), null);
+ assertEquals(customizer.getRuntimeOpts().getJobName(), "custom-name");
+ assertEquals(customizer.getRuntimeOpts().getJobNamespace(), "custom-ns");
+ }
+
+ @Test
+ public void TestMergeRuntimeOpts() {
+ Map<String, Object> configs = new Gson().fromJson(KubernetesRuntimeTest.createRuntimeCustomizerConfig(), HashMap.class);
+ BasicKubernetesManifestCustomizer customizer = new BasicKubernetesManifestCustomizer();
+ customizer.initialize(configs);
+ BasicKubernetesManifestCustomizer.RuntimeOpts newOpts = new BasicKubernetesManifestCustomizer.RuntimeOpts();
+ newOpts.setJobName("merged-name");
+ newOpts.setTolerations(Collections.emptyList());
+ V1Toleration toleration = new V1Toleration();
+ toleration.setKey("merge-key");
+ toleration.setEffect("NoSchedule");
+ toleration.setOperator("Equal");
+ toleration.setTolerationSeconds(6000L);
+ newOpts.setTolerations(Collections.singletonList(toleration));
+ V1ResourceRequirements resourceRequirements = new V1ResourceRequirements();
+ resourceRequirements.putLimitsItem("cpu", new Quantity("20"));
+ resourceRequirements.putLimitsItem("memory", new Quantity("10240"));
+ newOpts.setResourceRequirements(resourceRequirements);
+ newOpts.setNodeSelectorLabels(Collections.singletonMap("disktype", "ssd"));
+ newOpts.setExtraAnnotations(Collections.singletonMap("functiontype", "sink"));
+ newOpts.setExtraLabels(Collections.singletonMap("functiontype", "sink"));
+ BasicKubernetesManifestCustomizer.RuntimeOpts mergedOpts = BasicKubernetesManifestCustomizer.mergeRuntimeOpts(
+ customizer.getRuntimeOpts(), newOpts);
+
+ assertEquals(mergedOpts.getJobName(), "merged-name");
+ assertEquals(mergedOpts.getTolerations().size(), 2);
+ assertEquals(mergedOpts.getExtraAnnotations().size(), 2);
+ assertEquals(mergedOpts.getExtraLabels().size(), 2);
+ assertEquals(mergedOpts.getNodeSelectorLabels().size(), 2);
+ assertEquals(mergedOpts.getResourceRequirements().getLimits().get("cpu").getNumber().intValue(), 20);
+ assertEquals(mergedOpts.getResourceRequirements().getLimits().get("memory").getNumber().intValue(), 10240);
+ }
+}
diff --git a/pulsar-functions/runtime/src/test/java/org/apache/pulsar/functions/runtime/kubernetes/KubernetesRuntimeFactoryTest.java b/pulsar-functions/runtime/src/test/java/org/apache/pulsar/functions/runtime/kubernetes/KubernetesRuntimeFactoryTest.java
index 62d71c0..56a568a 100644
--- a/pulsar-functions/runtime/src/test/java/org/apache/pulsar/functions/runtime/kubernetes/KubernetesRuntimeFactoryTest.java
+++ b/pulsar-functions/runtime/src/test/java/org/apache/pulsar/functions/runtime/kubernetes/KubernetesRuntimeFactoryTest.java
@@ -36,7 +36,6 @@ import org.apache.pulsar.common.functions.AuthenticationConfig;
import org.apache.pulsar.functions.proto.Function;
import org.apache.pulsar.functions.proto.Function.FunctionDetails;
import org.apache.pulsar.functions.runtime.RuntimeCustomizer;
-import org.apache.pulsar.functions.runtime.thread.ThreadRuntime;
import org.apache.pulsar.functions.secretsprovider.ClearTextSecretsProvider;
import org.apache.pulsar.functions.secretsproviderconfigurator.DefaultSecretsProviderConfigurator;
import org.apache.pulsar.functions.secretsproviderconfigurator.SecretsProviderConfigurator;
@@ -58,7 +57,7 @@ import static org.testng.Assert.assertEquals;
import static org.testng.Assert.fail;
/**
- * Unit test of {@link ThreadRuntime}.
+ * Unit test of {@link KubernetesRuntimeFactoryTest}.
*/
public class KubernetesRuntimeFactoryTest {
diff --git a/pulsar-functions/runtime/src/test/java/org/apache/pulsar/functions/runtime/kubernetes/KubernetesRuntimeTest.java b/pulsar-functions/runtime/src/test/java/org/apache/pulsar/functions/runtime/kubernetes/KubernetesRuntimeTest.java
index 71e4fda..4443eb6 100644
--- a/pulsar-functions/runtime/src/test/java/org/apache/pulsar/functions/runtime/kubernetes/KubernetesRuntimeTest.java
+++ b/pulsar-functions/runtime/src/test/java/org/apache/pulsar/functions/runtime/kubernetes/KubernetesRuntimeTest.java
@@ -219,6 +219,8 @@ public class KubernetesRuntimeTest {
workerConfig.setAuthenticationEnabled(false);
workerConfig.setDownloadDirectory(downloadDirectory);
+ manifestCustomizer.ifPresent(runtimeCustomizer -> runtimeCustomizer.initialize(Optional.ofNullable(workerConfig.getRuntimeCustomizerConfig()).orElse(Collections.emptyMap())));
+
factory.initialize(workerConfig, null, new TestSecretProviderConfigurator(), Optional.empty(), manifestCustomizer);
doNothing().when(factory).setupClient();
@@ -778,41 +780,7 @@ public class KubernetesRuntimeTest {
public void testBasicKubernetesManifestCustomizer() throws Exception {
InstanceConfig config = createJavaInstanceConfig(FunctionDetails.Runtime.JAVA, false);
config.setFunctionDetails(createFunctionDetails(FunctionDetails.Runtime.JAVA, false, (fb) -> {
- JsonObject configObj = new JsonObject();
- configObj.addProperty("jobNamespace", "custom-ns");
- configObj.addProperty("jobName", "custom-name");
-
- JsonObject extraAnn = new JsonObject();
- extraAnn.addProperty("annotation", "test");
- configObj.add("extraAnnotations", extraAnn);
-
- JsonObject extraLabel = new JsonObject();
- extraLabel.addProperty("label", "test");
- configObj.add("extraLabels", extraLabel);
-
- JsonObject nodeLabels = new JsonObject();
- nodeLabels.addProperty("selector", "test");
- configObj.add("nodeSelectorLabels", nodeLabels);
-
- JsonArray tolerations = new JsonArray();
- JsonObject toleration = new JsonObject();
- toleration.addProperty("key", "test");
- toleration.addProperty("value", "test");
- toleration.addProperty("effect", "test");
- tolerations.add(toleration);
- configObj.add("tolerations", tolerations);
-
- JsonObject resourceRequirements = new JsonObject();
- JsonObject requests = new JsonObject();
- JsonObject limits = new JsonObject();
- requests.addProperty("cpu", 1);
- requests.addProperty("memory", "4G");
- limits.addProperty("cpu", 2);
- limits.addProperty("memory", "8G");
- resourceRequirements.add("requests", requests);
- resourceRequirements.add("limits", limits);
- configObj.add("resourceRequirements", resourceRequirements);
-
+ JsonObject configObj = createRuntimeCustomizerConfig();
return fb.setCustomRuntimeOptions(configObj.toString());
}));
@@ -1066,4 +1034,92 @@ public class KubernetesRuntimeTest {
verifyJavaInstance(config, pulsarRootDir + "/instances/deps", false, factory.getDownloadDirectory());
}
+ @Test
+ public void testBasicKubernetesManifestCustomizerWithRuntimeCustomizerConfig() throws Exception {
+ InstanceConfig config = createJavaInstanceConfig(FunctionDetails.Runtime.JAVA, false);
+
+ Map<String, Object> configs = new Gson().fromJson(createRuntimeCustomizerConfig(), HashMap.class);
+
+ factory = createKubernetesRuntimeFactory(null, 10, 1.0, 1.0,
+ "org.apache.pulsar.functions.runtime.kubernetes.BasicKubernetesManifestCustomizer", configs);
+
+ verifyJavaInstance(config, pulsarRootDir + "/instances/deps", false);
+ KubernetesRuntime container = factory.createContainer(config, userJarFile, userJarFile, 30l);
+ V1StatefulSet spec = container.createStatefulSet();
+ assertEquals(spec.getMetadata().getAnnotations().get("annotation"), "test");
+ assertEquals(spec.getMetadata().getLabels().get("label"), "test");
+ assertEquals(spec.getSpec().getTemplate().getSpec().getNodeSelector().get("selector"), "test");
+ List<V1Toleration> tols = spec.getSpec().getTemplate().getSpec().getTolerations();
+ // we add three by default, plus our custom
+ assertEquals(tols.size(), 4);
+ assertEquals(tols.get(3).getKey(), "test");
+ assertEquals(tols.get(3).getValue(), "test");
+ assertEquals(tols.get(3).getEffect(), "test");
+
+ V1Service serviceSpec = container.createService();
+ assertEquals(serviceSpec.getMetadata().getNamespace(), "custom-ns");
+ assertEquals(serviceSpec.getMetadata().getName(), "custom-name-2deb2c2b");
+ assertEquals(serviceSpec.getMetadata().getAnnotations().get("annotation"), "test");
+ assertEquals(serviceSpec.getMetadata().getLabels().get("label"), "test");
+
+ List<V1Container> containers = spec.getSpec().getTemplate().getSpec().getContainers();
+ containers.forEach(c -> {
+ V1ResourceRequirements resources = c.getResources();
+ Map<String, Quantity> limits = resources.getLimits();
+ Map<String, Quantity> requests = resources.getRequests();
+ assertEquals(requests.get("cpu").getNumber(), new BigDecimal(1) );
+ assertEquals(limits.get("cpu").getNumber(), new BigDecimal(2) );
+ assertEquals(requests.get("memory").getNumber(), new BigDecimal(4000000000L) );
+ assertEquals(limits.get("memory").getNumber(), new BigDecimal(8000000000L) );
+ });
+
+ }
+
+
+ @Test
+ public void testBasicKubernetesManifestCustomizerWithRuntimeCustomizerConfigOverwrite() throws Exception {
+ InstanceConfig config = createJavaInstanceConfig(FunctionDetails.Runtime.JAVA, false);
+ config.setFunctionDetails(createFunctionDetails(FunctionDetails.Runtime.JAVA, false, (fb) -> {
+ JsonObject configObj = new JsonObject();
+ configObj.addProperty("jobNamespace", "custom-ns-overwrite");
+ configObj.addProperty("jobName", "custom-name-overwrite");
+ return fb.setCustomRuntimeOptions(configObj.toString());
+ }));
+
+ Map<String, Object> configs = new Gson().fromJson(createRuntimeCustomizerConfig(), HashMap.class);
+
+ factory = createKubernetesRuntimeFactory(null, 10, 1.0, 1.0,
+ "org.apache.pulsar.functions.runtime.kubernetes.BasicKubernetesManifestCustomizer", configs);
+
+ verifyJavaInstance(config, pulsarRootDir + "/instances/deps", false);
+ KubernetesRuntime container = factory.createContainer(config, userJarFile, userJarFile, 30l);
+ V1StatefulSet spec = container.createStatefulSet();
+ assertEquals(spec.getMetadata().getAnnotations().get("annotation"), "test");
+ assertEquals(spec.getMetadata().getLabels().get("label"), "test");
+ assertEquals(spec.getSpec().getTemplate().getSpec().getNodeSelector().get("selector"), "test");
+ List<V1Toleration> tols = spec.getSpec().getTemplate().getSpec().getTolerations();
+ // we add three by default, plus our custom
+ assertEquals(tols.size(), 4);
+ assertEquals(tols.get(3).getKey(), "test");
+ assertEquals(tols.get(3).getValue(), "test");
+ assertEquals(tols.get(3).getEffect(), "test");
+
+ V1Service serviceSpec = container.createService();
+ assertEquals(serviceSpec.getMetadata().getNamespace(), "custom-ns-overwrite");
+ assertEquals(serviceSpec.getMetadata().getName(), "custom-name-overwrite-7757f1ff");
+ assertEquals(serviceSpec.getMetadata().getAnnotations().get("annotation"), "test");
+ assertEquals(serviceSpec.getMetadata().getLabels().get("label"), "test");
+
+ List<V1Container> containers = spec.getSpec().getTemplate().getSpec().getContainers();
+ containers.forEach(c -> {
+ V1ResourceRequirements resources = c.getResources();
+ Map<String, Quantity> limits = resources.getLimits();
+ Map<String, Quantity> requests = resources.getRequests();
+ assertEquals(requests.get("cpu").getNumber(), new BigDecimal(1) );
+ assertEquals(limits.get("cpu").getNumber(), new BigDecimal(2) );
+ assertEquals(requests.get("memory").getNumber(), new BigDecimal(4000000000L) );
+ assertEquals(limits.get("memory").getNumber(), new BigDecimal(8000000000L) );
+ });
+
+ }
}
[pulsar] 04/46: update LICENSE for presto 334 upversion
Posted by eo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 6761fcb8da4b5603913c06cc3a53e86b476c6d2b
Author: ming luo <it...@gmail.com>
AuthorDate: Wed Jan 13 20:37:52 2021 +0000
update LICENSE for presto 334 upversion
(cherry picked from commit cc5b83663d27cb00d54539d5df4f09b8801819e2)
---
distribution/server/src/assemble/LICENSE.bin.txt | 8 ++--
pulsar-sql/presto-distribution/LICENSE | 52 ++++++++++++++----------
2 files changed, 35 insertions(+), 25 deletions(-)
diff --git a/distribution/server/src/assemble/LICENSE.bin.txt b/distribution/server/src/assemble/LICENSE.bin.txt
index 105f3c5..56be02c 100644
--- a/distribution/server/src/assemble/LICENSE.bin.txt
+++ b/distribution/server/src/assemble/LICENSE.bin.txt
@@ -355,7 +355,7 @@ The Apache Software License, Version 2.0
- commons-logging-commons-logging-1.1.1.jar
- org.apache.commons-commons-collections4-4.1.jar
- org.apache.commons-commons-compress-1.19.jar
- - org.apache.commons-commons-lang3-3.6.jar
+ - org.apache.commons-commons-lang3-3.9.jar
* Netty
- io.netty-netty-buffer-4.1.60.Final.jar
- io.netty-netty-codec-4.1.60.Final.jar
@@ -569,8 +569,10 @@ CDDL-1.1 -- licenses/LICENSE-CDDL-1.1.txt
* Mimepull -- org.jvnet.mimepull-mimepull-1.9.13.jar
Eclipse Distribution License 1.0 -- licenses/LICENSE-EDL-1.0.txt
- * Jakarta Activation -- jakarta.activation-jakarta.activation-api-1.2.1.jar
- * Jakarta XML Binding -- jakarta.xml.bind-jakarta.xml.bind-api-2.3.2.jar
+ * Jakarta Activation -- com.sun.activation-jakarta.activation-1.2.2.jar
+ * Jakarta XML Binding
+ - jakarta.activation-jakarta.activation-api-1.2.2.jar
+ - jakarta.xml.bind-jakarta.xml.bind-api-2.3.3.jar
Eclipse Public License 1.0 -- licenses/LICENSE-AspectJ.txt
* AspectJ
diff --git a/pulsar-sql/presto-distribution/LICENSE b/pulsar-sql/presto-distribution/LICENSE
index d627cd9..6b230e2 100644
--- a/pulsar-sql/presto-distribution/LICENSE
+++ b/pulsar-sql/presto-distribution/LICENSE
@@ -230,7 +230,7 @@ The Apache Software License, Version 2.0
* Apache Commons
- commons-math3-3.6.1.jar
- commons-compress-1.19.jar
- - commons-lang3-3.6.jar
+ - commons-lang3-3.9.jar
* Netty
- netty-3.10.6.Final.jar
- netty-buffer-4.1.60.Final.jar
@@ -258,6 +258,7 @@ The Apache Software License, Version 2.0
- http2-http-client-transport-9.4.27.v20200227.jar
- http2-server-9.4.27.v20200227.jar
- jetty-alpn-client-9.4.27.v20200227.jar
+ - jetty-alpn-java-client-9.4.27.v20200227.jar
- jetty-client-9.4.27.v20200227.jar
- jetty-http-9.4.27.v20200227.jar
- jetty-io-9.4.27.v20200227.jar
@@ -282,11 +283,11 @@ The Apache Software License, Version 2.0
- configuration-0.195.jar
- discovery-0.195.jar
- discovery-server-1.29.jar
- - event-0.195.jar
- - http-client-0.195.jar
- - http-server-0.195.jar
- - jmx-0.195.jar
- - jmx-http-0.195.jar
+ - event-0.197.jar
+ - http-client-0.197.jar
+ - http-server-0.197.jar
+ - jmx-0.197.jar
+ - jmx-http-0.197.jar
- jmx-http-rpc-0.159.jar
- joni-2.1.5.3.jar
- json-0.199.jar
@@ -298,10 +299,10 @@ The Apache Software License, Version 2.0
- node-0.195.jar
- parameternames-1.4.jar
- resolver-1.5.jar
- - security-0.195.jar
+ - security-0.197.jar
- slice-0.38.jar
- - stats-0.195.jar
- - trace-token-0.195.jar
+ - stats-0.197.jar
+ - trace-token-0.197.jar
- units-1.6.jar
* Apache HTTP Client
- httpclient-4.5.5.jar
@@ -377,16 +378,17 @@ The Apache Software License, Version 2.0
* Okio
- okio-1.13.0.jar
* Presto
- - presto-array-332.jar
- - presto-cli-332.jar
- - presto-client-332.jar
- - presto-geospatial-toolkit-332.jar
- - presto-main-332.jar
- - presto-matching-332.jar
- - presto-memory-context-332.jar
- - presto-parser-332.jar
- - presto-plugin-toolkit-332.jar
- - presto-spi-332.jar
+ - presto-array-334.jar
+ - presto-cli-334.jar
+ - presto-client-334.jar
+ - presto-geospatial-toolkit-334.jar
+ - presto-main-334.jar
+ - presto-matching-334.jar
+ - presto-memory-context-334.jar
+ - presto-parser-334.jar
+ - presto-plugin-toolkit-334.jar
+ - presto-server-main-334.jar
+ - presto-spi-334.jar
* RocksDB JNI
- rocksdbjni-6.10.2.jar
* SnakeYAML
@@ -494,7 +496,7 @@ MIT License
- slf4j-jdk14-1.7.30.jar
* JCL 1.2 Implemented Over SLF4J
- jcl-over-slf4j-1.7.25.jar
- - jcl-over-slf4j-1.7.29.jar
+ - jcl-over-slf4j-1.7.30.jar
* JUL to SLF4J Bridge
- jul-to-slf4j-1.7.25.jar
* Checker Qual
@@ -509,15 +511,20 @@ CDDL-1.1 -- licenses/LICENSE-CDDL-1.1.txt
- javax.annotation-api-1.2.jar
- javax.annotation-api-1.3.2.jar
- javax.activation-1.2.0.jar
- - javax.activation-api-1.2.0.jar
+ - jakarta.activation-1.2.2.jar
- jakarta.activation-api-1.2.1.jar
+ - jakarta.activation-api-1.2.2.jar
+ - jakarta.annotation-api-1.3.5.jar
+ - jakarta.inject-2.6.1.jar
+ - jakarta.validation-api-2.0.2.jar
+ - jakarta.ws.rs-api-2.1.6.jar
* HK2 - Dependency Injection Kernel
- hk2-api-2.6.1.jar
- hk2-locator-2.6.1.jar
- hk2-utils-2.6.1.jar
- aopalliance-repackaged-2.6.1.jar
* Jersey
- - jaxrs-0.195.jar
+ - jaxrs-0.197.jar
- jersey-client-2.31.jar
- jersey-container-servlet-2.31.jar
- jersey-container-servlet-core-2.31.jar
@@ -549,6 +556,7 @@ CDDL-1.1 -- licenses/LICENSE-CDDL-1.1.txt
- mimepull-1.9.13.jar
* XML Bind API
- jakarta.xml.bind-api-2.3.2.jar
+ - jakarta.xml.bind-api-2.3.3.jar
Eclipse Public License - v2.0 -- licenses/LICENSE-EPL-2.0.txt
* jakarta.annotation-api-1.3.5.jar
[pulsar] 33/46: Add Schema.getNativeSchema (#10076)
Posted by eo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 2932b1057a2fda22354ac825e5ff34553131cc52
Author: Enrico Olivelli <eo...@gmail.com>
AuthorDate: Thu Apr 1 17:33:56 2021 +0200
Add Schema.getNativeSchema (#10076)
(cherry picked from commit 4128151412031f9a354d2296b69533d9533a9fbf)
---
.../apache/pulsar/client/api/SimpleSchemaTest.java | 5 +++++
.../java/org/apache/pulsar/client/api/Schema.java | 11 +++++++++++
.../client/impl/schema/AutoConsumeSchema.java | 2 +-
.../client/impl/schema/AvroBaseStructSchema.java | 7 +++++++
.../client/impl/schema/ProtobufNativeSchema.java | 6 ++++++
.../pulsar/client/impl/schema/AvroSchemaTest.java | 9 ++++++++-
.../pulsar/client/impl/schema/JSONSchemaTest.java | 4 ++++
.../impl/schema/ProtobufNativeSchemaTest.java | 23 +++++++++++++++++-----
8 files changed, 60 insertions(+), 7 deletions(-)
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleSchemaTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleSchemaTest.java
index ae44311..802d059 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleSchemaTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleSchemaTest.java
@@ -528,7 +528,12 @@ public class SimpleSchemaTest extends ProducerConsumerBase {
Message<GenericRecord> data = c.receive();
assertNotNull(data.getSchemaVersion());
assertEquals(data.getValue().getField("i"), i);
+ MessageImpl impl = (MessageImpl) data;
+
+ org.apache.avro.Schema avroSchema = (org.apache.avro.Schema) impl.getSchema().getNativeSchema().get();
+ assertNotNull(avroSchema);
}
+
}
}
diff --git a/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Schema.java b/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Schema.java
index 9b41836..a9df5cc 100644
--- a/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Schema.java
+++ b/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Schema.java
@@ -26,6 +26,8 @@ import java.time.LocalDate;
import java.time.LocalDateTime;
import java.time.LocalTime;
import java.util.Date;
+import java.util.Optional;
+
import org.apache.pulsar.client.api.schema.GenericRecord;
import org.apache.pulsar.client.api.schema.GenericSchema;
import org.apache.pulsar.client.api.schema.GenericObject;
@@ -158,6 +160,15 @@ public interface Schema<T> extends Cloneable{
Schema<byte[]> BYTES = DefaultImplementation.newBytesSchema();
/**
+ * Return the native schema that is wrapped by Pulsar API.
+ * For instance with this method you can access the Avro schema
+ * @return the internal schema or null if not present
+ */
+ default Optional<Object> getNativeSchema() {
+ return Optional.empty();
+ }
+
+ /**
* ByteBuffer Schema.
*/
Schema<ByteBuffer> BYTEBUFFER = DefaultImplementation.newByteBufferSchema();
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java
index 1e43fc4..3ec21a2 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java
@@ -117,7 +117,6 @@ public class AutoConsumeSchema implements Schema<GenericRecord> {
}
@Override
-<<<<<<< HEAD
public Schema<GenericRecord> clone() {
Schema<GenericRecord> schema = Schema.AUTO_CONSUME();
if (this.schema != null) {
@@ -131,6 +130,7 @@ public class AutoConsumeSchema implements Schema<GenericRecord> {
return schema;
}
+ @Override
public Optional<Object> getNativeSchema() {
ensureSchemaInitialized();
if (schema == null) {
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AvroBaseStructSchema.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AvroBaseStructSchema.java
index b464731..5e449b4 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AvroBaseStructSchema.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AvroBaseStructSchema.java
@@ -21,6 +21,8 @@ package org.apache.pulsar.client.impl.schema;
import org.apache.avro.Schema;
import org.apache.pulsar.common.schema.SchemaInfo;
+import java.util.Optional;
+
import static java.nio.charset.StandardCharsets.UTF_8;
import static org.apache.pulsar.client.impl.schema.util.SchemaUtil.parseAvroSchema;
@@ -46,4 +48,9 @@ public abstract class AvroBaseStructSchema<T> extends AbstractStructSchema<T>{
public Schema getAvroSchema(){
return schema;
}
+
+ @Override
+ public Optional<Object> getNativeSchema() {
+ return Optional.of(schema);
+ }
}
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/ProtobufNativeSchema.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/ProtobufNativeSchema.java
index 839cb17..02a1b46 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/ProtobufNativeSchema.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/ProtobufNativeSchema.java
@@ -38,6 +38,7 @@ import java.util.HashMap;
import java.util.LinkedList;
import java.util.List;
import java.util.Map;
+import java.util.Optional;
import java.util.function.Consumer;
/**
@@ -102,6 +103,11 @@ public class ProtobufNativeSchema<T extends GeneratedMessageV3> extends Abstract
return ProtobufNativeSchemaUtils.deserialize(this.schemaInfo.getSchema());
}
+ @Override
+ public Optional<Object> getNativeSchema() {
+ return Optional.of(getProtobufNativeSchema());
+ }
+
public static <T extends GeneratedMessageV3> ProtobufNativeSchema<T> of(Class<T> pojo) {
return of(pojo, new HashMap<>());
}
diff --git a/pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/AvroSchemaTest.java b/pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/AvroSchemaTest.java
index 9116e28..dcf3abe 100644
--- a/pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/AvroSchemaTest.java
+++ b/pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/AvroSchemaTest.java
@@ -25,6 +25,7 @@ import static org.apache.pulsar.client.impl.schema.SchemaTestUtils.SCHEMA_AVRO_A
import static org.testng.Assert.assertEquals;
import static org.testng.Assert.assertNotEquals;
import static org.testng.Assert.fail;
+import static org.testng.AssertJUnit.assertSame;
import java.math.BigDecimal;
import java.time.Instant;
@@ -126,10 +127,16 @@ public class AvroSchemaTest {
}
@Test
+ public void testGetNativeSchema() throws SchemaValidationException {
+ AvroSchema<StructWithAnnotations> schema2 = AvroSchema.of(StructWithAnnotations.class);
+ org.apache.avro.Schema avroSchema2 = (Schema) schema2.getNativeSchema().get();
+ assertSame(schema2.schema, avroSchema2);
+ }
+
+ @Test
public void testSchemaDefinition() throws SchemaValidationException {
org.apache.avro.Schema schema1 = ReflectData.get().getSchema(DefaultStruct.class);
AvroSchema<StructWithAnnotations> schema2 = AvroSchema.of(StructWithAnnotations.class);
-
String schemaDef1 = schema1.toString();
String schemaDef2 = new String(schema2.getSchemaInfo().getSchema(), UTF_8);
assertNotEquals(
diff --git a/pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/JSONSchemaTest.java b/pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/JSONSchemaTest.java
index 172bae3..586eb08 100644
--- a/pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/JSONSchemaTest.java
+++ b/pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/JSONSchemaTest.java
@@ -27,10 +27,14 @@ import static org.testng.Assert.assertEquals;
import static org.testng.Assert.assertTrue;
import static org.testng.AssertJUnit.assertSame;
+import static org.testng.AssertJUnit.assertSame;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
import io.netty.buffer.ByteBuf;
import io.netty.buffer.ByteBufAllocator;
import lombok.extern.slf4j.Slf4j;
import org.apache.avro.Schema;
+import org.apache.avro.SchemaValidationException;
import org.apache.pulsar.client.api.SchemaSerializationException;
import org.apache.pulsar.client.api.schema.Field;
import org.apache.pulsar.client.api.schema.GenericRecord;
diff --git a/pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/ProtobufNativeSchemaTest.java b/pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/ProtobufNativeSchemaTest.java
index 9300ce5..59f1b80 100644
--- a/pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/ProtobufNativeSchemaTest.java
+++ b/pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/ProtobufNativeSchemaTest.java
@@ -18,6 +18,7 @@
*/
package org.apache.pulsar.client.impl.schema;
+import com.google.protobuf.Descriptors;
import io.netty.buffer.ByteBuf;
import io.netty.buffer.ByteBufAllocator;
import lombok.extern.slf4j.Slf4j;
@@ -29,6 +30,10 @@ import java.nio.charset.StandardCharsets;
import java.util.Collections;
import java.util.HashMap;
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertNotNull;
+import static org.testng.AssertJUnit.assertSame;
+
@Slf4j
public class ProtobufNativeSchemaTest {
@@ -51,7 +56,7 @@ public class ProtobufNativeSchemaTest {
byte[] bytes = protobufSchema.encode(testMessage);
org.apache.pulsar.client.schema.proto.Test.TestMessage message = protobufSchema.decode(bytes);
- Assert.assertEquals(message.getStringField(), stringFieldValue);
+ assertEquals(message.getStringField(), stringFieldValue);
}
@Test
@@ -59,10 +64,10 @@ public class ProtobufNativeSchemaTest {
ProtobufNativeSchema<org.apache.pulsar.client.schema.proto.Test.TestMessage> protobufSchema
= ProtobufNativeSchema.of(org.apache.pulsar.client.schema.proto.Test.TestMessage.class);
- Assert.assertEquals(protobufSchema.getSchemaInfo().getType(), SchemaType.PROTOBUF_NATIVE);
+ assertEquals(protobufSchema.getSchemaInfo().getType(), SchemaType.PROTOBUF_NATIVE);
- Assert.assertNotNull(ProtobufNativeSchemaUtils.deserialize(protobufSchema.getSchemaInfo().getSchema()));
- Assert.assertEquals(new String(protobufSchema.getSchemaInfo().getSchema(), StandardCharsets.UTF_8), EXPECTED_SCHEMA_JSON);
+ assertNotNull(ProtobufNativeSchemaUtils.deserialize(protobufSchema.getSchemaInfo().getSchema()));
+ assertEquals(new String(protobufSchema.getSchemaInfo().getSchema(), StandardCharsets.UTF_8), EXPECTED_SCHEMA_JSON);
}
@Test
@@ -96,8 +101,16 @@ public class ProtobufNativeSchemaTest {
ByteBuf byteBuf = ByteBufAllocator.DEFAULT.buffer(bytes.length);
byteBuf.writeBytes(bytes);
- Assert.assertEquals(testMessage, protobufSchema.decode(byteBuf));
+ assertEquals(testMessage, protobufSchema.decode(byteBuf));
+
+ }
+ @Test
+ public void testGetNativeSchema() {
+ ProtobufNativeSchema<org.apache.pulsar.client.schema.proto.Test.TestMessage> protobufSchema
+ = ProtobufNativeSchema.of(org.apache.pulsar.client.schema.proto.Test.TestMessage.class);
+ Descriptors.Descriptor nativeSchema = (Descriptors.Descriptor) protobufSchema.getNativeSchema().get();
+ assertNotNull(nativeSchema);
}
}
[pulsar] 05/46: Issue 9130: Pulsar-admin sinks create: bad error
message "java.lang.NullPointerException: path is 'null'." in case of
missing "--name" parameter
Posted by eo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit e3081b3bc47de52f9881dacbbdae330ce9656f30
Author: Enrico Olivelli <eo...@apache.org>
AuthorDate: Tue Jan 5 12:10:10 2021 +0100
Issue 9130: Pulsar-admin sinks create: bad error message "java.lang.NullPointerException: path is 'null'." in case of missing "--name" parameter
---
.../main/java/org/apache/pulsar/client/admin/internal/SinksImpl.java | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/internal/SinksImpl.java b/pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/internal/SinksImpl.java
index a439823..8954adb 100644
--- a/pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/internal/SinksImpl.java
+++ b/pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/internal/SinksImpl.java
@@ -39,6 +39,7 @@ import javax.ws.rs.core.Response;
import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
import org.apache.pulsar.client.admin.PulsarAdminException;
import org.apache.pulsar.client.admin.Sink;
import org.apache.pulsar.client.admin.Sinks;
@@ -237,6 +238,10 @@ public class SinksImpl extends ComponentResource implements Sinks, Sink {
@Override
public CompletableFuture<Void> createSinkAsync(SinkConfig sinkConfig, String fileName) {
final CompletableFuture<Void> future = new CompletableFuture<>();
+ if (StringUtils.isBlank(sinkConfig.getName())) {
+ future.completeExceptionally(new PulsarAdminException("sink name is required"));
+ return future;
+ }
try {
RequestBuilder builder =
post(sink.path(sinkConfig.getTenant())
[pulsar] 07/46: update ds distro required io connectors
Posted by eo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 759c995d2528f72c0804cf8df5b2ee2e999d4399
Author: ming luo <it...@gmail.com>
AuthorDate: Sat Dec 26 20:14:29 2020 +0000
update ds distro required io connectors
---
docker/pulsar-all/Dockerfile | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/docker/pulsar-all/Dockerfile b/docker/pulsar-all/Dockerfile
index 42431fc..5daa406 100644
--- a/docker/pulsar-all/Dockerfile
+++ b/docker/pulsar-all/Dockerfile
@@ -27,5 +27,10 @@ ADD ${PULSAR_OFFLOADER_TARBALL} /
RUN mv /apache-pulsar-offloaders-*/offloaders /offloaders
FROM apachepulsar/pulsar:latest
-COPY --from=pulsar-all /connectors /pulsar/connectors
+COPY --from=pulsar-all /connectors/pulsar-io-elastic-search-*.nar /pulsar/connectors/
+COPY --from=pulsar-all /connectors/pulsar-io-kinesis-*.nar /pulsar/connectors/
+COPY --from=pulsar-all /connectors/pulsar-io-kafka-*.nar /pulsar/connectors/
+COPY --from=pulsar-all /connectors/pulsar-io-jdbc-*.nar /pulsar/connectors/
+COPY --from=pulsar-all /connectors/pulsar-io-debezium-*.nar /pulsar/connectors/
+COPY --from=pulsar-all /connectors/pulsar-io-data-generator-*.nar /pulsar/connectors/
COPY --from=pulsar-all /offloaders /pulsar/offloaders
[pulsar] 34/46: Add JsonRecordBuilder implementation (#10052)
Posted by eo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit b1ee56c5ce61e788ce79d5a2723776406d56f077
Author: Vincent Royer <vr...@strapdata.com>
AuthorDate: Tue Apr 20 04:07:37 2021 +0200
Add JsonRecordBuilder implementation (#10052)
Provide a JSON GenericRecord builder allowing to produce JsonGenericRecord from the GenericJsonSchema.
Add a new class org.apache.pulsar.client.impl.schema.generic.JsonRecordBuilderImpl
Modify the org.apache.pulsar.client.impl.schema.generic.GenericJsonSchema.newRecordBuilder()
Add a unit test in org.apache.pulsar.client.impl.schema.JSONSchemaTest
---
.../test/java/org/apache/pulsar/client/impl/schema/JSONSchemaTest.java | 2 --
1 file changed, 2 deletions(-)
diff --git a/pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/JSONSchemaTest.java b/pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/JSONSchemaTest.java
index 586eb08..d6bd455 100644
--- a/pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/JSONSchemaTest.java
+++ b/pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/JSONSchemaTest.java
@@ -27,8 +27,6 @@ import static org.testng.Assert.assertEquals;
import static org.testng.Assert.assertTrue;
import static org.testng.AssertJUnit.assertSame;
-import static org.testng.AssertJUnit.assertSame;
-
import com.fasterxml.jackson.core.JsonProcessingException;
import io.netty.buffer.ByteBuf;
import io.netty.buffer.ByteBufAllocator;
[pulsar] 44/46: Made OpAddEntry.toString() more robust to nulls to
prevent NPEs (#10548)
Posted by eo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit e2c52457d2719b5f913691bfbd19e2fd660afa09
Author: Devin Bost <de...@users.noreply.github.com>
AuthorDate: Wed May 12 19:00:15 2021 -0600
Made OpAddEntry.toString() more robust to nulls to prevent NPEs (#10548)
Adding test to show we don't throw NPE
Fixed concurrency issue in OpAddEntry.toString()
Co-authored-by: Devin Bost <db...@overstock.com>
(cherry picked from commit d2b7c5eaee925b253201e2a92fcb91ad7290ff93)
---
.../apache/bookkeeper/mledger/impl/OpAddEntry.java | 14 ++++++
.../bookkeeper/mledger/impl/ManagedLedgerTest.java | 58 ++++++++++++++++++++--
2 files changed, 67 insertions(+), 5 deletions(-)
diff --git a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java
index d72d237..b042183 100644
--- a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java
+++ b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java
@@ -301,4 +301,18 @@ class OpAddEntry extends SafeRunnable implements AddCallback, CloseCallback {
}
private static final Logger log = LoggerFactory.getLogger(OpAddEntry.class);
+
+ @Override
+ public String toString() {
+ ManagedLedgerImpl ml = this.ml;
+ LedgerHandle ledger = this.ledger;
+ return "OpAddEntry{" +
+ "mlName=" + ml != null ? ml.getName() : "null" +
+ ", ledgerId=" + ledger != null ? String.valueOf(ledger.getId()) : "null" +
+ ", entryId=" + entryId +
+ ", startTime=" + startTime +
+ ", dataLength=" + dataLength +
+ '}';
+ }
+
}
diff --git a/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerTest.java b/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerTest.java
index 1cc95b4..db9ee42 100644
--- a/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerTest.java
+++ b/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerTest.java
@@ -21,11 +21,8 @@ package org.apache.bookkeeper.mledger.impl;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyInt;
import static org.mockito.ArgumentMatchers.anyString;
-import static org.mockito.Mockito.doAnswer;
-import static org.mockito.Mockito.doNothing;
-import static org.mockito.Mockito.doReturn;
-import static org.mockito.Mockito.mock;
-import static org.mockito.Mockito.spy;
+import static org.mockito.Mockito.*;
+import static org.mockito.Mockito.when;
import static org.testng.Assert.assertEquals;
import static org.testng.Assert.assertFalse;
import static org.testng.Assert.assertNotNull;
@@ -2856,4 +2853,55 @@ public class ManagedLedgerTest extends MockedBookKeeperTestCase {
Thread.sleep(intSleepTimeInMillis + (intSleepTimeInMillis * i));
}
}
+
+ @Test
+ public void testManagedLedgerRollOverIfFull() throws Exception {
+ ManagedLedgerConfig config = new ManagedLedgerConfig();
+ config.setRetentionTime(1, TimeUnit.SECONDS);
+ config.setMaxEntriesPerLedger(2);
+ config.setMinimumRolloverTime(1, TimeUnit.MILLISECONDS);
+ config.setMaximumRolloverTime(500, TimeUnit.MILLISECONDS);
+
+ ManagedLedgerImpl ledger = (ManagedLedgerImpl)factory.open("test_managedLedger_rollOver", config);
+ ManagedCursor cursor = ledger.openCursor("c1");
+
+ int msgNum = 10;
+
+ for (int i = 0; i < msgNum; i++) {
+ ledger.addEntry(new byte[1024 * 1024]);
+ }
+
+ Assert.assertEquals(ledger.getLedgersInfoAsList().size(), msgNum / 2);
+ List<Entry> entries = cursor.readEntries(msgNum);
+ Assert.assertEquals(msgNum, entries.size());
+
+ for (Entry entry : entries) {
+ cursor.markDelete(entry.getPosition());
+ }
+ entries.forEach(e -> e.release());
+
+ // all the messages have benn acknowledged
+ // and all the ledgers have been removed except the last ledger
+ Thread.sleep(1000);
+ Assert.assertEquals(ledger.getLedgersInfoAsList().size(), 1);
+ Assert.assertEquals(ledger.getTotalSize(), 0);
+ }
+
+ @Test
+ public void testOpEntryAdd_toString_doesNotThrowNPE(){
+ ManagedLedger ml = mock(ManagedLedger.class);
+ LedgerHandle ledger = mock(LedgerHandle.class);
+ when(ml.getName()).thenReturn(null);
+ when(ledger.getId()).thenReturn(124L);
+ long entryId = 12L;
+ long startTime = 1245L;
+ int dataLength = 566;
+ String test = "OpAddEntry{" +
+ "mlName=" + ml != null ? ml.getName() : "null" +
+ ", ledgerId=" + ledger != null ? String.valueOf(ledger.getId()) : "null" +
+ ", entryId=" + entryId +
+ ", startTime=" + startTime +
+ ", dataLength=" + dataLength +
+ '}';
+ }
}
[pulsar] 45/46: Fix build
Posted by eo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 567d0771f6302f6cbba7784ecbebf9cfaa9b07b6
Author: Enrico Olivelli <eo...@apache.org>
AuthorDate: Thu May 13 09:40:18 2021 +0200
Fix build
---
.../src/main/java/org/apache/pulsar/client/impl/ConsumerBase.java | 1 -
1 file changed, 1 deletion(-)
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerBase.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerBase.java
index b296a21..e8495c0 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerBase.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerBase.java
@@ -931,7 +931,6 @@ public abstract class ConsumerBase<T> extends HandlerState implements Consumer<T
protected void clearIncomingMessages() {
// release messages if they are pooled messages
- incomingMessages.forEach(Message::release);
incomingMessages.clear();
resetIncomingMessageSize();
}
[pulsar] 15/46: Fix pom file
Posted by eo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
eolivelli pushed a commit to branch 2.7.2_ds_tmp
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 01405a58f96e415f77ff8bbe022f9383ba134089
Author: Enrico Olivelli <eo...@apache.org>
AuthorDate: Fri May 7 17:05:04 2021 +0200
Fix pom file
---
pom.xml | 64 ++++++++++++++++++++++++++++++----------------------------------
1 file changed, 30 insertions(+), 34 deletions(-)
diff --git a/pom.xml b/pom.xml
index c99802d..e9cc7cc 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1,40 +1,36 @@
- <?xml version="1.0" encoding="UTF-8"?>
- <!--
-
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
-
- -->
- <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
- xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
- <modelVersion>4.0.0</modelVersion>
- <packaging>pom</packaging>
- <parent>
- <groupId>org.apache</groupId>
- <artifactId>apache</artifactId>
- <version>18</version>
- </parent>
-
- <groupId>org.apache.pulsar</groupId>
- <artifactId>pulsar</artifactId>
-
- <version>2.7.2_1.0.0</version>
-
- <name>Pulsar</name>
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <packaging>pom</packaging>
+ <parent>
+ <groupId>org.apache</groupId>
+ <artifactId>apache</artifactId>
+ <version>18</version>
+ </parent>
+ <groupId>org.apache.pulsar</groupId>
+ <artifactId>pulsar</artifactId>
+ <version>2.7.2_1.0.0</version>
+ <name>Pulsar</name>
<description>Pulsar is a distributed pub-sub messaging platform with a very
flexible messaging model and an intuitive client API.</description>
<url>https://github.com/apache/pulsar</url>