You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/05/17 03:33:23 UTC
[GitHub] [pulsar] gaoran10 opened a new pull request #10604: [Schema] Support consume multiple schema types messages by AutoConsumeSchema
gaoran10 opened a new pull request #10604:
URL: https://github.com/apache/pulsar/pull/10604
Based on the PR https://github.com/apache/pulsar/pull/10573
### Motivation
Support consuming multiple schema types messages by AutoConsumeSchema.
### Modifications
*Describe the modifications you've done.*
### Verifying this change
Add a new unit test to verify consuming multiple schema type messages by AutoConsumeSchema.
### Does this pull request potentially affect one of the following parts:
*If `yes` was chosen, please highlight the changes*
- Dependencies (does it add or upgrade a dependency): (no)
- The public API: (no)
- The schema: (no)
- The default values of configurations: (no)
- The wire protocol: (no)
- The rest endpoints: (no)
- The admin cli options: (no)
- Anything that affects deployment: (no)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on a change in pull request #10604: [Schema] Support consume multiple schema types messages by AutoConsumeSchema
Posted by GitBox <gi...@apache.org>.
gaoran10 commented on a change in pull request #10604:
URL: https://github.com/apache/pulsar/pull/10604#discussion_r637068217
##########
File path: pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/LookupProxyHandler.java
##########
@@ -357,6 +358,11 @@ public void handleGetSchema(CommandGetSchema commandGetSchema) {
final long clientRequestId = commandGetSchema.getRequestId();
String serviceUrl = getBrokerServiceUrl(clientRequestId);
String topic = commandGetSchema.getTopic();
+ Optional<SchemaVersion> tempVersion = Optional.empty();
Review comment:
Sounds good!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on a change in pull request #10604: [Schema] Support consume multiple schema types messages by AutoConsumeSchema
Posted by GitBox <gi...@apache.org>.
gaoran10 commented on a change in pull request #10604:
URL: https://github.com/apache/pulsar/pull/10604#discussion_r633463546
##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java
##########
@@ -43,73 +44,82 @@
@Slf4j
public class AutoConsumeSchema implements Schema<GenericRecord> {
- private Schema<?> schema;
+ private final Map<SchemaVersion, Schema<?>> schemaMap = new HashMap<>();
private String topicName;
private String componentName;
private SchemaInfoProvider schemaInfoProvider;
+ public AutoConsumeSchema() {
+ schemaMap.put(BytesSchemaVersion.of(new byte[0]), Schema.BYTES);
Review comment:
The `Schema.BYTES` will not be uploaded to the broker and store in the schema storage, if the schema version in the message metadata is empty byte[], it means its schema is `Schema.BYTES`. I'll add this in the `schemaMap` init method and add a comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on a change in pull request #10604: [Schema] Support consume multiple schema types messages by AutoConsumeSchema
Posted by GitBox <gi...@apache.org>.
gaoran10 commented on a change in pull request #10604:
URL: https://github.com/apache/pulsar/pull/10604#discussion_r633484599
##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java
##########
@@ -43,73 +44,82 @@
@Slf4j
public class AutoConsumeSchema implements Schema<GenericRecord> {
- private Schema<?> schema;
+ private final Map<SchemaVersion, Schema<?>> schemaMap = new HashMap<>();
private String topicName;
private String componentName;
private SchemaInfoProvider schemaInfoProvider;
+ public AutoConsumeSchema() {
+ schemaMap.put(BytesSchemaVersion.of(new byte[0]), Schema.BYTES);
+ }
+
+ public void setSchema(SchemaVersion schemaVersion, Schema<?> schema) {
+ schemaMap.put(schemaVersion, schema);
+ }
+
public void setSchema(Schema<?> schema) {
- this.schema = schema;
+ schemaMap.put(SchemaVersion.Latest, schema);
}
- private void ensureSchemaInitialized() {
- checkState(null != schema, "Schema is not initialized before used");
+ private void ensureSchemaInitialized(SchemaVersion schemaVersion) {
+ checkState(schemaMap.containsKey(schemaVersion),
+ "Schema version " + schemaVersion + " is not initialized before used");
}
@Override
public void validate(byte[] message) {
- ensureSchemaInitialized();
+ ensureSchemaInitialized(SchemaVersion.Latest);
- schema.validate(message);
+ schemaMap.get(SchemaVersion.Latest).validate(message);
}
@Override
public byte[] encode(GenericRecord message) {
- ensureSchemaInitialized();
-
throw new UnsupportedOperationException("AutoConsumeSchema is not intended to be used for encoding");
}
@Override
public boolean supportSchemaVersioning() {
- return schema == null || schema.supportSchemaVersioning();
+ return true;
Review comment:
Yes, I also think this is a confusing point. I make some checks, the check `supportSchemaVersioning` is used to get the last version schema of the topic, and determine to decode message data in versioned mode or not, these all could work well.
If using the `SchemaCompatibilityStrategy.ALWAYS_COMPATIBLE` strategy, the topic may have multiple version schemas, the primitive schema may be a part of the multiple version schemas. If using other strategies, I think only the struct schema could have multiple version schemas and the schema type couldn't be changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on a change in pull request #10604: [Schema] Support consume multiple schema types messages by AutoConsumeSchema
Posted by GitBox <gi...@apache.org>.
gaoran10 commented on a change in pull request #10604:
URL: https://github.com/apache/pulsar/pull/10604#discussion_r637499517
##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java
##########
@@ -43,73 +45,97 @@
@Slf4j
public class AutoConsumeSchema implements Schema<GenericRecord> {
- private Schema<?> schema;
+ private final ConcurrentMap<SchemaVersion, Schema<?>> schemaMap = initSchemaMap();
private String topicName;
private String componentName;
private SchemaInfoProvider schemaInfoProvider;
+ private ConcurrentMap<SchemaVersion, Schema<?>> initSchemaMap() {
+ ConcurrentMap<SchemaVersion, Schema<?>> schemaMap = Maps.newConcurrentMap();
+ // The Schema.BYTES will not be uploaded to the broker and store in the schema storage,
+ // if the schema version in the message metadata is empty byte[], it means its schema is Schema.BYTES.
+ schemaMap.put(BytesSchemaVersion.of(new byte[0]), Schema.BYTES);
+ return schemaMap;
+ }
+
+ public void setSchema(SchemaVersion schemaVersion, Schema<?> schema) {
+ schemaMap.put(schemaVersion, schema);
+ }
+
public void setSchema(Schema<?> schema) {
- this.schema = schema;
+ schemaMap.put(SchemaVersion.Latest, schema);
}
- private void ensureSchemaInitialized() {
- checkState(null != schema, "Schema is not initialized before used");
+ private void ensureSchemaInitialized(SchemaVersion schemaVersion) {
+ checkState(schemaMap.containsKey(schemaVersion),
+ "Schema version " + schemaVersion + " is not initialized before used");
}
@Override
public void validate(byte[] message) {
- ensureSchemaInitialized();
+ ensureSchemaInitialized(SchemaVersion.Latest);
- schema.validate(message);
+ schemaMap.get(SchemaVersion.Latest).validate(message);
}
@Override
public byte[] encode(GenericRecord message) {
- ensureSchemaInitialized();
-
throw new UnsupportedOperationException("AutoConsumeSchema is not intended to be used for encoding");
}
@Override
public boolean supportSchemaVersioning() {
- return schema == null || schema.supportSchemaVersioning();
+ return true;
}
public Schema<?> atSchemaVersion(byte[] schemaVersion) {
- fetchSchemaIfNeeded();
- ensureSchemaInitialized();
- if (schema.supportSchemaVersioning() && schema instanceof AbstractSchema) {
- return ((AbstractSchema) schema).atSchemaVersion(schemaVersion);
+ SchemaVersion sv = BytesSchemaVersion.of(schemaVersion);
+ fetchSchemaIfNeeded(sv);
+ ensureSchemaInitialized(sv);
+ Schema<?> topicVersionedSchema = schemaMap.get(sv);
+ if (topicVersionedSchema.supportSchemaVersioning() && topicVersionedSchema instanceof AbstractSchema) {
+ return ((AbstractSchema<?>) topicVersionedSchema).atSchemaVersion(schemaVersion);
} else {
- return schema;
+ return topicVersionedSchema;
}
}
@Override
public GenericRecord decode(byte[] bytes, byte[] schemaVersion) {
- fetchSchemaIfNeeded();
- ensureSchemaInitialized();
- return adapt(schema.decode(bytes, schemaVersion), schemaVersion);
+ SchemaVersion sv = BytesSchemaVersion.of(schemaVersion);
+ fetchSchemaIfNeeded(sv);
+ ensureSchemaInitialized(sv);
+ return adapt(schemaMap.get(sv).decode(bytes, schemaVersion), schemaVersion);
Review comment:
Yes, good idea!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #10604: [Schema] Support consume multiple schema types messages by AutoConsumeSchema
Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #10604:
URL: https://github.com/apache/pulsar/pull/10604#issuecomment-846000321
@gaoran10 I would postpone fixing those problems to a follow up patch
this patch is already quite big.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on a change in pull request #10604: [Schema] Support consume multiple schema types messages by AutoConsumeSchema
Posted by GitBox <gi...@apache.org>.
gaoran10 commented on a change in pull request #10604:
URL: https://github.com/apache/pulsar/pull/10604#discussion_r635701239
##########
File path: pulsar-broker/src/test/java/org/apache/pulsar/schema/SchemaTest.java
##########
@@ -858,36 +862,62 @@ private void checkSchemaForAutoSchema(int index, Message<GenericRecord> message)
Assert.fail("Failed to get reader schema for auto consume multiple schema topic.");
}
Object nativeObject = message.getValue().getNativeObject();
+ String baseTopic = message.getProperty("baseTopic");
JsonNode jsonNode;
- switch (index) {
- case 0:
+ KeyValue kv;
+ switch (baseTopic) {
+ case "bytes_schema":
Assert.assertEquals(new String((byte[]) nativeObject), "bytes value");
break;
- case 1:
+ case "string_schema":
Assert.assertEquals((String) nativeObject, "string value");
break;
- case 2:
+ case "bool_schema":
Assert.assertEquals(nativeObject, Boolean.TRUE);
break;
- case 3:
+ case "json_one_schema":
jsonNode = (JsonNode) nativeObject;
Assert.assertEquals(jsonNode.get("id").intValue(), 1);
break;
- case 4:
+ case "json_three_schema":
jsonNode = (JsonNode) nativeObject;
Assert.assertEquals(jsonNode.get("id").intValue(), 3);
Assert.assertEquals(jsonNode.get("name").textValue(), "ran");
break;
- case 5:
+ case "json_four_schema":
jsonNode = (JsonNode) nativeObject;
Assert.assertEquals(jsonNode.get("id").intValue(), 4);
Assert.assertEquals(jsonNode.get("name").textValue(), "tang");
Assert.assertEquals(jsonNode.get("age").intValue(), 18);
break;
- case 6:
+ case "avro_one_schema":
org.apache.avro.generic.GenericRecord genericRecord =
(org.apache.avro.generic.GenericRecord) nativeObject;
- Assert.assertEquals(genericRecord.get("id"), 1);
+ Assert.assertEquals(genericRecord.get("id"), 10);
+ break;
+ case "k_one_v_three_schema_separate":
+ kv = (KeyValue<?, ?>) nativeObject;
+ jsonNode = ((GenericJsonRecord) kv.getKey()).getJsonNode();
+ Assert.assertEquals(jsonNode.get("id").intValue(), 1);
+ jsonNode = ((GenericJsonRecord) kv.getValue()).getJsonNode();
Review comment:
It seems that using the getNativeObject still needs to constraint convert to JsonNode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on pull request #10604: [Schema] Support consume multiple schema types messages by AutoConsumeSchema
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on pull request #10604:
URL: https://github.com/apache/pulsar/pull/10604#issuecomment-842721933
```
Error: Tests run: 6, Failures: 1, Errors: 0, Skipped: 5, Time elapsed: 0.273 s <<< FAILURE! - in org.apache.pulsar.client.impl.schema.generic.GenericSchemaTest
Error: testAutoAvroSchema(org.apache.pulsar.client.impl.schema.generic.GenericSchemaTest) Time elapsed: 0.034 s <<< FAILURE!
java.lang.NullPointerException
at org.apache.pulsar.client.impl.schema.generic.GenericSchemaTest.verifyFooRecord(GenericSchemaTest.java:191)
at org.apache.pulsar.client.impl.schema.generic.GenericSchemaTest.testEncodeAndDecodeGenericRecord(GenericSchemaTest.java:118)
at org.apache.pulsar.client.impl.schema.generic.GenericSchemaTest.testAutoAvroSchema(GenericSchemaTest.java:81)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a change in pull request #10604: [Schema] Support consume multiple schema types messages by AutoConsumeSchema
Posted by GitBox <gi...@apache.org>.
eolivelli commented on a change in pull request #10604:
URL: https://github.com/apache/pulsar/pull/10604#discussion_r636976987
##########
File path: pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/LookupProxyHandler.java
##########
@@ -357,6 +358,11 @@ public void handleGetSchema(CommandGetSchema commandGetSchema) {
final long clientRequestId = commandGetSchema.getRequestId();
String serviceUrl = getBrokerServiceUrl(clientRequestId);
String topic = commandGetSchema.getTopic();
+ Optional<SchemaVersion> tempVersion = Optional.empty();
Review comment:
what about:
```
final Optional<SchemaVersion> schemaVersion;
if (commandGetSchema.hasSchemaVersion()) {
schemaVersion = Optional.of(commandGetSchema.getSchemaVersion()).map(BytesSchemaVersion::of);
} else {
schemaVersion = Optional.empty()
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a change in pull request #10604: [Schema] Support consume multiple schema types messages by AutoConsumeSchema
Posted by GitBox <gi...@apache.org>.
eolivelli commented on a change in pull request #10604:
URL: https://github.com/apache/pulsar/pull/10604#discussion_r635176768
##########
File path: pulsar-broker/src/test/java/org/apache/pulsar/schema/SchemaTest.java
##########
@@ -858,36 +862,62 @@ private void checkSchemaForAutoSchema(int index, Message<GenericRecord> message)
Assert.fail("Failed to get reader schema for auto consume multiple schema topic.");
}
Object nativeObject = message.getValue().getNativeObject();
+ String baseTopic = message.getProperty("baseTopic");
JsonNode jsonNode;
- switch (index) {
- case 0:
+ KeyValue kv;
+ switch (baseTopic) {
+ case "bytes_schema":
Assert.assertEquals(new String((byte[]) nativeObject), "bytes value");
break;
- case 1:
+ case "string_schema":
Assert.assertEquals((String) nativeObject, "string value");
break;
- case 2:
+ case "bool_schema":
Assert.assertEquals(nativeObject, Boolean.TRUE);
break;
- case 3:
+ case "json_one_schema":
jsonNode = (JsonNode) nativeObject;
Assert.assertEquals(jsonNode.get("id").intValue(), 1);
break;
- case 4:
+ case "json_three_schema":
jsonNode = (JsonNode) nativeObject;
Assert.assertEquals(jsonNode.get("id").intValue(), 3);
Assert.assertEquals(jsonNode.get("name").textValue(), "ran");
break;
- case 5:
+ case "json_four_schema":
jsonNode = (JsonNode) nativeObject;
Assert.assertEquals(jsonNode.get("id").intValue(), 4);
Assert.assertEquals(jsonNode.get("name").textValue(), "tang");
Assert.assertEquals(jsonNode.get("age").intValue(), 18);
break;
- case 6:
+ case "avro_one_schema":
org.apache.avro.generic.GenericRecord genericRecord =
(org.apache.avro.generic.GenericRecord) nativeObject;
- Assert.assertEquals(genericRecord.get("id"), 1);
+ Assert.assertEquals(genericRecord.get("id"), 10);
+ break;
+ case "k_one_v_three_schema_separate":
+ kv = (KeyValue<?, ?>) nativeObject;
Review comment:
this should be a KeyValue<GenericRecord, GenericRecord>
##########
File path: pulsar-broker/src/test/java/org/apache/pulsar/schema/SchemaTest.java
##########
@@ -858,36 +862,62 @@ private void checkSchemaForAutoSchema(int index, Message<GenericRecord> message)
Assert.fail("Failed to get reader schema for auto consume multiple schema topic.");
}
Object nativeObject = message.getValue().getNativeObject();
+ String baseTopic = message.getProperty("baseTopic");
JsonNode jsonNode;
- switch (index) {
- case 0:
+ KeyValue kv;
+ switch (baseTopic) {
+ case "bytes_schema":
Assert.assertEquals(new String((byte[]) nativeObject), "bytes value");
break;
- case 1:
+ case "string_schema":
Assert.assertEquals((String) nativeObject, "string value");
break;
- case 2:
+ case "bool_schema":
Assert.assertEquals(nativeObject, Boolean.TRUE);
break;
- case 3:
+ case "json_one_schema":
jsonNode = (JsonNode) nativeObject;
Assert.assertEquals(jsonNode.get("id").intValue(), 1);
break;
- case 4:
+ case "json_three_schema":
jsonNode = (JsonNode) nativeObject;
Assert.assertEquals(jsonNode.get("id").intValue(), 3);
Assert.assertEquals(jsonNode.get("name").textValue(), "ran");
break;
- case 5:
+ case "json_four_schema":
jsonNode = (JsonNode) nativeObject;
Assert.assertEquals(jsonNode.get("id").intValue(), 4);
Assert.assertEquals(jsonNode.get("name").textValue(), "tang");
Assert.assertEquals(jsonNode.get("age").intValue(), 18);
break;
- case 6:
+ case "avro_one_schema":
org.apache.avro.generic.GenericRecord genericRecord =
(org.apache.avro.generic.GenericRecord) nativeObject;
- Assert.assertEquals(genericRecord.get("id"), 1);
+ Assert.assertEquals(genericRecord.get("id"), 10);
+ break;
+ case "k_one_v_three_schema_separate":
+ kv = (KeyValue<?, ?>) nativeObject;
+ jsonNode = ((GenericJsonRecord) kv.getKey()).getJsonNode();
+ Assert.assertEquals(jsonNode.get("id").intValue(), 1);
+ jsonNode = ((GenericJsonRecord) kv.getValue()).getJsonNode();
Review comment:
what about `kv.getValue().getNativeObject()` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eaba commented on a change in pull request #10604: [Schema] Support consume multiple schema types messages by AutoConsumeSchema
Posted by GitBox <gi...@apache.org>.
eaba commented on a change in pull request #10604:
URL: https://github.com/apache/pulsar/pull/10604#discussion_r633587766
##########
File path: pulsar-common/src/main/java/org/apache/pulsar/common/protocol/schema/SchemaHash.java
##########
@@ -36,22 +37,25 @@
private final HashCode hash;
- private SchemaHash(HashCode hash) {
+ private final SchemaType schemaType;
+
+ private SchemaHash(HashCode hash, SchemaType schemaType) {
this.hash = hash;
+ this.schemaType = schemaType;
}
public static SchemaHash of(Schema schema) {
return of(Optional.ofNullable(schema)
.map(Schema::getSchemaInfo)
- .map(SchemaInfo::getSchema).orElse(new byte[0]));
+ .map(SchemaInfo::getSchema).orElse(new byte[0]), schema.getSchemaInfo().getType());
}
public static SchemaHash of(SchemaData schemaData) {
- return of(schemaData.getData());
+ return of(schemaData.getData(), schemaData.getType());
}
- private static SchemaHash of(byte[] schemaBytes) {
Review comment:
u
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on pull request #10604: [Schema] Support consume multiple schema types messages by AutoConsumeSchema
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on pull request #10604:
URL: https://github.com/apache/pulsar/pull/10604#issuecomment-843668924
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui merged pull request #10604: [Schema] Support consume multiple schema types messages by AutoConsumeSchema
Posted by GitBox <gi...@apache.org>.
codelipenghui merged pull request #10604:
URL: https://github.com/apache/pulsar/pull/10604
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on a change in pull request #10604: [Schema] Support consume multiple schema types messages by AutoConsumeSchema
Posted by GitBox <gi...@apache.org>.
gaoran10 commented on a change in pull request #10604:
URL: https://github.com/apache/pulsar/pull/10604#discussion_r637500307
##########
File path: pulsar-broker/src/test/java/org/apache/pulsar/schema/SchemaTest.java
##########
@@ -781,4 +786,181 @@ public void testNullKey() throws Exception {
assertEquals("foo", message.getValue());
}
+ public void testConsumeMultipleSchemaMessages() throws Exception {
+ final String namespace = "test-namespace-" + randomName(16);
+ String ns = PUBLIC_TENANT + "/" + namespace;
+ admin.namespaces().createNamespace(ns, Sets.newHashSet(CLUSTER_NAME));
+ admin.namespaces().setSchemaCompatibilityStrategy(ns, SchemaCompatibilityStrategy.ALWAYS_COMPATIBLE);
+
+ final String autoProducerTopic = getTopicName(ns, "auto_produce_topic");
+ Producer<byte[]> autoProducer = pulsarClient.newProducer(Schema.AUTO_PRODUCE_BYTES())
+ .topic(autoProducerTopic)
+ .create();
+
+ AtomicInteger totalMsgCnt = new AtomicInteger(0);
+ generateDataByDifferentSchema(ns, "bytes_schema", Schema.BYTES, "bytes value".getBytes(),
+ autoProducer, totalMsgCnt);
+ generateDataByDifferentSchema(ns, "string_schema", Schema.STRING, "string value",
+ autoProducer, totalMsgCnt);
+ generateDataByDifferentSchema(ns, "bool_schema", Schema.BOOL, true,
+ autoProducer, totalMsgCnt);
+ generateDataByDifferentSchema(ns, "json_one_schema", Schema.JSON(Schemas.PersonOne.class),
+ new Schemas.PersonOne(1), autoProducer, totalMsgCnt);
+ generateDataByDifferentSchema(ns, "json_three_schema", Schema.JSON(Schemas.PersonThree.class),
+ new Schemas.PersonThree(3, "ran"), autoProducer, totalMsgCnt);
+ generateDataByDifferentSchema(ns, "json_four_schema", Schema.JSON(Schemas.PersonFour.class),
+ new Schemas.PersonFour(4, "tang", 18), autoProducer, totalMsgCnt);
+ generateDataByDifferentSchema(ns, "avro_one_schema", Schema.AVRO(Schemas.PersonOne.class),
+ new Schemas.PersonOne(10), autoProducer, totalMsgCnt);
+ generateDataByDifferentSchema(ns, "k_one_v_three_schema_separate",
+ Schema.KeyValue(Schema.JSON(Schemas.PersonOne.class),
+ Schema.JSON(Schemas.PersonThree.class), KeyValueEncodingType.SEPARATED),
+ new KeyValue<>(new Schemas.PersonOne(1), new Schemas.PersonThree(3, "kv-separate")),
+ autoProducer, totalMsgCnt);
+ generateDataByDifferentSchema(ns, "k_one_v_four_schema_inline",
+ Schema.KeyValue(Schema.JSON(Schemas.PersonOne.class),
+ Schema.JSON(Schemas.PersonFour.class), KeyValueEncodingType.INLINE),
+ new KeyValue<>(new Schemas.PersonOne(10), new Schemas.PersonFour(30, "kv-inline", 20)),
+ autoProducer, totalMsgCnt);
+ generateDataByDifferentSchema(ns, "k_int_v_three_schema_separate",
+ Schema.KeyValue(Schema.INT32, Schema.JSON(Schemas.PersonThree.class), KeyValueEncodingType.SEPARATED),
+ new KeyValue<>(100, new Schemas.PersonThree(40, "kv-separate")),
+ autoProducer, totalMsgCnt);
+
+ Consumer<GenericRecord> autoConsumer = pulsarClient.newConsumer(Schema.AUTO_CONSUME())
+ .topic(autoProducerTopic)
+ .subscriptionName("test")
+ .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
+ .subscribe();
+
+ Message<GenericRecord> message;
+ for (int i = 0; i < totalMsgCnt.get(); i++) {
+ message = autoConsumer.receive(5, TimeUnit.SECONDS);
+ if (message == null) {
+ Assert.fail("Failed to receive multiple schema message.");
+ }
+ log.info("auto consumer get native object class: {}, value: {}",
+ message.getValue().getNativeObject().getClass(), message.getValue().getNativeObject());
+ checkSchemaForAutoSchema(i, message);
+ }
+ }
+
+ private String getTopicName(String ns, String baseTopic) {
+ return ns + "/" + baseTopic;
+ }
+
+ private void generateDataByDifferentSchema(String ns,
+ String baseTopic,
+ Schema schema,
+ Object data,
+ Producer<?> autoProducer,
+ AtomicInteger totalMsgCnt) throws PulsarClientException {
+ String topic = getTopicName(ns, baseTopic);
+ Producer<Object> producer = pulsarClient.newProducer(schema)
+ .topic(topic)
+ .create();
+ producer.newMessage().value(data).property("baseTopic", baseTopic).send();
+ totalMsgCnt.incrementAndGet();
+
+ Consumer<GenericRecord> consumer = pulsarClient.newConsumer(Schema.AUTO_CONSUME())
+ .topic(topic)
+ .subscriptionName("test")
+ .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
+ .subscribe();
+
+ Message<GenericRecord> message = consumer.receive(5, TimeUnit.SECONDS);
+ if (message == null) {
+ Assert.fail("Failed to receive message for topic " + topic);
+ }
+ if (!message.getReaderSchema().isPresent()) {
+ Assert.fail("Failed to get reader schema for topic " + topic);
+ }
+ message.getValue();
+
+ Schema<?> readerSchema = message.getReaderSchema().get();
+ if (readerSchema instanceof KeyValueSchema
+ && ((KeyValueSchema<?, ?>) readerSchema)
+ .getKeyValueEncodingType().equals(KeyValueEncodingType.SEPARATED)) {
+ autoProducer.newMessage(
+ Schema.AUTO_PRODUCE_BYTES(message.getReaderSchema().get())).keyBytes(message.getKeyBytes())
+ .value(message.getData())
+ .properties(message.getProperties())
+ .send();
+ } else {
+ autoProducer.newMessage(Schema.AUTO_PRODUCE_BYTES(message.getReaderSchema().get()))
+ .properties(message.getProperties())
+ .value(message.getData())
+ .send();
+ }
+ producer.close();
+ consumer.close();
+ }
+
+ private void checkSchemaForAutoSchema(int index, Message<GenericRecord> message) {
+ if (!message.getReaderSchema().isPresent()) {
+ Assert.fail("Failed to get reader schema for auto consume multiple schema topic.");
+ }
+ Object nativeObject = message.getValue().getNativeObject();
+ String baseTopic = message.getProperty("baseTopic");
+ JsonNode jsonNode;
+ KeyValue<?, ?> kv;
+ switch (baseTopic) {
+ case "bytes_schema":
+ Assert.assertEquals(new String((byte[]) nativeObject), "bytes value");
+ break;
+ case "string_schema":
+ Assert.assertEquals((String) nativeObject, "string value");
+ break;
+ case "bool_schema":
+ Assert.assertEquals(nativeObject, Boolean.TRUE);
+ break;
+ case "json_one_schema":
+ jsonNode = (JsonNode) nativeObject;
+ Assert.assertEquals(jsonNode.get("id").intValue(), 1);
+ break;
+ case "json_three_schema":
+ jsonNode = (JsonNode) nativeObject;
+ Assert.assertEquals(jsonNode.get("id").intValue(), 3);
+ Assert.assertEquals(jsonNode.get("name").textValue(), "ran");
+ break;
+ case "json_four_schema":
+ jsonNode = (JsonNode) nativeObject;
+ Assert.assertEquals(jsonNode.get("id").intValue(), 4);
+ Assert.assertEquals(jsonNode.get("name").textValue(), "tang");
+ Assert.assertEquals(jsonNode.get("age").intValue(), 18);
+ break;
+ case "avro_one_schema":
+ org.apache.avro.generic.GenericRecord genericRecord =
Review comment:
It seems that if using the `Schema.AUTO_CONSUME()`, the reader doesn't know the Pojo `PersonOne` so the decoding result is `org.apache.avro.generic.GenericRecord`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a change in pull request #10604: [Schema] Support consume multiple schema types messages by AutoConsumeSchema
Posted by GitBox <gi...@apache.org>.
eolivelli commented on a change in pull request #10604:
URL: https://github.com/apache/pulsar/pull/10604#discussion_r633238917
##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java
##########
@@ -43,73 +44,82 @@
@Slf4j
public class AutoConsumeSchema implements Schema<GenericRecord> {
- private Schema<?> schema;
+ private final Map<SchemaVersion, Schema<?>> schemaMap = new HashMap<>();
private String topicName;
private String componentName;
private SchemaInfoProvider schemaInfoProvider;
+ public AutoConsumeSchema() {
+ schemaMap.put(BytesSchemaVersion.of(new byte[0]), Schema.BYTES);
+ }
+
+ public void setSchema(SchemaVersion schemaVersion, Schema<?> schema) {
+ schemaMap.put(schemaVersion, schema);
+ }
+
public void setSchema(Schema<?> schema) {
- this.schema = schema;
+ schemaMap.put(SchemaVersion.Latest, schema);
}
- private void ensureSchemaInitialized() {
- checkState(null != schema, "Schema is not initialized before used");
+ private void ensureSchemaInitialized(SchemaVersion schemaVersion) {
+ checkState(schemaMap.containsKey(schemaVersion),
+ "Schema version " + schemaVersion + " is not initialized before used");
}
@Override
public void validate(byte[] message) {
- ensureSchemaInitialized();
+ ensureSchemaInitialized(SchemaVersion.Latest);
- schema.validate(message);
+ schemaMap.get(SchemaVersion.Latest).validate(message);
}
@Override
public byte[] encode(GenericRecord message) {
- ensureSchemaInitialized();
-
throw new UnsupportedOperationException("AutoConsumeSchema is not intended to be used for encoding");
}
@Override
public boolean supportSchemaVersioning() {
- return schema == null || schema.supportSchemaVersioning();
+ return true;
Review comment:
I am not sure this is correct.
because for primitive types it is better to return "false"
I see that if we want to support ALWAYS schema compatibility we can have both primitive types and version aware type.
have you thought to the impact of always returning `true` with a topic that has a fixed schema of primitive types ? I would like not to add a significant overhead
##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java
##########
@@ -43,73 +44,82 @@
@Slf4j
public class AutoConsumeSchema implements Schema<GenericRecord> {
- private Schema<?> schema;
+ private final Map<SchemaVersion, Schema<?>> schemaMap = new HashMap<>();
Review comment:
this should be a ConcurrentHashMap because the Schema can be accessed by multiple threads concurrently
for instance a Consumer that consumes more messages and process them in parallel, calling `Message.getReaderSchema() `or `Message.getValue()`
##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java
##########
@@ -43,73 +44,82 @@
@Slf4j
public class AutoConsumeSchema implements Schema<GenericRecord> {
- private Schema<?> schema;
+ private final Map<SchemaVersion, Schema<?>> schemaMap = new HashMap<>();
private String topicName;
private String componentName;
private SchemaInfoProvider schemaInfoProvider;
+ public AutoConsumeSchema() {
+ schemaMap.put(BytesSchemaVersion.of(new byte[0]), Schema.BYTES);
Review comment:
why do we need to assume that an empty schema version is "BYTES" ? is this some kind of placeholder ? in this case we should use a constant and also explain in a comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on a change in pull request #10604: [Schema] Support consume multiple schema types messages by AutoConsumeSchema
Posted by GitBox <gi...@apache.org>.
gaoran10 commented on a change in pull request #10604:
URL: https://github.com/apache/pulsar/pull/10604#discussion_r634456264
##########
File path: pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/generic/MultiVersionSchemaInfoProviderTest.java
##########
@@ -61,7 +62,7 @@ public void testGetSchema() throws Exception {
any(TopicName.class),
any(byte[].class)))
.thenReturn(completableFuture);
- SchemaInfo schemaInfoByVersion = schemaProvider.getSchemaByVersion(new byte[0]).get();
+ SchemaInfo schemaInfoByVersion = schemaProvider.getSchemaByVersion(new LongSchemaVersion(0).bytes()).get();
Review comment:
The change of the class MultiVersionSchemaInfoProvider is not necessary and I revert this test. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a change in pull request #10604: [Schema] Support consume multiple schema types messages by AutoConsumeSchema
Posted by GitBox <gi...@apache.org>.
eolivelli commented on a change in pull request #10604:
URL: https://github.com/apache/pulsar/pull/10604#discussion_r634082029
##########
File path: pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/generic/MultiVersionSchemaInfoProviderTest.java
##########
@@ -61,7 +62,7 @@ public void testGetSchema() throws Exception {
any(TopicName.class),
any(byte[].class)))
.thenReturn(completableFuture);
- SchemaInfo schemaInfoByVersion = schemaProvider.getSchemaByVersion(new byte[0]).get();
+ SchemaInfo schemaInfoByVersion = schemaProvider.getSchemaByVersion(new LongSchemaVersion(0).bytes()).get();
Review comment:
it looks like we are changing the behaviour.
can this be a problem ?
do we need to add some compatibility layer ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on pull request #10604: [Schema] Support consume multiple schema types messages by AutoConsumeSchema
Posted by GitBox <gi...@apache.org>.
gaoran10 commented on pull request #10604:
URL: https://github.com/apache/pulsar/pull/10604#issuecomment-845954875
There are still two problems that need to be resolved when consuming keyValue schema data.
1. The keyValue schema data encoding does not consider the KeyValueEncodingType, so if the key and value schema are some but not some KeyValueEncodingType, this will cause an error when decoding message data.
For example, these two schemas belong to one topic, there will be only one schema version due to they have the same schema data.
```
Schema.KeyValue(Schema.JSON(Schemas.PersonOne.class),
Schema.JSON(Schemas.PersonFour.class), KeyValueEncodingType.SEPARATED)
Schema.KeyValue(Schema.JSON(Schemas.PersonOne.class),
Schema.JSON(Schemas.PersonFour.class), KeyValueEncodingType.INLINE)
```
2. Primitive schemas have empty schema data, this will cause the KeyValue schema data doesn't distinguish different primitive schemas.
For example.
```
Schema.KeyValue(Schema.INT32, Schema.JSON(Schemas.PersonThree.class), KeyValueEncodingType.SEPARATED)
Schema.KeyValue(Schema.BOOL, Schema.JSON(Schemas.PersonThree.class), KeyValueEncodingType.SEPARATED)
```
Maybe we could adjust the key-value schema encoding logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on a change in pull request #10604: [Schema] Support consume multiple schema types messages by AutoConsumeSchema
Posted by GitBox <gi...@apache.org>.
gaoran10 commented on a change in pull request #10604:
URL: https://github.com/apache/pulsar/pull/10604#discussion_r635683202
##########
File path: pulsar-broker/src/test/java/org/apache/pulsar/schema/SchemaTest.java
##########
@@ -858,36 +862,62 @@ private void checkSchemaForAutoSchema(int index, Message<GenericRecord> message)
Assert.fail("Failed to get reader schema for auto consume multiple schema topic.");
}
Object nativeObject = message.getValue().getNativeObject();
+ String baseTopic = message.getProperty("baseTopic");
JsonNode jsonNode;
- switch (index) {
- case 0:
+ KeyValue kv;
+ switch (baseTopic) {
+ case "bytes_schema":
Assert.assertEquals(new String((byte[]) nativeObject), "bytes value");
break;
- case 1:
+ case "string_schema":
Assert.assertEquals((String) nativeObject, "string value");
break;
- case 2:
+ case "bool_schema":
Assert.assertEquals(nativeObject, Boolean.TRUE);
break;
- case 3:
+ case "json_one_schema":
jsonNode = (JsonNode) nativeObject;
Assert.assertEquals(jsonNode.get("id").intValue(), 1);
break;
- case 4:
+ case "json_three_schema":
jsonNode = (JsonNode) nativeObject;
Assert.assertEquals(jsonNode.get("id").intValue(), 3);
Assert.assertEquals(jsonNode.get("name").textValue(), "ran");
break;
- case 5:
+ case "json_four_schema":
jsonNode = (JsonNode) nativeObject;
Assert.assertEquals(jsonNode.get("id").intValue(), 4);
Assert.assertEquals(jsonNode.get("name").textValue(), "tang");
Assert.assertEquals(jsonNode.get("age").intValue(), 18);
break;
- case 6:
+ case "avro_one_schema":
org.apache.avro.generic.GenericRecord genericRecord =
(org.apache.avro.generic.GenericRecord) nativeObject;
- Assert.assertEquals(genericRecord.get("id"), 1);
+ Assert.assertEquals(genericRecord.get("id"), 10);
+ break;
+ case "k_one_v_three_schema_separate":
+ kv = (KeyValue<?, ?>) nativeObject;
Review comment:
Ok, I'll fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eaba commented on a change in pull request #10604: [Schema] Support consume multiple schema types messages by AutoConsumeSchema
Posted by GitBox <gi...@apache.org>.
eaba commented on a change in pull request #10604:
URL: https://github.com/apache/pulsar/pull/10604#discussion_r633587766
##########
File path: pulsar-common/src/main/java/org/apache/pulsar/common/protocol/schema/SchemaHash.java
##########
@@ -36,22 +37,25 @@
private final HashCode hash;
- private SchemaHash(HashCode hash) {
+ private final SchemaType schemaType;
+
+ private SchemaHash(HashCode hash, SchemaType schemaType) {
this.hash = hash;
+ this.schemaType = schemaType;
}
public static SchemaHash of(Schema schema) {
return of(Optional.ofNullable(schema)
.map(Schema::getSchemaInfo)
- .map(SchemaInfo::getSchema).orElse(new byte[0]));
+ .map(SchemaInfo::getSchema).orElse(new byte[0]), schema.getSchemaInfo().getType());
}
public static SchemaHash of(SchemaData schemaData) {
- return of(schemaData.getData());
+ return of(schemaData.getData(), schemaData.getType());
}
- private static SchemaHash of(byte[] schemaBytes) {
Review comment:
u
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on a change in pull request #10604: [Schema] Support consume multiple schema types messages by AutoConsumeSchema
Posted by GitBox <gi...@apache.org>.
gaoran10 commented on a change in pull request #10604:
URL: https://github.com/apache/pulsar/pull/10604#discussion_r633461760
##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java
##########
@@ -43,73 +44,82 @@
@Slf4j
public class AutoConsumeSchema implements Schema<GenericRecord> {
- private Schema<?> schema;
+ private final Map<SchemaVersion, Schema<?>> schemaMap = new HashMap<>();
Review comment:
Yes, I'll fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 commented on a change in pull request #10604: [Schema] Support consume multiple schema types messages by AutoConsumeSchema
Posted by GitBox <gi...@apache.org>.
congbobo184 commented on a change in pull request #10604:
URL: https://github.com/apache/pulsar/pull/10604#discussion_r637487537
##########
File path: pulsar-broker/src/test/java/org/apache/pulsar/schema/SchemaTest.java
##########
@@ -781,4 +786,181 @@ public void testNullKey() throws Exception {
assertEquals("foo", message.getValue());
}
+ public void testConsumeMultipleSchemaMessages() throws Exception {
+ final String namespace = "test-namespace-" + randomName(16);
+ String ns = PUBLIC_TENANT + "/" + namespace;
+ admin.namespaces().createNamespace(ns, Sets.newHashSet(CLUSTER_NAME));
+ admin.namespaces().setSchemaCompatibilityStrategy(ns, SchemaCompatibilityStrategy.ALWAYS_COMPATIBLE);
+
+ final String autoProducerTopic = getTopicName(ns, "auto_produce_topic");
+ Producer<byte[]> autoProducer = pulsarClient.newProducer(Schema.AUTO_PRODUCE_BYTES())
+ .topic(autoProducerTopic)
+ .create();
+
+ AtomicInteger totalMsgCnt = new AtomicInteger(0);
+ generateDataByDifferentSchema(ns, "bytes_schema", Schema.BYTES, "bytes value".getBytes(),
+ autoProducer, totalMsgCnt);
+ generateDataByDifferentSchema(ns, "string_schema", Schema.STRING, "string value",
+ autoProducer, totalMsgCnt);
+ generateDataByDifferentSchema(ns, "bool_schema", Schema.BOOL, true,
+ autoProducer, totalMsgCnt);
+ generateDataByDifferentSchema(ns, "json_one_schema", Schema.JSON(Schemas.PersonOne.class),
+ new Schemas.PersonOne(1), autoProducer, totalMsgCnt);
+ generateDataByDifferentSchema(ns, "json_three_schema", Schema.JSON(Schemas.PersonThree.class),
+ new Schemas.PersonThree(3, "ran"), autoProducer, totalMsgCnt);
+ generateDataByDifferentSchema(ns, "json_four_schema", Schema.JSON(Schemas.PersonFour.class),
+ new Schemas.PersonFour(4, "tang", 18), autoProducer, totalMsgCnt);
+ generateDataByDifferentSchema(ns, "avro_one_schema", Schema.AVRO(Schemas.PersonOne.class),
+ new Schemas.PersonOne(10), autoProducer, totalMsgCnt);
+ generateDataByDifferentSchema(ns, "k_one_v_three_schema_separate",
+ Schema.KeyValue(Schema.JSON(Schemas.PersonOne.class),
+ Schema.JSON(Schemas.PersonThree.class), KeyValueEncodingType.SEPARATED),
+ new KeyValue<>(new Schemas.PersonOne(1), new Schemas.PersonThree(3, "kv-separate")),
+ autoProducer, totalMsgCnt);
+ generateDataByDifferentSchema(ns, "k_one_v_four_schema_inline",
+ Schema.KeyValue(Schema.JSON(Schemas.PersonOne.class),
+ Schema.JSON(Schemas.PersonFour.class), KeyValueEncodingType.INLINE),
+ new KeyValue<>(new Schemas.PersonOne(10), new Schemas.PersonFour(30, "kv-inline", 20)),
+ autoProducer, totalMsgCnt);
+ generateDataByDifferentSchema(ns, "k_int_v_three_schema_separate",
+ Schema.KeyValue(Schema.INT32, Schema.JSON(Schemas.PersonThree.class), KeyValueEncodingType.SEPARATED),
+ new KeyValue<>(100, new Schemas.PersonThree(40, "kv-separate")),
+ autoProducer, totalMsgCnt);
+
+ Consumer<GenericRecord> autoConsumer = pulsarClient.newConsumer(Schema.AUTO_CONSUME())
+ .topic(autoProducerTopic)
+ .subscriptionName("test")
+ .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
+ .subscribe();
+
+ Message<GenericRecord> message;
+ for (int i = 0; i < totalMsgCnt.get(); i++) {
+ message = autoConsumer.receive(5, TimeUnit.SECONDS);
+ if (message == null) {
+ Assert.fail("Failed to receive multiple schema message.");
+ }
+ log.info("auto consumer get native object class: {}, value: {}",
+ message.getValue().getNativeObject().getClass(), message.getValue().getNativeObject());
+ checkSchemaForAutoSchema(i, message);
+ }
+ }
+
+ private String getTopicName(String ns, String baseTopic) {
+ return ns + "/" + baseTopic;
+ }
+
+ private void generateDataByDifferentSchema(String ns,
+ String baseTopic,
+ Schema schema,
+ Object data,
+ Producer<?> autoProducer,
+ AtomicInteger totalMsgCnt) throws PulsarClientException {
+ String topic = getTopicName(ns, baseTopic);
+ Producer<Object> producer = pulsarClient.newProducer(schema)
+ .topic(topic)
+ .create();
+ producer.newMessage().value(data).property("baseTopic", baseTopic).send();
+ totalMsgCnt.incrementAndGet();
+
+ Consumer<GenericRecord> consumer = pulsarClient.newConsumer(Schema.AUTO_CONSUME())
+ .topic(topic)
+ .subscriptionName("test")
+ .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
+ .subscribe();
+
+ Message<GenericRecord> message = consumer.receive(5, TimeUnit.SECONDS);
+ if (message == null) {
+ Assert.fail("Failed to receive message for topic " + topic);
+ }
+ if (!message.getReaderSchema().isPresent()) {
+ Assert.fail("Failed to get reader schema for topic " + topic);
+ }
+ message.getValue();
+
+ Schema<?> readerSchema = message.getReaderSchema().get();
+ if (readerSchema instanceof KeyValueSchema
+ && ((KeyValueSchema<?, ?>) readerSchema)
+ .getKeyValueEncodingType().equals(KeyValueEncodingType.SEPARATED)) {
+ autoProducer.newMessage(
+ Schema.AUTO_PRODUCE_BYTES(message.getReaderSchema().get())).keyBytes(message.getKeyBytes())
+ .value(message.getData())
+ .properties(message.getProperties())
+ .send();
+ } else {
+ autoProducer.newMessage(Schema.AUTO_PRODUCE_BYTES(message.getReaderSchema().get()))
+ .properties(message.getProperties())
+ .value(message.getData())
+ .send();
+ }
+ producer.close();
+ consumer.close();
+ }
+
+ private void checkSchemaForAutoSchema(int index, Message<GenericRecord> message) {
+ if (!message.getReaderSchema().isPresent()) {
+ Assert.fail("Failed to get reader schema for auto consume multiple schema topic.");
+ }
+ Object nativeObject = message.getValue().getNativeObject();
+ String baseTopic = message.getProperty("baseTopic");
+ JsonNode jsonNode;
+ KeyValue<?, ?> kv;
+ switch (baseTopic) {
+ case "bytes_schema":
+ Assert.assertEquals(new String((byte[]) nativeObject), "bytes value");
+ break;
+ case "string_schema":
+ Assert.assertEquals((String) nativeObject, "string value");
+ break;
+ case "bool_schema":
+ Assert.assertEquals(nativeObject, Boolean.TRUE);
+ break;
+ case "json_one_schema":
+ jsonNode = (JsonNode) nativeObject;
+ Assert.assertEquals(jsonNode.get("id").intValue(), 1);
+ break;
+ case "json_three_schema":
+ jsonNode = (JsonNode) nativeObject;
+ Assert.assertEquals(jsonNode.get("id").intValue(), 3);
+ Assert.assertEquals(jsonNode.get("name").textValue(), "ran");
+ break;
+ case "json_four_schema":
+ jsonNode = (JsonNode) nativeObject;
+ Assert.assertEquals(jsonNode.get("id").intValue(), 4);
+ Assert.assertEquals(jsonNode.get("name").textValue(), "tang");
+ Assert.assertEquals(jsonNode.get("age").intValue(), 18);
+ break;
+ case "avro_one_schema":
+ org.apache.avro.generic.GenericRecord genericRecord =
Review comment:
does this type is `PersonOne` not `GenericAvroRecord` is right?
##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java
##########
@@ -43,73 +45,97 @@
@Slf4j
public class AutoConsumeSchema implements Schema<GenericRecord> {
- private Schema<?> schema;
+ private final ConcurrentMap<SchemaVersion, Schema<?>> schemaMap = initSchemaMap();
private String topicName;
private String componentName;
private SchemaInfoProvider schemaInfoProvider;
+ private ConcurrentMap<SchemaVersion, Schema<?>> initSchemaMap() {
+ ConcurrentMap<SchemaVersion, Schema<?>> schemaMap = Maps.newConcurrentMap();
+ // The Schema.BYTES will not be uploaded to the broker and store in the schema storage,
+ // if the schema version in the message metadata is empty byte[], it means its schema is Schema.BYTES.
+ schemaMap.put(BytesSchemaVersion.of(new byte[0]), Schema.BYTES);
+ return schemaMap;
+ }
+
+ public void setSchema(SchemaVersion schemaVersion, Schema<?> schema) {
+ schemaMap.put(schemaVersion, schema);
+ }
+
public void setSchema(Schema<?> schema) {
- this.schema = schema;
+ schemaMap.put(SchemaVersion.Latest, schema);
}
- private void ensureSchemaInitialized() {
- checkState(null != schema, "Schema is not initialized before used");
+ private void ensureSchemaInitialized(SchemaVersion schemaVersion) {
+ checkState(schemaMap.containsKey(schemaVersion),
+ "Schema version " + schemaVersion + " is not initialized before used");
}
@Override
public void validate(byte[] message) {
- ensureSchemaInitialized();
+ ensureSchemaInitialized(SchemaVersion.Latest);
- schema.validate(message);
+ schemaMap.get(SchemaVersion.Latest).validate(message);
}
@Override
public byte[] encode(GenericRecord message) {
- ensureSchemaInitialized();
-
throw new UnsupportedOperationException("AutoConsumeSchema is not intended to be used for encoding");
}
@Override
public boolean supportSchemaVersioning() {
- return schema == null || schema.supportSchemaVersioning();
+ return true;
}
public Schema<?> atSchemaVersion(byte[] schemaVersion) {
- fetchSchemaIfNeeded();
- ensureSchemaInitialized();
- if (schema.supportSchemaVersioning() && schema instanceof AbstractSchema) {
- return ((AbstractSchema) schema).atSchemaVersion(schemaVersion);
+ SchemaVersion sv = BytesSchemaVersion.of(schemaVersion);
+ fetchSchemaIfNeeded(sv);
+ ensureSchemaInitialized(sv);
+ Schema<?> topicVersionedSchema = schemaMap.get(sv);
+ if (topicVersionedSchema.supportSchemaVersioning() && topicVersionedSchema instanceof AbstractSchema) {
+ return ((AbstractSchema<?>) topicVersionedSchema).atSchemaVersion(schemaVersion);
} else {
- return schema;
+ return topicVersionedSchema;
}
}
@Override
public GenericRecord decode(byte[] bytes, byte[] schemaVersion) {
- fetchSchemaIfNeeded();
- ensureSchemaInitialized();
- return adapt(schema.decode(bytes, schemaVersion), schemaVersion);
+ SchemaVersion sv = BytesSchemaVersion.of(schemaVersion);
+ fetchSchemaIfNeeded(sv);
+ ensureSchemaInitialized(sv);
+ return adapt(schemaMap.get(sv).decode(bytes, schemaVersion), schemaVersion);
Review comment:
may don't need decode(bytes, schemaVersion), only use decode(bytes)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 commented on a change in pull request #10604: [Schema] Support consume multiple schema types messages by AutoConsumeSchema
Posted by GitBox <gi...@apache.org>.
congbobo184 commented on a change in pull request #10604:
URL: https://github.com/apache/pulsar/pull/10604#discussion_r637485530
##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java
##########
@@ -43,73 +45,97 @@
@Slf4j
public class AutoConsumeSchema implements Schema<GenericRecord> {
- private Schema<?> schema;
+ private final ConcurrentMap<SchemaVersion, Schema<?>> schemaMap = initSchemaMap();
private String topicName;
private String componentName;
private SchemaInfoProvider schemaInfoProvider;
+ private ConcurrentMap<SchemaVersion, Schema<?>> initSchemaMap() {
+ ConcurrentMap<SchemaVersion, Schema<?>> schemaMap = Maps.newConcurrentMap();
+ // The Schema.BYTES will not be uploaded to the broker and store in the schema storage,
+ // if the schema version in the message metadata is empty byte[], it means its schema is Schema.BYTES.
+ schemaMap.put(BytesSchemaVersion.of(new byte[0]), Schema.BYTES);
+ return schemaMap;
+ }
+
+ public void setSchema(SchemaVersion schemaVersion, Schema<?> schema) {
+ schemaMap.put(schemaVersion, schema);
+ }
+
public void setSchema(Schema<?> schema) {
- this.schema = schema;
+ schemaMap.put(SchemaVersion.Latest, schema);
}
- private void ensureSchemaInitialized() {
- checkState(null != schema, "Schema is not initialized before used");
+ private void ensureSchemaInitialized(SchemaVersion schemaVersion) {
+ checkState(schemaMap.containsKey(schemaVersion),
+ "Schema version " + schemaVersion + " is not initialized before used");
}
@Override
public void validate(byte[] message) {
- ensureSchemaInitialized();
+ ensureSchemaInitialized(SchemaVersion.Latest);
- schema.validate(message);
+ schemaMap.get(SchemaVersion.Latest).validate(message);
}
@Override
public byte[] encode(GenericRecord message) {
- ensureSchemaInitialized();
-
throw new UnsupportedOperationException("AutoConsumeSchema is not intended to be used for encoding");
}
@Override
public boolean supportSchemaVersioning() {
- return schema == null || schema.supportSchemaVersioning();
+ return true;
}
public Schema<?> atSchemaVersion(byte[] schemaVersion) {
- fetchSchemaIfNeeded();
- ensureSchemaInitialized();
- if (schema.supportSchemaVersioning() && schema instanceof AbstractSchema) {
- return ((AbstractSchema) schema).atSchemaVersion(schemaVersion);
+ SchemaVersion sv = BytesSchemaVersion.of(schemaVersion);
+ fetchSchemaIfNeeded(sv);
+ ensureSchemaInitialized(sv);
+ Schema<?> topicVersionedSchema = schemaMap.get(sv);
+ if (topicVersionedSchema.supportSchemaVersioning() && topicVersionedSchema instanceof AbstractSchema) {
+ return ((AbstractSchema<?>) topicVersionedSchema).atSchemaVersion(schemaVersion);
} else {
- return schema;
+ return topicVersionedSchema;
}
}
@Override
public GenericRecord decode(byte[] bytes, byte[] schemaVersion) {
- fetchSchemaIfNeeded();
- ensureSchemaInitialized();
- return adapt(schema.decode(bytes, schemaVersion), schemaVersion);
+ SchemaVersion sv = BytesSchemaVersion.of(schemaVersion);
+ fetchSchemaIfNeeded(sv);
+ ensureSchemaInitialized(sv);
+ return adapt(schemaMap.get(sv).decode(bytes, schemaVersion), schemaVersion);
Review comment:
may don't need decode(bytes, schemaVersion), only use decode(bytes) best?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on a change in pull request #10604: [Schema] Support consume multiple schema types messages by AutoConsumeSchema
Posted by GitBox <gi...@apache.org>.
gaoran10 commented on a change in pull request #10604:
URL: https://github.com/apache/pulsar/pull/10604#discussion_r633461760
##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java
##########
@@ -43,73 +44,82 @@
@Slf4j
public class AutoConsumeSchema implements Schema<GenericRecord> {
- private Schema<?> schema;
+ private final Map<SchemaVersion, Schema<?>> schemaMap = new HashMap<>();
Review comment:
Yes, I'll fix.
##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java
##########
@@ -43,73 +44,82 @@
@Slf4j
public class AutoConsumeSchema implements Schema<GenericRecord> {
- private Schema<?> schema;
+ private final Map<SchemaVersion, Schema<?>> schemaMap = new HashMap<>();
private String topicName;
private String componentName;
private SchemaInfoProvider schemaInfoProvider;
+ public AutoConsumeSchema() {
+ schemaMap.put(BytesSchemaVersion.of(new byte[0]), Schema.BYTES);
Review comment:
The `Schema.BYTES` will not be uploaded to the broker and store in the schema storage, if the schema version in the message metadata is empty byte[], it means its schema is `Schema.BYTES`. I'll add this in the `schemaMap` init method and add a comment.
##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java
##########
@@ -43,73 +44,82 @@
@Slf4j
public class AutoConsumeSchema implements Schema<GenericRecord> {
- private Schema<?> schema;
+ private final Map<SchemaVersion, Schema<?>> schemaMap = new HashMap<>();
private String topicName;
private String componentName;
private SchemaInfoProvider schemaInfoProvider;
+ public AutoConsumeSchema() {
+ schemaMap.put(BytesSchemaVersion.of(new byte[0]), Schema.BYTES);
+ }
+
+ public void setSchema(SchemaVersion schemaVersion, Schema<?> schema) {
+ schemaMap.put(schemaVersion, schema);
+ }
+
public void setSchema(Schema<?> schema) {
- this.schema = schema;
+ schemaMap.put(SchemaVersion.Latest, schema);
}
- private void ensureSchemaInitialized() {
- checkState(null != schema, "Schema is not initialized before used");
+ private void ensureSchemaInitialized(SchemaVersion schemaVersion) {
+ checkState(schemaMap.containsKey(schemaVersion),
+ "Schema version " + schemaVersion + " is not initialized before used");
}
@Override
public void validate(byte[] message) {
- ensureSchemaInitialized();
+ ensureSchemaInitialized(SchemaVersion.Latest);
- schema.validate(message);
+ schemaMap.get(SchemaVersion.Latest).validate(message);
}
@Override
public byte[] encode(GenericRecord message) {
- ensureSchemaInitialized();
-
throw new UnsupportedOperationException("AutoConsumeSchema is not intended to be used for encoding");
}
@Override
public boolean supportSchemaVersioning() {
- return schema == null || schema.supportSchemaVersioning();
+ return true;
Review comment:
Yes, I also think this is a confusing point. I make some checks, the check `supportSchemaVersioning` is used to get the last version schema of the topic, and determine to decode message data in versioned mode or not, these all could work well.
If using the `SchemaCompatibilityStrategy.ALWAYS_COMPATIBLE` strategy, the topic may have multiple version schemas, the primitive schema may be a part of the multiple version schemas. If using other strategies, I think only the struct schema could have multiple version schemas and the schema type couldn't be changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a change in pull request #10604: [Schema] Support consume multiple schema types messages by AutoConsumeSchema
Posted by GitBox <gi...@apache.org>.
eolivelli commented on a change in pull request #10604:
URL: https://github.com/apache/pulsar/pull/10604#discussion_r633238917
##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java
##########
@@ -43,73 +44,82 @@
@Slf4j
public class AutoConsumeSchema implements Schema<GenericRecord> {
- private Schema<?> schema;
+ private final Map<SchemaVersion, Schema<?>> schemaMap = new HashMap<>();
private String topicName;
private String componentName;
private SchemaInfoProvider schemaInfoProvider;
+ public AutoConsumeSchema() {
+ schemaMap.put(BytesSchemaVersion.of(new byte[0]), Schema.BYTES);
+ }
+
+ public void setSchema(SchemaVersion schemaVersion, Schema<?> schema) {
+ schemaMap.put(schemaVersion, schema);
+ }
+
public void setSchema(Schema<?> schema) {
- this.schema = schema;
+ schemaMap.put(SchemaVersion.Latest, schema);
}
- private void ensureSchemaInitialized() {
- checkState(null != schema, "Schema is not initialized before used");
+ private void ensureSchemaInitialized(SchemaVersion schemaVersion) {
+ checkState(schemaMap.containsKey(schemaVersion),
+ "Schema version " + schemaVersion + " is not initialized before used");
}
@Override
public void validate(byte[] message) {
- ensureSchemaInitialized();
+ ensureSchemaInitialized(SchemaVersion.Latest);
- schema.validate(message);
+ schemaMap.get(SchemaVersion.Latest).validate(message);
}
@Override
public byte[] encode(GenericRecord message) {
- ensureSchemaInitialized();
-
throw new UnsupportedOperationException("AutoConsumeSchema is not intended to be used for encoding");
}
@Override
public boolean supportSchemaVersioning() {
- return schema == null || schema.supportSchemaVersioning();
+ return true;
Review comment:
I am not sure this is correct.
because for primitive types it is better to return "false"
I see that if we want to support ALWAYS schema compatibility we can have both primitive types and version aware type.
have you thought to the impact of always returning `true` with a topic that has a fixed schema of primitive types ? I would like not to add a significant overhead
##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java
##########
@@ -43,73 +44,82 @@
@Slf4j
public class AutoConsumeSchema implements Schema<GenericRecord> {
- private Schema<?> schema;
+ private final Map<SchemaVersion, Schema<?>> schemaMap = new HashMap<>();
Review comment:
this should be a ConcurrentHashMap because the Schema can be accessed by multiple threads concurrently
for instance a Consumer that consumes more messages and process them in parallel, calling `Message.getReaderSchema() `or `Message.getValue()`
##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java
##########
@@ -43,73 +44,82 @@
@Slf4j
public class AutoConsumeSchema implements Schema<GenericRecord> {
- private Schema<?> schema;
+ private final Map<SchemaVersion, Schema<?>> schemaMap = new HashMap<>();
private String topicName;
private String componentName;
private SchemaInfoProvider schemaInfoProvider;
+ public AutoConsumeSchema() {
+ schemaMap.put(BytesSchemaVersion.of(new byte[0]), Schema.BYTES);
Review comment:
why do we need to assume that an empty schema version is "BYTES" ? is this some kind of placeholder ? in this case we should use a constant and also explain in a comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eaba commented on a change in pull request #10604: [Schema] Support consume multiple schema types messages by AutoConsumeSchema
Posted by GitBox <gi...@apache.org>.
eaba commented on a change in pull request #10604:
URL: https://github.com/apache/pulsar/pull/10604#discussion_r633587766
##########
File path: pulsar-common/src/main/java/org/apache/pulsar/common/protocol/schema/SchemaHash.java
##########
@@ -36,22 +37,25 @@
private final HashCode hash;
- private SchemaHash(HashCode hash) {
+ private final SchemaType schemaType;
+
+ private SchemaHash(HashCode hash, SchemaType schemaType) {
this.hash = hash;
+ this.schemaType = schemaType;
}
public static SchemaHash of(Schema schema) {
return of(Optional.ofNullable(schema)
.map(Schema::getSchemaInfo)
- .map(SchemaInfo::getSchema).orElse(new byte[0]));
+ .map(SchemaInfo::getSchema).orElse(new byte[0]), schema.getSchemaInfo().getType());
}
public static SchemaHash of(SchemaData schemaData) {
- return of(schemaData.getData());
+ return of(schemaData.getData(), schemaData.getType());
}
- private static SchemaHash of(byte[] schemaBytes) {
Review comment:
u
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org