You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/12/07 08:16:04 UTC
[GitHub] [pulsar] mattisonchao opened a new pull request #13094: [ Java Client ] Support Protobuf ``AutoSchema``
mattisonchao opened a new pull request #13094:
URL: https://github.com/apache/pulsar/pull/13094
Fixes #12969
Master Issue: #12969
### Motivation
- Support protobuf AutoSchema
### Modifications
- Support protobuf AutoSchema
### Verifying this change
- [x] Make sure that the change passes the CI checks.
### Does this pull request potentially affect one of the following parts:
- The schema: (yes)
### Documentation
Need to update docs?
- [x] `doc-required`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao closed pull request #13094: [ Java Client ] Support Protobuf ``AutoSchema``
Posted by GitBox <gi...@apache.org>.
mattisonchao closed pull request #13094:
URL: https://github.com/apache/pulsar/pull/13094
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] frankxieke removed a comment on pull request #13094: [ Client ] Support Protobuf ``AutoSchema``
Posted by GitBox <gi...@apache.org>.
frankxieke removed a comment on pull request #13094:
URL: https://github.com/apache/pulsar/pull/13094#issuecomment-987500768
I think PROTOBUF schema is also baesd on avro. Why not `GenericProtobufSchema` inheritage `GenericSchemaImpl` just like `GenericAvroSchema` and `GenericJsonSchema`. I think they all `AvroBaseStructSchema`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on pull request #13094: [ Client ] Support Protobuf ``AutoSchema``
Posted by GitBox <gi...@apache.org>.
mattisonchao commented on pull request #13094:
URL: https://github.com/apache/pulsar/pull/13094#issuecomment-987570488
> I think PROTOBUF schema is also baesd on avro. Why not `GenericProtobufSchema` inheritage `GenericSchemaImpl` just like `GenericAvroSchema` and `GenericJsonSchema`. I think they all `AvroBaseStructSchema`.
I once wanted to use Avro, but I got some problems when deserializing, I think something might be missing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on pull request #13094: [ Java Client ] Support Protobuf ``AutoSchema``
Posted by GitBox <gi...@apache.org>.
mattisonchao commented on pull request #13094:
URL: https://github.com/apache/pulsar/pull/13094#issuecomment-988379569
@Anonymitaet
Hi , Anonymitaet
i think we just need to add ``Protobuf`` to the ``AUTO_CONSUME`` section as below.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao edited a comment on pull request #13094: [ Java Client ] Support Protobuf ``AutoSchema``
Posted by GitBox <gi...@apache.org>.
mattisonchao edited a comment on pull request #13094:
URL: https://github.com/apache/pulsar/pull/13094#issuecomment-992133368
@congbobo184
I think i got somethings wrong.
When the consumer use ``AUTO_CONSUME``, an error will be generated if the consumer does not have the same Protobuf class.
Now I have tried some methods:
1. Use Avro to support this feature. (Although ``GenericRecord`` can be built, you will get an error when read the value, because we use Protobuf serialization, but use Avro to deserialize).
2. Use Protobuf Descriptor. ( Since only ``FieldDescriptor`` is stored during Serialization, Protobuf Descriptor cannot be generate by this data. )
Could you give me more advice for this PR :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] frankxieke commented on pull request #13094: [ Client ] Support Protobuf ``AutoSchema``
Posted by GitBox <gi...@apache.org>.
frankxieke commented on pull request #13094:
URL: https://github.com/apache/pulsar/pull/13094#issuecomment-987500768
I think PROTOBUF schema is also baesd on avro. Why not `GenericProtobufSchema` inheritage `GenericSchemaImpl` just like `GenericAvroSchema` and `GenericJsonSchema`. I think they all `AvroBaseStructSchema`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on pull request #13094: [ Client ] Support Protobuf ``AutoSchema``
Posted by GitBox <gi...@apache.org>.
mattisonchao commented on pull request #13094:
URL: https://github.com/apache/pulsar/pull/13094#issuecomment-987631632
/pulsarbot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao removed a comment on pull request #13094: [ Java Client ] Support Protobuf ``AutoSchema``
Posted by GitBox <gi...@apache.org>.
mattisonchao removed a comment on pull request #13094:
URL: https://github.com/apache/pulsar/pull/13094#issuecomment-987674424
/pulsarbot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] frankxieke commented on pull request #13094: [ Java Client ] Support Protobuf ``AutoSchema``
Posted by GitBox <gi...@apache.org>.
frankxieke commented on pull request #13094:
URL: https://github.com/apache/pulsar/pull/13094#issuecomment-987721929
> > > @freeznet PTAL :)
> >
> >
> > > I once wanted to use Avro, but I got some problems when deserializing, I think something might be missing.
> >
> >
> > Maybe we should follow this way. That looks like more clear, do not you think?
>
> @frankxieke I have tried to use `GenericProtobufSchema` inheritage `GenericSchemaImpl`, but an exception occurred during parsing. I think it should be a format problem during serialization. maybe i can spend more times to use another way to resolve this problem. but I think we can support the feature first, and then optimize it. What do you think? :)
I am not very sure about this problem. Maybe we should wait for other reviewer's option. thanks for you time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao removed a comment on pull request #13094: [ Client ] Support Protobuf ``AutoSchema``
Posted by GitBox <gi...@apache.org>.
mattisonchao removed a comment on pull request #13094:
URL: https://github.com/apache/pulsar/pull/13094#issuecomment-985154656
/pulsarbot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on pull request #13094: [ Client ] Support Protobuf ``AutoSchema``
Posted by GitBox <gi...@apache.org>.
mattisonchao commented on pull request #13094:
URL: https://github.com/apache/pulsar/pull/13094#issuecomment-985154656
/pulsarbot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on pull request #13094: [ Client ] Support Protobuf ``AutoSchema``
Posted by GitBox <gi...@apache.org>.
mattisonchao commented on pull request #13094:
URL: https://github.com/apache/pulsar/pull/13094#issuecomment-986388365
@codelipenghui @congbobo184 @eolivelli PTAL :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on pull request #13094: [ Java Client ] Support Protobuf ``AutoSchema``
Posted by GitBox <gi...@apache.org>.
mattisonchao commented on pull request #13094:
URL: https://github.com/apache/pulsar/pull/13094#issuecomment-987674329
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] frankxieke commented on a change in pull request #13094: [ Client ] Support Protobuf ``AutoSchema``
Posted by GitBox <gi...@apache.org>.
frankxieke commented on a change in pull request #13094:
URL: https://github.com/apache/pulsar/pull/13094#discussion_r763578352
##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/MultiVersionGenericProtobufReader.java
##########
@@ -0,0 +1,90 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.impl.schema.generic;
+
+import com.google.protobuf.Descriptors;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.avro.Schema;
+import org.apache.avro.protobuf.ProtobufData;
+import org.apache.pulsar.client.api.SchemaSerializationException;
+import org.apache.pulsar.client.api.schema.GenericRecord;
+import org.apache.pulsar.client.api.schema.SchemaReader;
+import org.apache.pulsar.client.impl.schema.SchemaUtils;
+import org.apache.pulsar.client.impl.schema.reader.AbstractMultiVersionReader;
+import org.apache.pulsar.common.protocol.schema.BytesSchemaVersion;
+import org.apache.pulsar.common.schema.SchemaInfo;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.nio.charset.StandardCharsets;
+
+@Slf4j
+public class MultiVersionGenericProtobufReader extends AbstractMultiVersionReader<GenericRecord>
+ implements SchemaReader<GenericRecord> {
+
+ private final boolean useProvidedSchemaAsReaderSchema;
+ private final SchemaInfo schemaInfo;
+ private final Descriptors.Descriptor descriptor;
+
+
+ public MultiVersionGenericProtobufReader(boolean useProvidedSchemaAsReaderSchema,
+ SchemaInfo schemaInfo) {
+ super(new GenericProtobufReader(parseAvroBaseSchemaToProtobuf(schemaInfo)));
Review comment:
parseAvroBaseSchemaToProtobuf will be called two times.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao closed pull request #13094: [ Java Client ] Support Protobuf ``AutoSchema``
Posted by GitBox <gi...@apache.org>.
mattisonchao closed pull request #13094:
URL: https://github.com/apache/pulsar/pull/13094
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao removed a comment on pull request #13094: [ Java Client ] Support Protobuf ``AutoSchema``
Posted by GitBox <gi...@apache.org>.
mattisonchao removed a comment on pull request #13094:
URL: https://github.com/apache/pulsar/pull/13094#issuecomment-989596403
@congbobo184
I feel I did something wrong, I need to resubmit a PR to support this feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on pull request #13094: [ Java Client ] Support Protobuf ``AutoSchema``
Posted by GitBox <gi...@apache.org>.
mattisonchao commented on pull request #13094:
URL: https://github.com/apache/pulsar/pull/13094#issuecomment-989596403
@congbobo184
I feel I did something wrong, I need to resubmit a PR to support this feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on pull request #13094: [ Java Client ] Support Protobuf ``AutoSchema``
Posted by GitBox <gi...@apache.org>.
mattisonchao commented on pull request #13094:
URL: https://github.com/apache/pulsar/pull/13094#issuecomment-989567438
@congbobo184
I have refactored ``Protobuf`` and ``Protobuf Native``.
PTAL, when you have time : )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on pull request #13094: [ Java Client ] Support Protobuf ``AutoSchema``
Posted by GitBox <gi...@apache.org>.
mattisonchao commented on pull request #13094:
URL: https://github.com/apache/pulsar/pull/13094#issuecomment-987728023
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao removed a comment on pull request #13094: [ Java Client ] Support Protobuf ``AutoSchema``
Posted by GitBox <gi...@apache.org>.
mattisonchao removed a comment on pull request #13094:
URL: https://github.com/apache/pulsar/pull/13094#issuecomment-987728193
/pulsarbot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on a change in pull request #13094: [ Client ] Support Protobuf ``AutoSchema``
Posted by GitBox <gi...@apache.org>.
mattisonchao commented on a change in pull request #13094:
URL: https://github.com/apache/pulsar/pull/13094#discussion_r762358150
##########
File path: pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/generic/MultiVersionGenericProtobufReaderTest.java
##########
@@ -0,0 +1,48 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.impl.schema.generic;
+
+import com.google.protobuf.Descriptors;
+import org.apache.avro.Schema;
+import org.apache.avro.protobuf.ProtobufData;
+import org.apache.pulsar.client.impl.schema.SchemaInfoImpl;
+import org.apache.pulsar.common.schema.SchemaInfo;
+import org.apache.pulsar.common.schema.SchemaType;
+import org.junit.Test;
+import org.testng.Assert;
+import java.nio.charset.StandardCharsets;
+
+public class MultiVersionGenericProtobufReaderTest {
+
+ @Test
Review comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on a change in pull request #13094: [ Java Client ] Support Protobuf ``AutoSchema``
Posted by GitBox <gi...@apache.org>.
mattisonchao commented on a change in pull request #13094:
URL: https://github.com/apache/pulsar/pull/13094#discussion_r765467865
##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericProtobufReader.java
##########
@@ -0,0 +1,126 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.impl.schema.generic;
+
+import com.google.protobuf.Descriptors;
+import com.google.protobuf.DynamicMessage;
+import com.google.protobuf.InvalidProtocolBufferException;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.client.api.SchemaSerializationException;
+import org.apache.pulsar.client.api.schema.Field;
+import org.apache.pulsar.client.api.schema.GenericRecord;
+import org.apache.pulsar.client.api.schema.SchemaReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+/**
+ * Generic protobuf reader.
+ *
+ * @see GenericProtobufNativeReader
+ */
+@Slf4j
+public class GenericProtobufReader implements SchemaReader<GenericRecord> {
+ private final Descriptors.Descriptor descriptor;
+ private final byte[] schemaVersion;
+ private final List<Field> fields;
+
+ /**
+ * Create a new Generic protobuf reader by protobuf descriptor.
+ *
+ * @param descriptor protobuf descriptor
+ * @see Descriptors##descriptor
+ */
+ public GenericProtobufReader(Descriptors.Descriptor descriptor) {
+ this(descriptor, null);
+ }
+
+ /**
+ * Create a new Generic protobuf reader by protobuf descriptor and schema version.
+ *
+ * @param descriptor protobuf descriptor
+ * @see Descriptors##descriptor
+ */
+ public GenericProtobufReader(Descriptors.Descriptor descriptor, byte[] schemaVersion) {
+ try {
+ this.schemaVersion = schemaVersion;
+ this.descriptor = descriptor;
+ this.fields = descriptor.getFields()
+ .stream()
+ .map(f -> new Field(f.getName(), f.getIndex()))
+ .collect(Collectors.toList());
+ } catch (Exception e) {
+ log.error("GenericProtobufReader init error", e);
Review comment:
fixed
##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericProtobufRecord.java
##########
@@ -0,0 +1,47 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.impl.schema.generic;
+
+import com.google.protobuf.Descriptors;
+import com.google.protobuf.DynamicMessage;
+import org.apache.pulsar.client.api.schema.Field;
+
+import java.util.List;
+
+/**
+ * Generic protobuf record base on #{GenericProtobufNativeRecord}
+ *
+ * @see GenericProtobufNativeRecord
+ */
+public class GenericProtobufRecord extends GenericProtobufNativeRecord {
Review comment:
fixed
##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/MultiVersionGenericProtobufReader.java
##########
@@ -0,0 +1,94 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.impl.schema.generic;
+
+import com.google.protobuf.Descriptors;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.avro.Schema;
+import org.apache.avro.protobuf.ProtobufData;
+import org.apache.pulsar.client.api.SchemaSerializationException;
+import org.apache.pulsar.client.api.schema.GenericRecord;
+import org.apache.pulsar.client.api.schema.SchemaReader;
+import org.apache.pulsar.client.impl.schema.SchemaUtils;
+import org.apache.pulsar.client.impl.schema.reader.AbstractMultiVersionReader;
+import org.apache.pulsar.common.protocol.schema.BytesSchemaVersion;
+import org.apache.pulsar.common.schema.SchemaInfo;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.nio.charset.StandardCharsets;
+
+@Slf4j
+public class MultiVersionGenericProtobufReader extends AbstractMultiVersionReader<GenericRecord>
Review comment:
fixed
##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/ProtobufRecordBuilderImpl.java
##########
@@ -0,0 +1,72 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.impl.schema.generic;
+
+import com.google.protobuf.Descriptors;
+import com.google.protobuf.DynamicMessage;
+import org.apache.pulsar.client.api.schema.Field;
+import org.apache.pulsar.client.api.schema.GenericRecord;
+import org.apache.pulsar.client.api.schema.GenericRecordBuilder;
+
+public class ProtobufRecordBuilderImpl implements GenericRecordBuilder {
Review comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao closed pull request #13094: [ Java Client ] Support Protobuf ``AutoSchema``
Posted by GitBox <gi...@apache.org>.
mattisonchao closed pull request #13094:
URL: https://github.com/apache/pulsar/pull/13094
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on a change in pull request #13094: [ Client ] Support Protobuf ``AutoSchema``
Posted by GitBox <gi...@apache.org>.
mattisonchao commented on a change in pull request #13094:
URL: https://github.com/apache/pulsar/pull/13094#discussion_r763629980
##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/MultiVersionGenericProtobufReader.java
##########
@@ -0,0 +1,90 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.impl.schema.generic;
+
+import com.google.protobuf.Descriptors;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.avro.Schema;
+import org.apache.avro.protobuf.ProtobufData;
+import org.apache.pulsar.client.api.SchemaSerializationException;
+import org.apache.pulsar.client.api.schema.GenericRecord;
+import org.apache.pulsar.client.api.schema.SchemaReader;
+import org.apache.pulsar.client.impl.schema.SchemaUtils;
+import org.apache.pulsar.client.impl.schema.reader.AbstractMultiVersionReader;
+import org.apache.pulsar.common.protocol.schema.BytesSchemaVersion;
+import org.apache.pulsar.common.schema.SchemaInfo;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.nio.charset.StandardCharsets;
+
+@Slf4j
+public class MultiVersionGenericProtobufReader extends AbstractMultiVersionReader<GenericRecord>
+ implements SchemaReader<GenericRecord> {
+
+ private final boolean useProvidedSchemaAsReaderSchema;
+ private final SchemaInfo schemaInfo;
+ private final Descriptors.Descriptor descriptor;
+
+
+ public MultiVersionGenericProtobufReader(boolean useProvidedSchemaAsReaderSchema,
+ SchemaInfo schemaInfo) {
+ super(new GenericProtobufReader(parseAvroBaseSchemaToProtobuf(schemaInfo)));
Review comment:
I will fix ASAP :)
##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java
##########
@@ -177,6 +178,8 @@ public void configureSchemaInfo(String topicName,
case JSON:
case AVRO:
return extractFromAvroSchema(schemaInfo, useProvidedSchemaAsReaderSchema);
+ case PROTOBUF:
Review comment:
I will fix ASAP :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao removed a comment on pull request #13094: [ Java Client ] Support Protobuf ``AutoSchema``
Posted by GitBox <gi...@apache.org>.
mattisonchao removed a comment on pull request #13094:
URL: https://github.com/apache/pulsar/pull/13094#issuecomment-987631632
/pulsarbot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao edited a comment on pull request #13094: [ Client ] Support Protobuf ``AutoSchema``
Posted by GitBox <gi...@apache.org>.
mattisonchao edited a comment on pull request #13094:
URL: https://github.com/apache/pulsar/pull/13094#issuecomment-987624748
> > @freeznet PTAL :)
>
> > I once wanted to use Avro, but I got some problems when deserializing, I think something might be missing.
>
> Maybe we should follow this way. That looks like more clear, do not you think?
@frankxieke
I have tried to use ``GenericProtobufSchema`` inheritage ``GenericSchemaImpl``, but an exception occurred during parsing. I think it should be a format problem during serialization.
maybe i can spend more times to use another way to resolve this problem. but I think we can support the feature first, and then optimize it. What do you think? :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on pull request #13094: [ Client ] Support Protobuf ``AutoSchema``
Posted by GitBox <gi...@apache.org>.
mattisonchao commented on pull request #13094:
URL: https://github.com/apache/pulsar/pull/13094#issuecomment-987624748
> > @freeznet PTAL :)
>
> > I once wanted to use Avro, but I got some problems when deserializing, I think something might be missing.
>
> Maybe we should follow this way. That looks like more clear, do not you think?
I have tried to use ``GenericProtobufSchema`` inheritage ``GenericSchemaImpl``, but an exception occurred during parsing. I think it should be a format problem during serialization.
maybe i can spend more times to use another way to resolve this problem. but I think we can support the feature first, and then optimize it. What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on pull request #13094: [ Java Client ] Support Protobuf ``AutoSchema``
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on pull request #13094:
URL: https://github.com/apache/pulsar/pull/13094#issuecomment-987838211
Hi @congbobo184 @mattisonchao
For the doc side:
- Need to add `Protobuf` to the [AUTO_CONSUME section](https://pulsar.apache.org/docs/en/next/schema-understand/#auto_consume) as below?
- Need to add any other explanations?
FYI: we already have documented these info:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao removed a comment on pull request #13094: [ Client ] Support Protobuf ``AutoSchema``
Posted by GitBox <gi...@apache.org>.
mattisonchao removed a comment on pull request #13094:
URL: https://github.com/apache/pulsar/pull/13094#issuecomment-984739037
/pulsarbot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Shoothzj commented on a change in pull request #13094: [ Client ] Support Protobuf ``AutoSchema``
Posted by GitBox <gi...@apache.org>.
Shoothzj commented on a change in pull request #13094:
URL: https://github.com/apache/pulsar/pull/13094#discussion_r761956188
##########
File path: pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/generic/MultiVersionGenericProtobufReaderTest.java
##########
@@ -0,0 +1,48 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.impl.schema.generic;
+
+import com.google.protobuf.Descriptors;
+import org.apache.avro.Schema;
+import org.apache.avro.protobuf.ProtobufData;
+import org.apache.pulsar.client.impl.schema.SchemaInfoImpl;
+import org.apache.pulsar.common.schema.SchemaInfo;
+import org.apache.pulsar.common.schema.SchemaType;
+import org.junit.Test;
+import org.testng.Assert;
+import java.nio.charset.StandardCharsets;
+
+public class MultiVersionGenericProtobufReaderTest {
+
+ @Test
Review comment:
use `import org.testng.annotations.Test` instead of `import org.junit.Test;`
##########
File path: pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/generic/MultiVersionGenericProtobufReaderTest.java
##########
@@ -0,0 +1,48 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.impl.schema.generic;
+
+import com.google.protobuf.Descriptors;
+import org.apache.avro.Schema;
+import org.apache.avro.protobuf.ProtobufData;
+import org.apache.pulsar.client.impl.schema.SchemaInfoImpl;
+import org.apache.pulsar.common.schema.SchemaInfo;
+import org.apache.pulsar.common.schema.SchemaType;
+import org.junit.Test;
+import org.testng.Assert;
+import java.nio.charset.StandardCharsets;
+
+public class MultiVersionGenericProtobufReaderTest {
+
+ @Test
+ public void testParseAvroBaseSchemaToProtobuf(){
+ org.apache.pulsar.client.schema.proto.Test.TestMessage testPojo =
Review comment:
There is no need to use the full package name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on pull request #13094: [ Java Client ] Support Protobuf ``AutoSchema``
Posted by GitBox <gi...@apache.org>.
mattisonchao commented on pull request #13094:
URL: https://github.com/apache/pulsar/pull/13094#issuecomment-1002925219
Avro-based protobuf will lose some properties during serialization, and it is difficult to complete the task according to the current situation.
This is why Pulsar created Protobuf-Native Schema.
I think I can try another method to use the current properties to generate the protobuf dynamic descriptor.
If it cannot be completed, I will close this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] frankxieke commented on a change in pull request #13094: [ Client ] Support Protobuf ``AutoSchema``
Posted by GitBox <gi...@apache.org>.
frankxieke commented on a change in pull request #13094:
URL: https://github.com/apache/pulsar/pull/13094#discussion_r762682345
##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoConsumeSchema.java
##########
@@ -177,6 +178,8 @@ public void configureSchemaInfo(String topicName,
case JSON:
case AVRO:
return extractFromAvroSchema(schemaInfo, useProvidedSchemaAsReaderSchema);
+ case PROTOBUF:
Review comment:
space after dot
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] frankxieke commented on pull request #13094: [ Client ] Support Protobuf ``AutoSchema``
Posted by GitBox <gi...@apache.org>.
frankxieke commented on pull request #13094:
URL: https://github.com/apache/pulsar/pull/13094#issuecomment-987500462
I think PROTOBUF schema is also baesd on avro. Why not `GenericProtobufSchema` inheritage `GenericSchemaImpl` just like `GenericAvroSchema` and `GenericJsonSchema`. I think they all `AvroBaseStructSchema`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet edited a comment on pull request #13094: [ Java Client ] Support Protobuf ``AutoSchema``
Posted by GitBox <gi...@apache.org>.
Anonymitaet edited a comment on pull request #13094:
URL: https://github.com/apache/pulsar/pull/13094#issuecomment-987838211
Hi @congbobo184 @mattisonchao
For the doc side:
- Need to add `Protobuf` to the [AUTO_CONSUME section](https://pulsar.apache.org/docs/en/next/schema-understand/#auto_consume) as below?
- Need to add any other explanations?
FYI: we already have documented this info:
https://pulsar.apache.org/docs/en/next/schema-understand/#complex-type
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on pull request #13094: [ Client ] Support Protobuf ``AutoSchema``
Posted by GitBox <gi...@apache.org>.
mattisonchao commented on pull request #13094:
URL: https://github.com/apache/pulsar/pull/13094#issuecomment-984671645
@freeznet PTAL :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao removed a comment on pull request #13094: [ Java Client ] Support Protobuf ``AutoSchema``
Posted by GitBox <gi...@apache.org>.
mattisonchao removed a comment on pull request #13094:
URL: https://github.com/apache/pulsar/pull/13094#issuecomment-987674329
/pulsarbot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao removed a comment on pull request #13094: [ Java Client ] Support Protobuf ``AutoSchema``
Posted by GitBox <gi...@apache.org>.
mattisonchao removed a comment on pull request #13094:
URL: https://github.com/apache/pulsar/pull/13094#issuecomment-989567438
@congbobo184
I have refactored ``Protobuf`` and ``Protobuf Native``.
PTAL, when you have time : )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 commented on a change in pull request #13094: [ Java Client ] Support Protobuf ``AutoSchema``
Posted by GitBox <gi...@apache.org>.
congbobo184 commented on a change in pull request #13094:
URL: https://github.com/apache/pulsar/pull/13094#discussion_r764944799
##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericProtobufReader.java
##########
@@ -0,0 +1,126 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.impl.schema.generic;
+
+import com.google.protobuf.Descriptors;
+import com.google.protobuf.DynamicMessage;
+import com.google.protobuf.InvalidProtocolBufferException;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.client.api.SchemaSerializationException;
+import org.apache.pulsar.client.api.schema.Field;
+import org.apache.pulsar.client.api.schema.GenericRecord;
+import org.apache.pulsar.client.api.schema.SchemaReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+/**
+ * Generic protobuf reader.
+ *
+ * @see GenericProtobufNativeReader
+ */
+@Slf4j
+public class GenericProtobufReader implements SchemaReader<GenericRecord> {
+ private final Descriptors.Descriptor descriptor;
+ private final byte[] schemaVersion;
+ private final List<Field> fields;
+
+ /**
+ * Create a new Generic protobuf reader by protobuf descriptor.
+ *
+ * @param descriptor protobuf descriptor
+ * @see Descriptors##descriptor
+ */
+ public GenericProtobufReader(Descriptors.Descriptor descriptor) {
+ this(descriptor, null);
+ }
+
+ /**
+ * Create a new Generic protobuf reader by protobuf descriptor and schema version.
+ *
+ * @param descriptor protobuf descriptor
+ * @see Descriptors##descriptor
+ */
+ public GenericProtobufReader(Descriptors.Descriptor descriptor, byte[] schemaVersion) {
+ try {
+ this.schemaVersion = schemaVersion;
+ this.descriptor = descriptor;
+ this.fields = descriptor.getFields()
+ .stream()
+ .map(f -> new Field(f.getName(), f.getIndex()))
+ .collect(Collectors.toList());
+ } catch (Exception e) {
+ log.error("GenericProtobufReader init error", e);
Review comment:
why cache Exception?
##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericProtobufRecord.java
##########
@@ -0,0 +1,47 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.impl.schema.generic;
+
+import com.google.protobuf.Descriptors;
+import com.google.protobuf.DynamicMessage;
+import org.apache.pulsar.client.api.schema.Field;
+
+import java.util.List;
+
+/**
+ * Generic protobuf record base on #{GenericProtobufNativeRecord}
+ *
+ * @see GenericProtobufNativeRecord
+ */
+public class GenericProtobufRecord extends GenericProtobufNativeRecord {
Review comment:
may we should extends GenericProtobufRecordBase and GenericProtobufNativeRecord also should extends GenericProtobufRecordBase, GenericProtobufRecord should not extends GenericProtobufNativeRecord
##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/MultiVersionGenericProtobufReader.java
##########
@@ -0,0 +1,94 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.impl.schema.generic;
+
+import com.google.protobuf.Descriptors;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.avro.Schema;
+import org.apache.avro.protobuf.ProtobufData;
+import org.apache.pulsar.client.api.SchemaSerializationException;
+import org.apache.pulsar.client.api.schema.GenericRecord;
+import org.apache.pulsar.client.api.schema.SchemaReader;
+import org.apache.pulsar.client.impl.schema.SchemaUtils;
+import org.apache.pulsar.client.impl.schema.reader.AbstractMultiVersionReader;
+import org.apache.pulsar.common.protocol.schema.BytesSchemaVersion;
+import org.apache.pulsar.common.schema.SchemaInfo;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.nio.charset.StandardCharsets;
+
+@Slf4j
+public class MultiVersionGenericProtobufReader extends AbstractMultiVersionReader<GenericRecord>
Review comment:
the same as MultiVersionGenericProtobufReader
##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/ProtobufRecordBuilderImpl.java
##########
@@ -0,0 +1,72 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.impl.schema.generic;
+
+import com.google.protobuf.Descriptors;
+import com.google.protobuf.DynamicMessage;
+import org.apache.pulsar.client.api.schema.Field;
+import org.apache.pulsar.client.api.schema.GenericRecord;
+import org.apache.pulsar.client.api.schema.GenericRecordBuilder;
+
+public class ProtobufRecordBuilderImpl implements GenericRecordBuilder {
Review comment:
the same as ProtobufNativeRecordBuilderImpl
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on pull request #13094: [ Java Client ] Support Protobuf ``AutoSchema``
Posted by GitBox <gi...@apache.org>.
mattisonchao commented on pull request #13094:
URL: https://github.com/apache/pulsar/pull/13094#issuecomment-992133368
@congbobo184
I think i got somethings wrong.
When the consumer use AUTO_CONSUME, an error will be generated if the consumer does not have the same Protobuf class.
Now I have tried some methods:
1. Use Avro to support this feature. (Although ``GenericRecord`` can be built, you will get an error when read the value, because we use Protobuf serialization, but use Avro to deserialize).
2. Use Protobuf Descriptor. ( Since only ``FieldDescriptor`` is stored during Serialization, Protobuf Descriptor cannot be generate by this data. )
Could you give me more advice for this PR :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on pull request #13094: [ Java Client ] Support Protobuf ``AutoSchema``
Posted by GitBox <gi...@apache.org>.
mattisonchao commented on pull request #13094:
URL: https://github.com/apache/pulsar/pull/13094#issuecomment-998382704
> @congbobo184
>
> I think i got somethings wrong.
>
> When the consumer use `AUTO_CONSUME`, an error will be generated if the consumer does not have the same Protobuf class.
>
> Now I have tried some methods:
>
> 1. Use Avro to support this feature. (Although `GenericRecord` can be built, you will get an error when read the value, because we use Protobuf serialization, but use Avro to deserialize).
> 2. Use Protobuf Descriptor. ( Since only `FieldDescriptor` is stored during Serialization, Protobuf Descriptor cannot be generate by this data. )
>
> 
>
> Could you give me more advice for this PR :)
@eolivelli
Would you have time to help me with this problem ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on a change in pull request #13094: [ Client ] Support Protobuf ``AutoSchema``
Posted by GitBox <gi...@apache.org>.
mattisonchao commented on a change in pull request #13094:
URL: https://github.com/apache/pulsar/pull/13094#discussion_r762358181
##########
File path: pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/generic/MultiVersionGenericProtobufReaderTest.java
##########
@@ -0,0 +1,48 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.impl.schema.generic;
+
+import com.google.protobuf.Descriptors;
+import org.apache.avro.Schema;
+import org.apache.avro.protobuf.ProtobufData;
+import org.apache.pulsar.client.impl.schema.SchemaInfoImpl;
+import org.apache.pulsar.common.schema.SchemaInfo;
+import org.apache.pulsar.common.schema.SchemaType;
+import org.junit.Test;
+import org.testng.Assert;
+import java.nio.charset.StandardCharsets;
+
+public class MultiVersionGenericProtobufReaderTest {
+
+ @Test
+ public void testParseAvroBaseSchemaToProtobuf(){
+ org.apache.pulsar.client.schema.proto.Test.TestMessage testPojo =
Review comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] frankxieke commented on pull request #13094: [ Client ] Support Protobuf ``AutoSchema``
Posted by GitBox <gi...@apache.org>.
frankxieke commented on pull request #13094:
URL: https://github.com/apache/pulsar/pull/13094#issuecomment-987604676
> @freeznet PTAL :)
> I once wanted to use Avro, but I got some problems when deserializing, I think something might be missing.
Maybe we should follow this way. That looks like more clear, do not you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on pull request #13094: [ Client ] Support Protobuf ``AutoSchema``
Posted by GitBox <gi...@apache.org>.
mattisonchao commented on pull request #13094:
URL: https://github.com/apache/pulsar/pull/13094#issuecomment-984739037
/pulsarbot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on pull request #13094: [ Client ] Support Protobuf ``AutoSchema``
Posted by GitBox <gi...@apache.org>.
mattisonchao commented on pull request #13094:
URL: https://github.com/apache/pulsar/pull/13094#issuecomment-985178695
/pulsarbot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org