You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/07/06 15:44:43 UTC
[GitHub] [pulsar] Zhen-hao opened a new pull request #11238: Issue 11007
Zhen-hao opened a new pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238
<!--
### Contribution Checklist
- Name the pull request in the form "[Issue XYZ][component] Title of the pull request", where *XYZ* should be replaced by the actual issue number.
Skip *Issue XYZ* if there is no associated github issue for this pull request.
Skip *component* if you are unsure about which is the best component. E.g. `[docs] Fix typo in produce method`.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- Each pull request should address only one issue, not mix up code from multiple issues.
- Each commit in the pull request has a meaningful commit message
- Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
**(The sections below can be removed for hotfixes of typos)**
-->
*(If this PR fixes a github issue, please add `Fixes #<xyz>`.)*
Fixes #<xyz>
*(or if this PR is one task of a github issue, please add `Master Issue: #<xyz>` to link to the master issue.)*
Master Issue: #<xyz>
### Motivation
*Explain here the context, and why you're making that change. What is the problem you're trying to solve.*
### Modifications
*Describe the modifications you've done.*
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
*(Please pick either of the following options)*
This change is a trivial rework / code cleanup without any test coverage.
*(or)*
This change is already covered by existing tests, such as *(please describe tests)*.
*(or)*
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end deployment with large payloads (10MB)*
- *Extended integration test for recovery after broker failure*
### Does this pull request potentially affect one of the following parts:
*If `yes` was chosen, please highlight the changes*
- Dependencies (does it add or upgrade a dependency): (yes / no)
- The public API: (yes / no)
- The schema: (yes / no / don't know)
- The default values of configurations: (yes / no)
- The wire protocol: (yes / no)
- The rest endpoints: (yes / no)
- The admin cli options: (yes / no)
- Anything that affects deployment: (yes / no / don't know)
### Documentation
- Does this pull request introduce a new feature? (yes / no)
- If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
- If a feature is not applicable for documentation, explain why?
- If a feature is not documented yet in this PR, please create a followup issue for adding the documentation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on a change in pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
sijie commented on a change in pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#discussion_r666560078
##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoProduceValidatedAvroBytesSchema.java
##########
@@ -0,0 +1,100 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.impl.schema;
+
+import static com.google.common.base.Preconditions.checkState;
+
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.schema.SchemaDefinition;
+import org.apache.pulsar.common.schema.SchemaInfo;
+import org.apache.pulsar.common.schema.SchemaType;
+
+import java.util.Optional;
+
+/**
+ * Schema from a native Apache Avro schema.
+ * This class is supposed to be used for working with existing data serialized in Avro, possibly stored in another system like Kafka.
+ * For this reason, it will not perform bytes validation against the schema in encoding and decoding, which are just identify functions.
+ * This class also makes it possible for users to bring in their own Avro serializer/deserializer.
+ */
+public class AutoProduceValidatedAvroBytesSchema<T> implements Schema<byte[]> {
Review comment:
If we call it `NATIVE_AVRO`, why not just change the name to `NativeAvroBytesSchema`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Zhen-hao commented on a change in pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
Zhen-hao commented on a change in pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#discussion_r664740305
##########
File path: pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Schema.java
##########
@@ -426,6 +426,19 @@ default void configureSchemaInfo(String topic, String componentName,
return DefaultImplementation.newAutoProduceSchema(schema);
}
+ /**
+ * Create a schema instance that accepts a serialized Avro payload
+ * without validating it against the schema specified.
+ * It can be useful when migrating data from existing event or message stores.
+ *
+ * @return the auto schema instance
+ * @since 2.9.0
+ * @see #AUTO_PRODUCE_BYTES()
+ */
+ static Schema<byte[]> AUTO_PRODUCE_VALIDATED_AVRO_BYTES(Schema<?> schema) {
Review comment:
my idea was exactly to prevent other schema formats from abusing this feature.
the use cases I described already have some external system to guarantee the validity of event data against schemas and the compatibility of schema.
I had thought about `AUTO_PRODUCE_BYTES(Schema<?> schema, boolean validate)` but decided that it would give this "dangerous option" to many formats that should never use it.
I think it is better to add this feature in a case-by-case fashion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
sijie commented on pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#issuecomment-883023197
@Zhen-hao looks good to me!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Zhen-hao commented on pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
Zhen-hao commented on pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#issuecomment-876180696
@Anonymitaet I see `site2/docs/io-develop.md`, `site2/website/versioned_docs/version-2.8.0/io-develop.md`, and `site2/website/versioned_docs/version-2.8.1/io-develop.md`.
Are the latter two manually maintained?
Should I only change `site2/docs/io-develop.md`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Zhen-hao commented on pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
Zhen-hao commented on pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#issuecomment-879866997
I see `site2/docs/io-develop.md` has a section `Handle schema information` which mentions `AUTO_PRODUCE_BYTES`.
I'm thinking of adding some words about `NATIVE_AVRO`.
maybe @Anonymitaet knows better if this is necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on a change in pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
sijie commented on a change in pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#discussion_r664820336
##########
File path: pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Schema.java
##########
@@ -426,6 +426,19 @@ default void configureSchemaInfo(String topic, String componentName,
return DefaultImplementation.newAutoProduceSchema(schema);
}
+ /**
+ * Create a schema instance that accepts a serialized Avro payload
+ * without validating it against the schema specified.
+ * It can be useful when migrating data from existing event or message stores.
+ *
+ * @return the auto schema instance
+ * @since 2.9.0
+ * @see #AUTO_PRODUCE_BYTES()
+ */
+ static Schema<byte[]> AUTO_PRODUCE_VALIDATED_AVRO_BYTES(Schema<?> schema) {
Review comment:
+1 with the current approach. I think we should make it more explicitly instead of introducing a flag to the `AUTO_PRODUC_BYTES` schema
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#issuecomment-892305341
@Zhen-hao thanks for your great work. Would you like to add docs accordingly? Then you can ping me to review, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#issuecomment-877644588
Usually you can run tests with
mvn verify
On the root directory
I suggest you to not run every test locally, it will take more than an hour.
CI can run the tests for you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Zhen-hao commented on pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
Zhen-hao commented on pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#issuecomment-878056647
to help me fix this test error
```
Error: newNativeAvroProducerForMessageSchemaOnTopicWithMultiVersionSchema(org.apache.pulsar.client.api.SimpleSchemaTest) Time elapsed: 0.134 s <<< FAILURE!
java.lang.AssertionError: Expected SchemaSerializationException to be thrown, but nothing was thrown
at org.testng.Assert.expectThrows(Assert.java:1730)
at org.apache.pulsar.client.api.SimpleSchemaTest.newNativeAvroProducerForMessageSchemaOnTopicWithMultiVersionSchema(SimpleSchemaTest.java:347)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
```
@eolivelli can you please explain the meaning of
```java
Assert.expectThrows(SchemaSerializationException.class, () -> p.send(contentV1));
```
When will `p.send(contentV1)` run?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Zhen-hao commented on pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
Zhen-hao commented on pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#issuecomment-879860029
> @Zhen-hao I believe you can remove the "draft" flag from this PR
Cool. I still need to update some documentation page before we can merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Zhen-hao commented on pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
Zhen-hao commented on pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#issuecomment-879896646
> it is a good idea, btw we can separate code changes from documentation changes. you can follow up with a separate PR.
If other reviews agree, I don't mind doing that;)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet removed a comment on pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
Anonymitaet removed a comment on pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#issuecomment-876190557
@Zhen-hao Would you like to add docs accordingly? Then you can ping me to review, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#issuecomment-899906463
> > @Zhen-hao thanks for your great work. Would you like to add docs accordingly? Then you can ping me to review, thanks
>
> That's the plan. I was overwhelmed by work. I will find time later this week or next week to make a new PR.
Hi @Zhen-hao any progress on the docs? Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Zhen-hao commented on pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
Zhen-hao commented on pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#issuecomment-876202208
I don't understand why the build is failing with
```
Error: Medium: Unread field: org.apache.pulsar.client.impl.schema.AutoProduceValidatedAvroBytesSchema.schema [org.apache.pulsar.client.impl.schema.AutoProduceValidatedAvroBytesSchema] At AutoProduceValidatedAvroBytesSchema.java:[line 43] URF_UNREAD_FIELD
```
In `AutoProduceValidatedAvroBytesSchema`'s super class `AutoProduceBytesSchema`, the `schema` field is used in
```java
@Override
public SchemaInfo getSchemaInfo() {
ensureSchemaInitialized();
return schema.getSchemaInfo();
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#issuecomment-879861912
> Cool. I still need to update some documentation page before we can merge.
where do you want to add docs apart from the JavaDocs ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Zhen-hao edited a comment on pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
Zhen-hao edited a comment on pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#issuecomment-876259485
> OK. I think the easiest to do is copy the same code to the new class.
~~Or, only use `super.schema` in the new class.~~
Never mind. `schema` in the super class is private...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a change in pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
eolivelli commented on a change in pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#discussion_r665201011
##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoProduceValidatedAvroBytesSchema.java
##########
@@ -0,0 +1,75 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.impl.schema;
+
+import static com.google.common.base.Preconditions.checkState;
+
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.schema.SchemaDefinition;
+import org.apache.pulsar.client.internal.DefaultImplementation;
+import org.apache.pulsar.common.schema.SchemaInfo;
+import org.apache.pulsar.common.schema.SchemaType;
+
+import java.util.Optional;
+
+/**
+ * Auto detect schema.
Review comment:
This comment is wrong
##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoProduceValidatedAvroBytesSchema.java
##########
@@ -0,0 +1,75 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.impl.schema;
+
+import static com.google.common.base.Preconditions.checkState;
+
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.schema.SchemaDefinition;
+import org.apache.pulsar.client.internal.DefaultImplementation;
+import org.apache.pulsar.common.schema.SchemaInfo;
+import org.apache.pulsar.common.schema.SchemaType;
+
+import java.util.Optional;
+
+/**
+ * Auto detect schema.
+ */
+public class AutoProduceValidatedAvroBytesSchema<T> extends AutoProduceBytesSchema<T> {
+
+ private Schema<T> schema;
+
+
+ public AutoProduceValidatedAvroBytesSchema(org.apache.avro.Schema schema) {
+ SchemaDefinition schemaDefinition = SchemaDefinition.builder().withJsonDef(schema.toString(false)).build();
+
+ setSchema(DefaultImplementation.newAvroSchema(schemaDefinition));
+ }
+
+ public AutoProduceValidatedAvroBytesSchema(Schema<T> schema) {
+ this.schema = null;
Review comment:
why do you need to set this to `null` if you are setting it in the line below ?
##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoProduceValidatedAvroBytesSchema.java
##########
@@ -0,0 +1,75 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.impl.schema;
+
+import static com.google.common.base.Preconditions.checkState;
+
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.schema.SchemaDefinition;
+import org.apache.pulsar.client.internal.DefaultImplementation;
+import org.apache.pulsar.common.schema.SchemaInfo;
+import org.apache.pulsar.common.schema.SchemaType;
+
+import java.util.Optional;
+
+/**
+ * Auto detect schema.
+ */
+public class AutoProduceValidatedAvroBytesSchema<T> extends AutoProduceBytesSchema<T> {
+
+ private Schema<T> schema;
+
+
+ public AutoProduceValidatedAvroBytesSchema(org.apache.avro.Schema schema) {
+ SchemaDefinition schemaDefinition = SchemaDefinition.builder().withJsonDef(schema.toString(false)).build();
+
+ setSchema(DefaultImplementation.newAvroSchema(schemaDefinition));
Review comment:
here we are inside the "impl" package.
So we must not use DefaultImplementation, use directly the AvroSchema constructor
##########
File path: pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Schema.java
##########
@@ -426,6 +426,19 @@ default void configureSchemaInfo(String topic, String componentName,
return DefaultImplementation.newAutoProduceSchema(schema);
}
+ /**
+ * Create a schema instance that accepts a serialized Avro payload
+ * without validating it against the schema specified.
+ * It can be useful when migrating data from existing event or message stores.
+ *
+ * @return the auto schema instance
+ * @since 2.9.0
+ * @see #AUTO_PRODUCE_BYTES()
+ */
+ static Schema<byte[]> AUTO_PRODUCE_VALIDATED_AVRO_BYTES(Schema<?> schema) {
+ return DefaultImplementation.newAutoProduceValidatedAvroSchema(schema);
Review comment:
IIUC this "schema" is a "org.apache.avro.Schema"
so you have to declare your method as:
`static Schema<byte[]> AUTO_PRODUCE_VALIDATED_AVRO_BYTES(Object nativeAvroSchema)`
you can do the Cast inside AutoProduceValidatedAvroBytesSchema in which you will have the constructor like
```
AutoProduceValidatedAvroBytesSchema(Object schema) {
if (! schema instanceof org.apache.avro.Schema) throw new IllegalArgumentException...
}
```
I suggest to rename the method to `Schema<byte[]> NATIVE_AVRO(Object schema)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a change in pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
eolivelli commented on a change in pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#discussion_r664703366
##########
File path: pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Schema.java
##########
@@ -426,6 +426,19 @@ default void configureSchemaInfo(String topic, String componentName,
return DefaultImplementation.newAutoProduceSchema(schema);
}
+ /**
+ * Create a schema instance that accepts a serialized Avro payload
+ * without validating it against the schema specified.
+ * It can be useful when migrating data from existing event or message stores.
+ *
+ * @return the auto schema instance
+ * @since 2.9.0
+ * @see #AUTO_PRODUCE_BYTES()
+ */
+ static Schema<byte[]> AUTO_PRODUCE_VALIDATED_AVRO_BYTES(Schema<?> schema) {
Review comment:
why not adding simply a method that creates a AUTO_PRODUCE_BYTES schema without validation ?
```
static Schema<byte[]> AUTO_PRODUCE_BYTES(Schema<?> schema, boolean validate) {
return DefaultImplementation.newAutoProduceSchema(schema, validate);
}
```
this "feature" will be useful for other Schemas other than Avro
@jerrypeng @sijie @codelipenghui @congbobo184
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
sijie commented on pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#issuecomment-883023197
@Zhen-hao looks good to me!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Zhen-hao commented on pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
Zhen-hao commented on pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#issuecomment-892430900
> @Zhen-hao thanks for your great work. Would you like to add docs accordingly? Then you can ping me to review, thanks
That's the plan. I was overwhelmed by work. I will find time later this week or next week to make a new PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Zhen-hao commented on pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
Zhen-hao commented on pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#issuecomment-875343384
> @Zhen-hao thanks for your contribution. For this PR, do we need to update docs?
I think so. Maybe next to that of `AUTO_PRODUCE_BYTES`?
I need to figure out why the build is failing and where to add tests.
Any guidance is welcome!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Zhen-hao commented on pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
Zhen-hao commented on pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#issuecomment-879718042
there is one test error I can't figure out:
```
org.apache.pulsar.client.api.SchemaSerializationException: java.io.EOFException
at org.apache.pulsar.client.impl.schema.reader.AvroReader.read(AvroReader.java:82)
at org.apache.pulsar.client.api.schema.SchemaReader.read(SchemaReader.java:41)
at org.apache.pulsar.client.api.SimpleSchemaTest.newNativeAvroProducerForMessageSchemaWithBatch(SimpleSchemaTest.java:597)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.io.EOFException
at org.apache.avro.io.BinaryDecoder.ensureBounds(BinaryDecoder.java:542)
at org.apache.avro.io.BinaryDecoder.readInt(BinaryDecoder.java:173)
at org.apache.avro.io.BinaryDecoder.readIndex(BinaryDecoder.java:493)
at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:282)
at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:187)
at org.apache.avro.reflect.ReflectDatumReader.readField(ReflectDatumReader.java:291)
at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247)
at org.apache.avro.specific.SpecificDatumReader.readRecord(SpecificDatumReader.java:123)
at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
at org.apache.pulsar.client.impl.schema.reader.AvroReader.read(AvroReader.java:80)
... 14 more
```
from [this code](https://github.com/apache/pulsar/pull/11238/files#diff-fc75ba3c89c9736af642bb5bed844836caeb748e606262f97eabd69ef066cdcdR597)
what am I missing here? @eolivelli
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#issuecomment-879853090
@Zhen-hao I believe you can remove the "draft" flag from this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a change in pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
eolivelli commented on a change in pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#discussion_r667338845
##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/NativeAvroBytesSchema.java
##########
@@ -0,0 +1,100 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.impl.schema;
+
+import static com.google.common.base.Preconditions.checkState;
+
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.schema.SchemaDefinition;
+import org.apache.pulsar.common.schema.SchemaInfo;
+import org.apache.pulsar.common.schema.SchemaType;
+
+import java.util.Optional;
+
+/**
+ * Schema from a native Apache Avro schema.
+ * This class is supposed to be used for working with existing data serialized in Avro, possibly stored in another system like Kafka.
+ * For this reason, it will not perform bytes validation against the schema in encoding and decoding, which are just identify functions.
+ * This class also makes it possible for users to bring in their own Avro serializer/deserializer.
+ */
+public class NativeAvroBytesSchema<T> implements Schema<byte[]> {
+
+ private Schema<T> schema;
+
+ public NativeAvroBytesSchema(org.apache.avro.Schema schema) {
+ SchemaDefinition schemaDefinition = SchemaDefinition.builder().withJsonDef(schema.toString(false)).build();
+ this.schema = AvroSchema.of(schemaDefinition);
+ }
+
+ public NativeAvroBytesSchema(Object schema) {
+ this(validateSchema(schema));
+ }
+
+ public void setSchema(Schema<T> schema) {
+ if (SchemaType.AVRO != schema.getSchemaInfo().getType()) {
+ throw new IllegalArgumentException("The type of input schema is not 'SchemaType.AVRO'.");
+ }
+ this.schema = schema;
+ }
+
+ public boolean schemaInitialized() {
+ return schema != null;
+ }
+
+ private static org.apache.avro.Schema validateSchema (Object schema) {
+ if (! (schema instanceof org.apache.avro.Schema)) throw new IllegalArgumentException("The input schema is not of type 'org.apache.avro.Schema'.");
+ return (org.apache.avro.Schema) schema;
+ }
+
+ private void ensureSchemaInitialized() {
+ checkState(schemaInitialized(), "Schema is not initialized before used");
+ }
+
+ @Override
+ public byte[] encode(byte[] message) {
+ ensureSchemaInitialized();
+
+ return message;
+ }
+
+ @Override
+ public byte[] decode(byte[] bytes, byte[] schemaVersion) {
+ ensureSchemaInitialized();
+
+ return bytes;
+ }
+
+ @Override
+ public SchemaInfo getSchemaInfo() {
+ ensureSchemaInitialized();
+
+ return schema.getSchemaInfo();
+ }
+
+ @Override
+ public Optional<Object> getNativeSchema() {
+ return Optional.ofNullable(schema);
Review comment:
Here you have to return the original org.apache.avro.Schema object.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Zhen-hao commented on pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
Zhen-hao commented on pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#issuecomment-877639845
@eolivelli
I've added tests that make sense to me. Let me know if there is anything missing.
I couldn't get the tests to run on my machine because of this:
```
[INFO] Pulsar Build Tools ................................. FAILURE [ 0.633 s]
[INFO] Pulsar ............................................. SKIPPED
[INFO] Pulsar Test Mocks .................................. SKIPPED
[INFO] Pulsar Client :: API ............................... SKIPPED
[INFO] Pulsar Client Admin :: API ......................... SKIPPED
[INFO] Pulsar Common ...................................... SKIPPED
[INFO] Pulsar Metadata .................................... SKIPPED
[INFO] Managed Ledger ..................................... SKIPPED
[INFO] Apache Pulsar :: Jclouds shaded .................... SKIPPED
[INFO] Apache Pulsar :: Tiered Storage :: Parent .......... SKIPPED
[INFO] Apache Pulsar :: Tiered Storage :: JCloud .......... SKIPPED
[INFO] Apache Pulsar :: Tiered Storage :: File System ..... SKIPPED
[INFO] Pulsar ZooKeeper Utils ............................. SKIPPED
[INFO] pulsar-broker-common ............................... SKIPPED
[INFO] Pulsar Transaction :: Parent ....................... SKIPPED
[INFO] Pulsar Transaction :: Common ....................... SKIPPED
[INFO] Apache Pulsar :: Bouncy Castle :: Parent ........... SKIPPED
[INFO] Apache Pulsar :: Bouncy Castle :: BC ............... SKIPPED
[INFO] pulsar-client-messagecrypto-bc ..................... SKIPPED
[INFO] Pulsar Functions :: Parent ......................... SKIPPED
[INFO] Pulsar Functions :: Proto .......................... SKIPPED
[INFO] Pulsar Client Java ................................. SKIPPED
[INFO] Pulsar Discovery Service WAR ....................... SKIPPED
[INFO] Pulsar WebSocket ................................... SKIPPED
[INFO] Apache Pulsar :: Package Management ................ SKIPPED
[INFO] Apache Pulsar :: Package Management :: Core ........ SKIPPED
[INFO] Pulsar Client Admin Original ....................... SKIPPED
[INFO] Pulsar Functions :: API ............................ SKIPPED
[INFO] Pulsar IO :: Parent ................................ SKIPPED
[INFO] Pulsar IO :: IO .................................... SKIPPED
[INFO] Pulsar IO :: Batch Discovery Triggerers ............ SKIPPED
[INFO] pulsar-config-validation ........................... SKIPPED
[INFO] Pulsar Functions :: Utils .......................... SKIPPED
[INFO] Pulsar IO :: IO Common ............................. SKIPPED
[INFO] Pulsar Functions :: Secrets ........................ SKIPPED
[INFO] Pulsar Functions :: Instance ....................... SKIPPED
[INFO] Pulsar Functions :: Runtime ........................ SKIPPED
[INFO] Pulsar IO :: Cassandra ............................. SKIPPED
[INFO] Pulsar IO :: Twitter ............................... SKIPPED
[INFO] Pulsar Functions :: API Examples ................... SKIPPED
[INFO] Pulsar Functions :: Worker ......................... SKIPPED
[INFO] Pulsar Functions :: Local Runner original .......... SKIPPED
[INFO] Pulsar IO :: Batch Data Generator .................. SKIPPED
[INFO] Pulsar IO :: Data Generator ........................ SKIPPED
[INFO] Pulsar Transaction :: Coordinator .................. SKIPPED
[INFO] Pulsar Broker ...................................... SKIPPED
[INFO] Pulsar Shaded Broker ............................... SKIPPED
[INFO] Pulsar Client Java ................................. SKIPPED
[INFO] Pulsar Client 1.x Compatibility Base ............... SKIPPED
[INFO] Pulsar Client 2.x Shaded API ....................... SKIPPED
[INFO] Pulsar Client 1.x Compatibility API ................ SKIPPED
[INFO] Pulsar Client Admin ................................ SKIPPED
[INFO] Pulsar Client Tools ................................ SKIPPED
[INFO] Pulsar Client Tools Test ........................... SKIPPED
[INFO] Pulsar Client All .................................. SKIPPED
[INFO] Pulsar Proxy ....................................... SKIPPED
[INFO] Pulsar Test Client ................................. SKIPPED
[INFO] pulsar-broker-auth-athenz .......................... SKIPPED
[INFO] pulsar-client-auth-athenz .......................... SKIPPED
[INFO] Pulsar SQL :: Parent ............................... SKIPPED
[INFO] Pulsar SQL :: Pulsar Presto Connector Packaging .... SKIPPED
[INFO] Pulsar SQL :: Pulsar Presto Connector .............. SKIPPED
[INFO] Pulsar SQL :: Pulsar Presto Distribution ........... SKIPPED
[INFO] Apache Pulsar :: Docker Images ..................... SKIPPED
[INFO] Apache Pulsar :: Docker Images :: Pulsar Dashboard . SKIPPED
[INFO] pulsar-client-auth-sasl ............................ SKIPPED
[INFO] pulsar-broker-auth-sasl ............................ SKIPPED
[INFO] Pulsar Functions :: Runtime All .................... SKIPPED
[INFO] Pulsar Functions :: Local Runner Shaded ............ SKIPPED
[INFO] Pulsar IO :: Aerospike ............................. SKIPPED
[INFO] Pulsar IO :: Canal ................................. SKIPPED
[INFO] Apache Pulsar :: Kafka Connect Avro Converter shaded SKIPPED
[INFO] Pulsar IO :: Kafka Connect Adaptor ................. SKIPPED
[INFO] Pulsar IO :: Debezium .............................. SKIPPED
[INFO] Pulsar IO :: Debezium :: Core ...................... SKIPPED
[INFO] Pulsar IO :: Debezium :: mongodb ................... SKIPPED
[INFO] Pulsar IO :: Debezium :: mysql ..................... SKIPPED
[INFO] Pulsar IO :: Debezium :: postgres .................. SKIPPED
[INFO] Pulsar IO :: IO AWS ................................ SKIPPED
[INFO] Pulsar IO :: DynamoDB .............................. SKIPPED
[INFO] Pulsar IO :: ElasticSearch ......................... SKIPPED
[INFO] Pulsar IO :: File .................................. SKIPPED
[INFO] Pulsar IO :: Flume ................................. SKIPPED
[INFO] Pulsar IO :: Hbase ................................. SKIPPED
[INFO] Pulsar IO :: Hdfs2 ................................. SKIPPED
[INFO] Pulsar IO :: Hdfs3 ................................. SKIPPED
[INFO] Pulsar IO :: InfluxDB .............................. SKIPPED
[INFO] Pulsar IO :: Jdbc .................................. SKIPPED
[INFO] Pulsar IO :: Jdbc :: Core .......................... SKIPPED
[INFO] Pulsar IO :: Jdbc :: ClickHouse .................... SKIPPED
[INFO] Pulsar IO :: Jdbc :: Mariadb ....................... SKIPPED
[INFO] Pulsar IO :: Jdbc :: Postgres ...................... SKIPPED
[INFO] Pulsar IO :: Jdbc :: Sqlite ........................ SKIPPED
[INFO] Pulsar IO :: Kafka ................................. SKIPPED
[INFO] Pulsar IO :: Kafka Connect Adaptor NAR ............. SKIPPED
[INFO] Pulsar IO :: Kinesis ............................... SKIPPED
[INFO] Pulsar IO :: MongoDB ............................... SKIPPED
[INFO] Pulsar IO :: Netty ................................. SKIPPED
[INFO] Pulsar IO :: NSQ ................................... SKIPPED
[INFO] Pulsar IO :: RabbitMQ .............................. SKIPPED
[INFO] Pulsar IO :: Redis ................................. SKIPPED
[INFO] Pulsar IO :: Solr .................................. SKIPPED
[INFO] Pulsar IO :: Docs .................................. SKIPPED
[INFO] Apache Pulsar :: Bouncy Castle :: BC-FIPS .......... SKIPPED
[INFO] Pulsar Bouncy Castle FIPS Test ..................... SKIPPED
[INFO] Apache Pulsar :: Package Management :: BookKeeper Storage SKIPPED
[INFO] Pulsar :: Distribution ............................. SKIPPED
[INFO] Pulsar :: Distribution :: IO ....................... SKIPPED
[INFO] Pulsar :: Distribution :: Offloader ................ SKIPPED
[INFO] Pulsar :: Distribution :: Server ................... SKIPPED
[INFO] Apache Pulsar :: Docker Images :: Pulsar Latest Version SKIPPED
[INFO] Apache Pulsar :: Docker Images :: Grafana .......... SKIPPED
[INFO] Apache Pulsar :: Docker Images :: Pulsar Latest Version (Include All Components) SKIPPED
[INFO] Apache Pulsar :: Docker Images :: Pulsar Standalone SKIPPED
[INFO] Apache Pulsar :: Tests ............................. SKIPPED
[INFO] Apache Pulsar :: Tests :: Docker Images ............ SKIPPED
[INFO] Apache Pulsar :: Tests :: Docker Images :: Java Test Functions SKIPPED
[INFO] Apache Pulsar :: Tests :: Docker Images :: Latest Version Testing SKIPPED
[INFO] Apache Pulsar :: Tests :: Docker Images :: Java Test Image SKIPPED
[INFO] Apache Pulsar :: Tests :: Integration .............. SKIPPED
[INFO] Apache Pulsar :: Tests :: Backwards Client Compatibility 2.0.0-rc1-incubating SKIPPED
[INFO] Apache Pulsar :: Tests :: Backwards Client Compatibility 2.0.1-incubating SKIPPED
[INFO] Apache Pulsar :: Tests :: Backwards Client Compatibility 2.6.0 SKIPPED
[INFO] Apache Pulsar :: Tests :: Pulsar-Client-All-Shade Test SKIPPED
[INFO] Apache Pulsar :: Tests :: Pulsar-Client-Shade Test . SKIPPED
[INFO] Apache Pulsar :: Tests :: Pulsar-Client-Admin-Shade Test SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.608 s
[INFO] Finished at: 2021-07-10T12:41:30Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M3:test (default-test) on project buildtools: No tests were executed! (Set -DfailIfNoTests=false to ignore this error.) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Zhen-hao commented on a change in pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
Zhen-hao commented on a change in pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#discussion_r665246482
##########
File path: pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Schema.java
##########
@@ -426,6 +426,19 @@ default void configureSchemaInfo(String topic, String componentName,
return DefaultImplementation.newAutoProduceSchema(schema);
}
+ /**
+ * Create a schema instance that accepts a serialized Avro payload
+ * without validating it against the schema specified.
+ * It can be useful when migrating data from existing event or message stores.
+ *
+ * @return the auto schema instance
+ * @since 2.9.0
+ * @see #AUTO_PRODUCE_BYTES()
+ */
+ static Schema<byte[]> AUTO_PRODUCE_VALIDATED_AVRO_BYTES(Schema<?> schema) {
+ return DefaultImplementation.newAutoProduceValidatedAvroSchema(schema);
Review comment:
I see what you mean. will make the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a change in pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
eolivelli commented on a change in pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#discussion_r669385483
##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/NativeAvroBytesSchema.java
##########
@@ -0,0 +1,100 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.impl.schema;
+
+import static com.google.common.base.Preconditions.checkState;
+
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.schema.SchemaDefinition;
+import org.apache.pulsar.common.schema.SchemaInfo;
+import org.apache.pulsar.common.schema.SchemaType;
+
+import java.util.Optional;
+
+/**
+ * Schema from a native Apache Avro schema.
+ * This class is supposed to be used for working with existing data serialized in Avro, possibly stored in another system like Kafka.
+ * For this reason, it will not perform bytes validation against the schema in encoding and decoding, which are just identify functions.
+ * This class also makes it possible for users to bring in their own Avro serializer/deserializer.
+ */
+public class NativeAvroBytesSchema<T> implements Schema<byte[]> {
+
+ private Schema<T> schema;
+
+ public NativeAvroBytesSchema(org.apache.avro.Schema schema) {
+ SchemaDefinition schemaDefinition = SchemaDefinition.builder().withJsonDef(schema.toString(false)).build();
+ this.schema = AvroSchema.of(schemaDefinition);
+ }
+
+ public NativeAvroBytesSchema(Object schema) {
+ this(validateSchema(schema));
+ }
+
+ public void setSchema(Schema<T> schema) {
+ if (SchemaType.AVRO != schema.getSchemaInfo().getType()) {
+ throw new IllegalArgumentException("The type of input schema is not 'SchemaType.AVRO'.");
+ }
+ this.schema = schema;
+ }
+
+ public boolean schemaInitialized() {
+ return schema != null;
+ }
+
+ private static org.apache.avro.Schema validateSchema (Object schema) {
+ if (! (schema instanceof org.apache.avro.Schema)) throw new IllegalArgumentException("The input schema is not of type 'org.apache.avro.Schema'.");
Review comment:
nit:
line to long, please add a new line
##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/NativeAvroBytesSchema.java
##########
@@ -0,0 +1,100 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.impl.schema;
+
+import static com.google.common.base.Preconditions.checkState;
+
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.schema.SchemaDefinition;
+import org.apache.pulsar.common.schema.SchemaInfo;
+import org.apache.pulsar.common.schema.SchemaType;
+
+import java.util.Optional;
+
+/**
+ * Schema from a native Apache Avro schema.
+ * This class is supposed to be used for working with existing data serialized in Avro, possibly stored in another system like Kafka.
+ * For this reason, it will not perform bytes validation against the schema in encoding and decoding, which are just identify functions.
+ * This class also makes it possible for users to bring in their own Avro serializer/deserializer.
+ */
+public class NativeAvroBytesSchema<T> implements Schema<byte[]> {
+
+ private Schema<T> schema;
+
+ public NativeAvroBytesSchema(org.apache.avro.Schema schema) {
+ SchemaDefinition schemaDefinition = SchemaDefinition.builder().withJsonDef(schema.toString(false)).build();
+ this.schema = AvroSchema.of(schemaDefinition);
+ }
+
+ public NativeAvroBytesSchema(Object schema) {
+ this(validateSchema(schema));
+ }
+
+ public void setSchema(Schema<T> schema) {
+ if (SchemaType.AVRO != schema.getSchemaInfo().getType()) {
+ throw new IllegalArgumentException("The type of input schema is not 'SchemaType.AVRO'.");
+ }
+ this.schema = schema;
+ }
+
+ public boolean schemaInitialized() {
+ return schema != null;
+ }
+
+ private static org.apache.avro.Schema validateSchema (Object schema) {
+ if (! (schema instanceof org.apache.avro.Schema)) throw new IllegalArgumentException("The input schema is not of type 'org.apache.avro.Schema'.");
+ return (org.apache.avro.Schema) schema;
+ }
+
+ private void ensureSchemaInitialized() {
+ checkState(schemaInitialized(), "Schema is not initialized before used");
+ }
+
+ @Override
+ public byte[] encode(byte[] message) {
+ ensureSchemaInitialized();
+
+ return message;
+ }
+
+ @Override
+ public byte[] decode(byte[] bytes, byte[] schemaVersion) {
+ ensureSchemaInitialized();
+
+ return bytes;
+ }
+
+ @Override
+ public SchemaInfo getSchemaInfo() {
+ ensureSchemaInitialized();
+
+ return schema.getSchemaInfo();
+ }
+
+ @Override
+ public Optional<Object> getNativeSchema() {
+ return Optional.ofNullable(schema);
Review comment:
this comment looks not addressed yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Zhen-hao commented on pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
Zhen-hao commented on pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#issuecomment-901378726
@Anonymitaet unfortunately, I haven't got to it yet. but it is on my list...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a change in pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
eolivelli commented on a change in pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#discussion_r664867171
##########
File path: pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Schema.java
##########
@@ -426,6 +426,19 @@ default void configureSchemaInfo(String topic, String componentName,
return DefaultImplementation.newAutoProduceSchema(schema);
}
+ /**
+ * Create a schema instance that accepts a serialized Avro payload
+ * without validating it against the schema specified.
+ * It can be useful when migrating data from existing event or message stores.
+ *
+ * @return the auto schema instance
+ * @since 2.9.0
+ * @see #AUTO_PRODUCE_BYTES()
+ */
+ static Schema<byte[]> AUTO_PRODUCE_VALIDATED_AVRO_BYTES(Schema<?> schema) {
Review comment:
Agreed.
Thank you @sijie for your quick answer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#issuecomment-879868181
it is a good idea, btw we can separate code changes from documentation changes. you can follow up with a separate PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
sijie commented on pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#issuecomment-883023197
@Zhen-hao looks good to me!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie merged pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
sijie merged pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Zhen-hao commented on pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
Zhen-hao commented on pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#issuecomment-876258391
> AutoProduceBytesSchema already has a "schema" field, and you are adding a new field that hides the inherited one
OK. I think the easiest to do is copy the same code to the new class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a change in pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
eolivelli commented on a change in pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#discussion_r669427393
##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/NativeAvroBytesSchema.java
##########
@@ -0,0 +1,100 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.impl.schema;
+
+import static com.google.common.base.Preconditions.checkState;
+
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.schema.SchemaDefinition;
+import org.apache.pulsar.common.schema.SchemaInfo;
+import org.apache.pulsar.common.schema.SchemaType;
+
+import java.util.Optional;
+
+/**
+ * Schema from a native Apache Avro schema.
+ * This class is supposed to be used for working with existing data serialized in Avro, possibly stored in another system like Kafka.
+ * For this reason, it will not perform bytes validation against the schema in encoding and decoding, which are just identify functions.
+ * This class also makes it possible for users to bring in their own Avro serializer/deserializer.
+ */
+public class NativeAvroBytesSchema<T> implements Schema<byte[]> {
+
+ private Schema<T> schema;
+ private org.apache.avro.Schema nativeSchema;
+
+ public NativeAvroBytesSchema(org.apache.avro.Schema schema) {
+ setSchema(schema);
+ }
+
+ public NativeAvroBytesSchema(Object schema) {
+ this(validateSchema(schema));
+ }
+
+ public void setSchema(org.apache.avro.Schema schema) {
+ SchemaDefinition schemaDefinition = SchemaDefinition.builder().withJsonDef(schema.toString(false)).build();
+ this.nativeSchema = schema;
+ this.schema = AvroSchema.of(schemaDefinition);
+ }
+
+ public boolean schemaInitialized() {
+ return schema != null;
+ }
+
+ private static org.apache.avro.Schema validateSchema (Object schema) {
+ if (! (schema instanceof org.apache.avro.Schema))
+ throw new IllegalArgumentException("The input schema is not of type 'org.apache.avro.Schema'.");
+ return (org.apache.avro.Schema) schema;
+ }
+
+ private void ensureSchemaInitialized() {
+ checkState(schemaInitialized(), "Schema is not initialized before used");
+ }
+
+ @Override
+ public byte[] encode(byte[] message) {
+ ensureSchemaInitialized();
+
+ return message;
+ }
+
+ @Override
+ public byte[] decode(byte[] bytes, byte[] schemaVersion) {
+ ensureSchemaInitialized();
+
+ return bytes;
+ }
+
+ @Override
+ public SchemaInfo getSchemaInfo() {
+ ensureSchemaInitialized();
+
+ return schema.getSchemaInfo();
+ }
+
+ @Override
+ public Optional<Object> getNativeSchema() {
+ return Optional.ofNullable(this.nativeSchema);
Review comment:
it cannot be null
##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/NativeAvroBytesSchema.java
##########
@@ -0,0 +1,100 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.impl.schema;
+
+import static com.google.common.base.Preconditions.checkState;
+
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.schema.SchemaDefinition;
+import org.apache.pulsar.common.schema.SchemaInfo;
+import org.apache.pulsar.common.schema.SchemaType;
+
+import java.util.Optional;
+
+/**
+ * Schema from a native Apache Avro schema.
+ * This class is supposed to be used for working with existing data serialized in Avro, possibly stored in another system like Kafka.
+ * For this reason, it will not perform bytes validation against the schema in encoding and decoding, which are just identify functions.
+ * This class also makes it possible for users to bring in their own Avro serializer/deserializer.
+ */
+public class NativeAvroBytesSchema<T> implements Schema<byte[]> {
+
+ private Schema<T> schema;
+ private org.apache.avro.Schema nativeSchema;
+
+ public NativeAvroBytesSchema(org.apache.avro.Schema schema) {
+ setSchema(schema);
+ }
+
+ public NativeAvroBytesSchema(Object schema) {
+ this(validateSchema(schema));
+ }
+
+ public void setSchema(org.apache.avro.Schema schema) {
+ SchemaDefinition schemaDefinition = SchemaDefinition.builder().withJsonDef(schema.toString(false)).build();
+ this.nativeSchema = schema;
+ this.schema = AvroSchema.of(schemaDefinition);
+ }
+
+ public boolean schemaInitialized() {
+ return schema != null;
+ }
+
+ private static org.apache.avro.Schema validateSchema (Object schema) {
+ if (! (schema instanceof org.apache.avro.Schema))
+ throw new IllegalArgumentException("The input schema is not of type 'org.apache.avro.Schema'.");
+ return (org.apache.avro.Schema) schema;
+ }
+
+ private void ensureSchemaInitialized() {
+ checkState(schemaInitialized(), "Schema is not initialized before used");
+ }
+
+ @Override
+ public byte[] encode(byte[] message) {
+ ensureSchemaInitialized();
+
+ return message;
+ }
+
+ @Override
+ public byte[] decode(byte[] bytes, byte[] schemaVersion) {
+ ensureSchemaInitialized();
+
+ return bytes;
+ }
+
+ @Override
+ public SchemaInfo getSchemaInfo() {
+ ensureSchemaInitialized();
+
+ return schema.getSchemaInfo();
+ }
+
+ @Override
+ public Optional<Object> getNativeSchema() {
+ return Optional.ofNullable(this.nativeSchema);
+ }
+
+ @Override
+ public Schema<byte[]> clone() {
+ return new AutoProduceBytesSchema<>(schema.clone());
Review comment:
`return new NativeAvroBytesSchema(nativeSchema)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#issuecomment-876249537
> I don't understand why the build is failing with
>
> ```
> Error: Medium: Unread field: org.apache.pulsar.client.impl.schema.AutoProduceValidatedAvroBytesSchema.schema [org.apache.pulsar.client.impl.schema.AutoProduceValidatedAvroBytesSchema] At AutoProduceValidatedAvroBytesSchema.java:[line 43] URF_UNREAD_FIELD
> ```
>
> In `AutoProduceValidatedAvroBytesSchema`'s super class `AutoProduceBytesSchema`, the `schema` field is used in
>
> ```java
> @Override
> public SchemaInfo getSchemaInfo() {
> ensureSchemaInitialized();
>
> return schema.getSchemaInfo();
> }
> ```
AutoProduceBytesSchema already has a "schema" field, and you are adding a new field that hides the inherited one
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Zhen-hao commented on pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
Zhen-hao commented on pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#issuecomment-876181322
how often does the CI run? I find the feedback loop too long.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#issuecomment-875209918
@Zhen-hao thanks for your contribution. For this PR, do we need to update docs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#issuecomment-876190557
@Zhen-hao Would you like to add docs accordingly? Then you can ping me to review, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Zhen-hao commented on pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
Zhen-hao commented on pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#issuecomment-878051485
I don't understand this error in `unit-tests`
```
Error: Tests run: 28, Failures: 12, Errors: 0, Skipped: 13, Time elapsed: 608.077 s <<< FAILURE! - in org.apache.pulsar.functions.source.batch.BatchSourceExecutorTest
Error: testPushLifeCycle(org.apache.pulsar.functions.source.batch.BatchSourceExecutorTest) Time elapsed: 300.009 s <<< FAILURE!
org.testng.internal.thread.ThreadTimeoutException: Method org.apache.pulsar.functions.source.batch.BatchSourceExecutorTest.testPushLifeCycle() didn't finish within the time-out 300000
at org.testng.internal.MethodInvocationHelper.invokeWithTimeoutWithNewExecutor(MethodInvocationHelper.java:371)
at org.testng.internal.MethodInvocationHelper.invokeWithTimeout(MethodInvocationHelper.java:282)
at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:605)
at org.testng.internal.TestInvoker.retryFailed(TestInvoker.java:214)
at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:58)
at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:822)
at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:147)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
at org.testng.TestRunner.privateRun(TestRunner.java:764)
at org.testng.TestRunner.run(TestRunner.java:585)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:384)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:378)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:337)
at org.testng.SuiteRunner.run(SuiteRunner.java:286)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1218)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1140)
at org.testng.TestNG.runSuites(TestNG.java:1069)
at org.testng.TestNG.run(TestNG.java:1037)
at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:135)
at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeSingleClass(TestNGDirectoryTestSuite.java:112)
at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeLazy(TestNGDirectoryTestSuite.java:123)
at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:90)
at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:146)
at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Zhen-hao commented on a change in pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
Zhen-hao commented on a change in pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#discussion_r669419833
##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/NativeAvroBytesSchema.java
##########
@@ -0,0 +1,100 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.impl.schema;
+
+import static com.google.common.base.Preconditions.checkState;
+
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.schema.SchemaDefinition;
+import org.apache.pulsar.common.schema.SchemaInfo;
+import org.apache.pulsar.common.schema.SchemaType;
+
+import java.util.Optional;
+
+/**
+ * Schema from a native Apache Avro schema.
+ * This class is supposed to be used for working with existing data serialized in Avro, possibly stored in another system like Kafka.
+ * For this reason, it will not perform bytes validation against the schema in encoding and decoding, which are just identify functions.
+ * This class also makes it possible for users to bring in their own Avro serializer/deserializer.
+ */
+public class NativeAvroBytesSchema<T> implements Schema<byte[]> {
+
+ private Schema<T> schema;
+
+ public NativeAvroBytesSchema(org.apache.avro.Schema schema) {
+ SchemaDefinition schemaDefinition = SchemaDefinition.builder().withJsonDef(schema.toString(false)).build();
+ this.schema = AvroSchema.of(schemaDefinition);
+ }
+
+ public NativeAvroBytesSchema(Object schema) {
+ this(validateSchema(schema));
+ }
+
+ public void setSchema(Schema<T> schema) {
+ if (SchemaType.AVRO != schema.getSchemaInfo().getType()) {
+ throw new IllegalArgumentException("The type of input schema is not 'SchemaType.AVRO'.");
+ }
+ this.schema = schema;
+ }
+
+ public boolean schemaInitialized() {
+ return schema != null;
+ }
+
+ private static org.apache.avro.Schema validateSchema (Object schema) {
+ if (! (schema instanceof org.apache.avro.Schema)) throw new IllegalArgumentException("The input schema is not of type 'org.apache.avro.Schema'.");
Review comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#issuecomment-878125734
`Assert.expectThrows(SchemaSerializationException.class, () -> p.send(contentV1));`
expectThows will execute the code wrapped by a try/catch block, this is TestNG code:
```
public static <T extends Throwable> T expectThrows(
Class<T> throwableClass, ThrowingRunnable runnable) {
try {
runnable.run();
} catch (Throwable t) {
if (throwableClass.isInstance(t)) {
return throwableClass.cast(t);
} else {
String mismatchMessage =
String.format(
"Expected %s to be thrown, but %s was thrown",
throwableClass.getSimpleName(), t.getClass().getSimpleName());
throw new AssertionError(mismatchMessage, t);
}
}
String message =
String.format(
"Expected %s to be thrown, but nothing was thrown", throwableClass.getSimpleName());
throw new AssertionError(message);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Zhen-hao commented on pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
Zhen-hao commented on pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#issuecomment-879818983
> Did you try to use the debugger with a break point in that point ? probably you have not overridden every method in the Schema interface.
>
> by the way I believe that your new `NativeAvroBytesSchema` is useful only for the Producer.
> if you want to read raw bytes you simply do not set a Schema in the consumer or you use Schema.BYTES.
>
> `NativeAvroBytesSchema`does not perform validation and it does not decode the payload, so it is not worth to be used on the Consumer side.
>
> I suggest you to:
>
> * throw UnsupportedOperationException in all "decode" methods of NativeAvroBytesSchema
> * write explicitly in the Javadoc that this is a Schema to be used on the Producer side
> * use Schema.BYTES (or no Schema) in the Consumer in your test
You are right. I've made the changes as you suggested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Zhen-hao edited a comment on pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
Zhen-hao edited a comment on pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#issuecomment-879896646
> it is a good idea, btw we can separate code changes from documentation changes. you can follow up with a separate PR.
If other reviewers agree, I don't mind doing that;)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#issuecomment-876191639
> @Anonymitaet I see `site2/docs/io-develop.md`, `site2/website/versioned_docs/version-2.8.0/io-develop.md`, and `site2/website/versioned_docs/version-2.8.1/io-develop.md`.
> Are the latter two manually maintained?
>
> Should I only change `site2/docs/io-develop.md`?
They are updated manually. If the doc changes apply to several versions, we need to update them all.
But we can update one version first, after the contents get approved, then copy them to the rest versions.
Would you like to add docs accordingly? Then you can ping me to review, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Zhen-hao commented on a change in pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
Zhen-hao commented on a change in pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#discussion_r664740305
##########
File path: pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Schema.java
##########
@@ -426,6 +426,19 @@ default void configureSchemaInfo(String topic, String componentName,
return DefaultImplementation.newAutoProduceSchema(schema);
}
+ /**
+ * Create a schema instance that accepts a serialized Avro payload
+ * without validating it against the schema specified.
+ * It can be useful when migrating data from existing event or message stores.
+ *
+ * @return the auto schema instance
+ * @since 2.9.0
+ * @see #AUTO_PRODUCE_BYTES()
+ */
+ static Schema<byte[]> AUTO_PRODUCE_VALIDATED_AVRO_BYTES(Schema<?> schema) {
Review comment:
my idea was exactly to prevent other schema formats from abusing this feature.
the use cases I described already have some external system to guarantee the validity of event data against their schemas and the compatibility of the schemas.
I had thought about `AUTO_PRODUCE_BYTES(Schema<?> schema, boolean validate)` but decided that it would give this "dangerous option" to many formats that should never use it.
I think it is better to add this feature in a case-by-case fashion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Zhen-hao commented on pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
Zhen-hao commented on pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#issuecomment-876990881
> In my opinion we are on our way.
> We have to add tests.
> You can find examples like SimpleSchemaTest or SchemaTest....
>
> @sijie is current approach following your expectations?
>
> Tagging @codelipenghui @congbobo184 @gaoran10
I will find time over the weekend to add the tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Zhen-hao commented on a change in pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
Zhen-hao commented on a change in pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#discussion_r665244378
##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AutoProduceValidatedAvroBytesSchema.java
##########
@@ -0,0 +1,75 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.impl.schema;
+
+import static com.google.common.base.Preconditions.checkState;
+
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.schema.SchemaDefinition;
+import org.apache.pulsar.client.internal.DefaultImplementation;
+import org.apache.pulsar.common.schema.SchemaInfo;
+import org.apache.pulsar.common.schema.SchemaType;
+
+import java.util.Optional;
+
+/**
+ * Auto detect schema.
+ */
+public class AutoProduceValidatedAvroBytesSchema<T> extends AutoProduceBytesSchema<T> {
+
+ private Schema<T> schema;
+
+
+ public AutoProduceValidatedAvroBytesSchema(org.apache.avro.Schema schema) {
+ SchemaDefinition schemaDefinition = SchemaDefinition.builder().withJsonDef(schema.toString(false)).build();
+
+ setSchema(DefaultImplementation.newAvroSchema(schemaDefinition));
+ }
+
+ public AutoProduceValidatedAvroBytesSchema(Schema<T> schema) {
+ this.schema = null;
Review comment:
I don't know. the early build failed with a bug report about "unread field" or something, which I don't understand at all.
I thought `setSchema` could only be called on initialized values. In the constructor, it has to initialize all its fields before calling other methods.
This looks "stupid" to me as well. But I don't have other ideas fixing that "bug".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Zhen-hao commented on pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
Zhen-hao commented on pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#issuecomment-876259485
> OK. I think the easiest to do is copy the same code to the new class.
Or, only use `super.schema` in the new class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on pull request #11238: [Issue 11007] add a version of AUTO_PRODUCE_BYTES that doesn't validate the message in `encode`
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on pull request #11238:
URL: https://github.com/apache/pulsar/pull/11238#issuecomment-901521790
@Zhen-hao thanks for your feedback. Please do not forget to add docs to let users know the changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org