You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "liangyepianzhou (via GitHub)" <gi...@apache.org> on 2023/07/27 06:26:23 UTC

[GitHub] [pulsar-site] liangyepianzhou opened a new pull request, #656: [optimize][schema] Add schema compatibility between Go and Java client

liangyepianzhou opened a new pull request, #656:
URL: https://github.com/apache/pulsar-site/pull/656

   <!--
   
   ### Contribution Checklist
     
     - PR title format should be *[type][component] summary*. For details, see *[Guideline - Pulsar PR Naming Convention](https://pulsar.apache.org/contribute/develop-semantic-title/)*. 
   
   -->
   
   <!-- Either this PR adds a doc for a code PR, -->
   
   This PR adds doc for #xyz
   
   <!-- or fixes a doc issue -->
   
   This PR fixes #xyz 
   
   <!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
   
   - [x] `doc` <!-- Your PR contains doc changes. Please attach the local preview screenshots (run `./preview.sh` at root path) to your PR description, or else your PR might not get merged. -->
   - [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
   - [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
   - [ ] `doc-complete` <!-- Docs have been already added -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar-site] shibd commented on a diff in pull request #656: [optimize][schema] Add schema compatibility between Go and Java client

Posted by "shibd (via GitHub)" <gi...@apache.org>.
shibd commented on code in PR #656:
URL: https://github.com/apache/pulsar-site/pull/656#discussion_r1278763869


##########
docs/client-libraries-schema.md:
##########
@@ -142,4 +142,96 @@ while True:
     except Exception:
         # Message failed to be processed
         consumer.negative_acknowledge(msg)
-```
\ No newline at end of file
+```
+
+## Work with the Go schema
+Working with Go schema has slight differences from Java schema.
+This part will introduce the schema compatibility between Go client and Java client.
+
+#### Avro Schema
+Avro schema in Go and Java are compatible, but there are some differences in how the schemas are defined. Go typically uses schema definitions, a string JSON, to create schemas. However, Java often uses class types for schema creation. As a result, Java allows non-primitive fields to be nullable by default, while in Go's schema definition, the nullability of fields needs to be explicitly stated.
+GO:
+```go
+// Compatible with defining a schema in Java
+exampleSchemaDefCompatible := NewAvroSchema(`{"fields":
+    [
+        {"name":"id","type":"int"},{"default":null,"name":"name","type":["null","string"]}
+    ],
+    "name":"MyAvro","namespace":"schemaNotFoundTestCase","type":"record"}`, nil)
+// Not compatible with defining a schema in Java
+exampleSchemaDefIncompatible := NewAvroSchema(`{"fields":
+    [
+        {"name":"id","type":"int"},{"default":null,"name":"name","type":["string"]}
+    ],
+    "name":"MyAvro","namespace":"schemaNotFoundTestCase","type":"record"}`, nil)
+Producer := NewAvroSchema(exampleSchemaDef, nil)
+
+```
+JAVA:
+```java
+@AllArgsConstructor
+@NoArgsConstructor
+public static class Example {
+    public String name;
+    public int id;
+}
+
+Producer<Example> producer = pulsarClient.newProducer(Schema.AVRO(Example.class))
+                .topic(topic).create();
+```
+
+#### JSON Schema
+The situation with JSON schema is similar to Avro Schema.
+```go
+// Compatible with defining a schema in Java
+exampleSchemaDefCompatible := "{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\"," +
+	"\"fields\":[{\"name\":\"ID\",\"type\":\"int\"},{\"name\":\"Name\",\"type\":[\"null\", \"string\"]}]}"
+
+consumerJSCompatible := NewJSONSchema(exampleSchemaDefCompatible, nil)
+// Not compatible with defining a schema in Java

Review Comment:
   Which define code of Java is this incompatible with?



##########
docs/client-libraries-schema.md:
##########
@@ -142,4 +142,96 @@ while True:
     except Exception:
         # Message failed to be processed
         consumer.negative_acknowledge(msg)
-```
\ No newline at end of file
+```
+
+## Work with the Go schema
+Working with Go schema has slight differences from Java schema.
+This part will introduce the schema compatibility between Go client and Java client.
+
+#### Avro Schema
+Avro schema in Go and Java are compatible, but there are some differences in how the schemas are defined. Go typically uses schema definitions, a string JSON, to create schemas. However, Java often uses class types for schema creation. As a result, Java allows non-primitive fields to be nullable by default, while in Go's schema definition, the nullability of fields needs to be explicitly stated.
+GO:
+```go
+// Compatible with defining a schema in Java
+exampleSchemaDefCompatible := NewAvroSchema(`{"fields":
+    [
+        {"name":"id","type":"int"},{"default":null,"name":"name","type":["null","string"]}
+    ],
+    "name":"MyAvro","namespace":"schemaNotFoundTestCase","type":"record"}`, nil)
+// Not compatible with defining a schema in Java

Review Comment:
   Does this refer to the following Java example?
   
   If the Go Client is used like this, the Java client should also support it, right? 
   
   In java client, there are two ways:
   1. Use SchemaDefinition to set schema JsonDef
   
   ``` 
             String jsonDef = "{"fields":[{"name":"id","type":"int"},{"default":null,"name":"name","type":["null","string"]}],"name":"MyAvro","namespace":"schemaNotFoundTestCase","type":"record"}"
              Schema schema = Schema.AVRO(SchemaDefinition.builder().withJsonDef("").build());
   ```
   
   2. Or, to set not allow field is null.
   ```
         SchemaDefinition<Example> schemaDefinition =
                        SchemaDefinition.builder().withPojo(Example.class).withAlwaysAllowNull(false).build();
         Schema schema = Schema.AVRO(schemaDefinition);
         
       Producer<Example> producer = pulsarClient.newProducer(schema)
                   .topic(topic).create();
   ```
   
   we should show the solution.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar-site] Anonymitaet commented on pull request #656: [optimize][schema] Add schema compatibility between Go and Java client

Posted by "Anonymitaet (via GitHub)" <gi...@apache.org>.
Anonymitaet commented on PR #656:
URL: https://github.com/apache/pulsar-site/pull/656#issuecomment-1657397999

   @BewareMyPower @shibd 
   Could you please help review this PR? Thank you!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar-site] liangyepianzhou commented on a diff in pull request #656: [optimize][schema] Add schema compatibility between Go and Java client

Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou commented on code in PR #656:
URL: https://github.com/apache/pulsar-site/pull/656#discussion_r1286096873


##########
docs/client-libraries-schema.md:
##########
@@ -142,4 +142,96 @@ while True:
     except Exception:
         # Message failed to be processed
         consumer.negative_acknowledge(msg)
-```
\ No newline at end of file
+```
+
+## Work with the Go schema
+Working with Go schema has slight differences from Java schema.
+This part will introduce the schema compatibility between Go client and Java client.
+
+#### Avro Schema
+Avro schema in Go and Java are compatible, but there are some differences in how the schemas are defined. Go typically uses schema definitions, a string JSON, to create schemas. However, Java often uses class types for schema creation. As a result, Java allows non-primitive fields to be nullable by default, while in Go's schema definition, the nullability of fields needs to be explicitly stated.
+GO:
+```go
+// Compatible with defining a schema in Java
+exampleSchemaDefCompatible := NewAvroSchema(`{"fields":
+    [
+        {"name":"id","type":"int"},{"default":null,"name":"name","type":["null","string"]}
+    ],
+    "name":"MyAvro","namespace":"schemaNotFoundTestCase","type":"record"}`, nil)
+// Not compatible with defining a schema in Java

Review Comment:
   Oh, you are right.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar-site] liangyepianzhou commented on a diff in pull request #656: [optimize][schema] Add schema compatibility between Go and Java client

Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou commented on code in PR #656:
URL: https://github.com/apache/pulsar-site/pull/656#discussion_r1286098695


##########
docs/client-libraries-schema.md:
##########
@@ -142,4 +142,96 @@ while True:
     except Exception:
         # Message failed to be processed
         consumer.negative_acknowledge(msg)
-```
\ No newline at end of file
+```
+
+## Work with the Go schema
+Working with Go schema has slight differences from Java schema.
+This part will introduce the schema compatibility between Go client and Java client.
+
+#### Avro Schema
+Avro schema in Go and Java are compatible, but there are some differences in how the schemas are defined. Go typically uses schema definitions, a string JSON, to create schemas. However, Java often uses class types for schema creation. As a result, Java allows non-primitive fields to be nullable by default, while in Go's schema definition, the nullability of fields needs to be explicitly stated.
+GO:
+```go
+// Compatible with defining a schema in Java
+exampleSchemaDefCompatible := NewAvroSchema(`{"fields":
+    [
+        {"name":"id","type":"int"},{"default":null,"name":"name","type":["null","string"]}
+    ],
+    "name":"MyAvro","namespace":"schemaNotFoundTestCase","type":"record"}`, nil)
+// Not compatible with defining a schema in Java
+exampleSchemaDefIncompatible := NewAvroSchema(`{"fields":
+    [
+        {"name":"id","type":"int"},{"default":null,"name":"name","type":["string"]}
+    ],
+    "name":"MyAvro","namespace":"schemaNotFoundTestCase","type":"record"}`, nil)
+Producer := NewAvroSchema(exampleSchemaDef, nil)
+
+```
+JAVA:
+```java
+@AllArgsConstructor
+@NoArgsConstructor
+public static class Example {
+    public String name;
+    public int id;
+}
+
+Producer<Example> producer = pulsarClient.newProducer(Schema.AVRO(Example.class))
+                .topic(topic).create();
+```
+
+#### JSON Schema
+The situation with JSON schema is similar to Avro Schema.
+```go
+// Compatible with defining a schema in Java
+exampleSchemaDefCompatible := "{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\"," +
+	"\"fields\":[{\"name\":\"ID\",\"type\":\"int\"},{\"name\":\"Name\",\"type\":[\"null\", \"string\"]}]}"
+
+consumerJSCompatible := NewJSONSchema(exampleSchemaDefCompatible, nil)
+// Not compatible with defining a schema in Java

Review Comment:
   Like Avro, is a problem with null fields.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar-site] liangyepianzhou closed pull request #656: [improve][docs] Add schema compatibility between Go and Java client

Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou closed pull request #656: [improve][docs] Add schema compatibility between Go and Java client
URL: https://github.com/apache/pulsar-site/pull/656


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org