You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/11/11 09:35:36 UTC
[GitHub] [pulsar] momo-jun opened a new pull request, #18434: [refactor][doc] Improve schema docs
momo-jun opened a new pull request, #18434:
URL: https://github.com/apache/pulsar/pull/18434
### Motivation
<!-- Explain here the context, and why you're making that change. What is the problem you're trying to solve. -->
This is a follow-up PR to https://github.com/apache/pulsar/pull/18242 to improve the Schema docs, aiming to provide Pulsar users with a more accurate, easy-to-read, and comprehensive information experience.
### Modifications
<!-- Describe the modifications you've done. -->
The content is revised and reviewed through [Google docs](https://docs.google.com/document/d/1YYansXopyVV66NaQ6TR9tfpq9MEhcbvCqv8ByFWuadA/edit?usp=sharing).
1. Improve the description and illustrations of `How Pulsar schema works`.
2. Add an illustration to introduce Pulsar schema.
3. Incorporate `schema evolution` and subtopics into `Understand schema`.
4. Relocate `native Avro` to `struct` schema.
5. Add incremental changes to align with the code.
6. Add/correct descriptions, commands, and code examples.
7. Remove duplicate or redundant information.
Many thanks to @congbobo184 and @RobertIndie for providing technical inputs and review comments.
**The review is still in Process. Feel free to leave any comments.**
### Documentation
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [x] `doc` <!-- Your PR contains doc changes. Please attach the local preview screenshots (run `sh start.sh` at `pulsar/site2/website`) to your PR description, or else your PR might not get merged. -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] labuladong commented on a diff in pull request #18434: [refactor][doc] Improve schema docs
Posted by GitBox <gi...@apache.org>.
labuladong commented on code in PR #18434:
URL: https://github.com/apache/pulsar/pull/18434#discussion_r1034316020
##########
site2/docs/schema-get-started.md:
##########
@@ -11,80 +11,283 @@ import TabItem from '@theme/TabItem';
````
-This hands-on tutorial provides instructions and examples on how to construct and customize schemas.
+This hands-on tutorial provides instructions and examples on how to construct schemas. For the instructions on administrative tasks, see [Manage schema](admin-api-schemas.md).
-## Construct a string schema
+## Construct a schema
-This example demonstrates how to construct a [string schema](schema-understand.md#primitive-type) and use it to produce and consume messages in Java.
+- [Construct a bytes schema](#bytes)
+- [Construct a string schema](#string)
+- [Construct a key/value schema](#keyvalue)
+- [Construct an Avro schema](#avro)
+- [Construct a JSON schema](#json)
+- [Construct a Protobuf schema](#protobuf)
+- [Construct a ProtobufNative schema](#protobufnative)
+- [Construct a native Avro schema](#nativeavro)
+- [Construct an AUTO_PRODUCE schema](#auto_produce)
+- [Construct an AUTO_CONSUME schema](#auto_consume)
-1. Create a producer with a string schema and send messages.
+### `bytes`
+
+This example demonstrates how to construct a [bytes schema](schema-understand.md#primitive-type) using language-specific clients and use it to produce and consume messages.
+
+````mdx-code-block
+<Tabs groupId="api-choice"
+ defaultValue="Java"
+ values={[{"label":"Java","value":"Java"},{"label":"C++","value":"C++"},{"label":"Python","value":"Python"},{"label":"Go","value":"Go"}]}>
+
+<TabItem value="Java">
+
+```java
+ .topic("my-topic")
+ .create();
+Consumer<byte[]> consumer = pulsarClient.newConsumer(Schema.BYTES)
Review Comment:
```java
Producer<byte[]> producer = client.newProducer(Schema.BYTES)
```
I think adding this part would be ok.
##########
site2/docs/schema-get-started.md:
##########
@@ -11,80 +11,283 @@ import TabItem from '@theme/TabItem';
````
-This hands-on tutorial provides instructions and examples on how to construct and customize schemas.
+This hands-on tutorial provides instructions and examples on how to construct schemas. For the instructions on administrative tasks, see [Manage schema](admin-api-schemas.md).
-## Construct a string schema
+## Construct a schema
-This example demonstrates how to construct a [string schema](schema-understand.md#primitive-type) and use it to produce and consume messages in Java.
+- [Construct a bytes schema](#bytes)
+- [Construct a string schema](#string)
+- [Construct a key/value schema](#keyvalue)
+- [Construct an Avro schema](#avro)
+- [Construct a JSON schema](#json)
+- [Construct a Protobuf schema](#protobuf)
+- [Construct a ProtobufNative schema](#protobufnative)
+- [Construct a native Avro schema](#nativeavro)
+- [Construct an AUTO_PRODUCE schema](#auto_produce)
+- [Construct an AUTO_CONSUME schema](#auto_consume)
-1. Create a producer with a string schema and send messages.
+### `bytes`
+
+This example demonstrates how to construct a [bytes schema](schema-understand.md#primitive-type) using language-specific clients and use it to produce and consume messages.
+
+````mdx-code-block
+<Tabs groupId="api-choice"
+ defaultValue="Java"
+ values={[{"label":"Java","value":"Java"},{"label":"C++","value":"C++"},{"label":"Python","value":"Python"},{"label":"Go","value":"Go"}]}>
+
+<TabItem value="Java">
+
+```java
+ .topic("my-topic")
+ .create();
+Consumer<byte[]> consumer = pulsarClient.newConsumer(Schema.BYTES)
Review Comment:
```java
Producer<byte[]> producer = client.newProducer(Schema.BYTES)
```
I think adding this part would be ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] momo-jun commented on a diff in pull request #18434: [refactor][doc] Improve schema docs
Posted by GitBox <gi...@apache.org>.
momo-jun commented on code in PR #18434:
URL: https://github.com/apache/pulsar/pull/18434#discussion_r1042920951
##########
site2/docs/schema-understand.md:
##########
@@ -48,92 +46,227 @@ Pulsar supports various schema types, which are mainly divided into two categori
The following table outlines the primitive types that Pulsar schema supports, and the conversions between **schema types** and **language-specific primitive types**.
-| Primitive Type | Description | Java Type| Python Type | Go Type |
-|---|---|---|---|---|
-| `BOOLEAN` | A binary value | boolean | bool | bool |
-| `INT8` | A 8-bit signed integer | int | | int8 |
-| `INT16` | A 16-bit signed integer | int | | int16 |
-| `INT32` | A 32-bit signed integer | int | | int32 |
-| `INT64` | A 64-bit signed integer | int | | int64 |
-| `FLOAT` | A single precision (32-bit) IEEE 754 floating-point number | float | float | float32 |
-| `DOUBLE` | A double-precision (64-bit) IEEE 754 floating-point number | double | float | float64|
-| `BYTES` | A sequence of 8-bit unsigned bytes | byte[], ByteBuffer, ByteBuf | bytes | []byte |
-| `STRING` | A Unicode character sequence | string | str | string|
-| `TIMESTAMP` (`DATE`, `TIME`) | A logic type represents a specific instant in time with millisecond precision. <br />It stores the number of milliseconds since `January 1, 1970, 00:00:00 GMT` as an `INT64` value | java.sql.Timestamp (java.sql.Time, java.util.Date) | | |
-| INSTANT | A single instantaneous point on the time-line with nanoseconds precision| java.time.Instant | | |
-| LOCAL_DATE | An immutable date-time object that represents a date, often viewed as year-month-day| java.time.LocalDate | | |
-| LOCAL_TIME | An immutable date-time object that represents a time, often viewed as hour-minute-second. Time is represented to nanosecond precision.| java.time.LocalDateTime | |
-| LOCAL_DATE_TIME | An immutable date-time object that represents a date-time, often viewed as year-month-day-hour-minute-second | java.time.LocalTime | |
+| Primitive Type | Description | Java Type| Python Type | Go Type | C++ Type | C# Type|
+|---|---|---|---|---|---|---|
+| `BOOLEAN` | A binary value. | boolean | bool | bool | bool | bool |
+| `INT8` | A 8-bit signed integer. | int | int | int8 | int8_t | byte |
+| `INT16` | A 16-bit signed integer. | int | int | int16 | int16_t | short |
+| `INT32` | A 32-bit signed integer. | int | int | int32 | int32_t | int |
+| `INT64` | A 64-bit signed integer. | int | int | int64 | int64_t | long |
+| `FLOAT` | A single precision (32-bit) IEEE 754 floating-point number. | float | float | float32 | float | float |
+| `DOUBLE` | A double-precision (64-bit) IEEE 754 floating-point number. | double | double | float64| double | double |
+| `BYTES` | A sequence of 8-bit unsigned bytes. | byte[], ByteBuffer, ByteBuf | bytes | []byte | void * | byte[], `ReadOnlySequence<byte>` |
Review Comment:
I used it as an escape from syntax errors. Without it, the compilation fails. The error message reads "Expected corresponding JSX closing tag for <byte>". Did you know any other ways to resolve it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] momo-jun commented on a diff in pull request #18434: [refactor][doc] Improve schema docs
Posted by GitBox <gi...@apache.org>.
momo-jun commented on code in PR #18434:
URL: https://github.com/apache/pulsar/pull/18434#discussion_r1045725158
##########
site2/docs/schema-get-started.md:
##########
@@ -228,88 +329,350 @@ Producer<SensorReading> producer = client.newProducer(AvroSchema.of(SensorReadin
.create();
```
-### Avro-based schema using Java
-
-The following schema formats are currently available for Java:
+</TabItem>
+<TabItem value="C++">
-* No schema or the byte array schema (which can be applied using `Schema.BYTES`):
+ ```cpp
+ // Send messages
+ static const std::string exampleSchema =
+ "{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\","
+ "\"fields\":[{\"name\":\"a\",\"type\":\"int\"},{\"name\":\"b\",\"type\":\"int\"}]}";
+ Producer producer;
+ ProducerConfiguration producerConf;
+ producerConf.setSchema(SchemaInfo(AVRO, "Avro", exampleSchema));
+ client.createProducer("topic-avro", producerConf, producer);
- ```java
- Producer<byte[]> bytesProducer = client.newProducer(Schema.BYTES)
- .topic("some-raw-bytes-topic")
- .create();
+ // Receive messages
+ static const std::string exampleSchema =
+ "{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\","
+ "\"fields\":[{\"name\":\"a\",\"type\":\"int\"},{\"name\":\"b\",\"type\":\"int\"}]}";
+ ConsumerConfiguration consumerConf;
+ Consumer consumer;
+ consumerConf.setSchema(SchemaInfo(AVRO, "Avro", exampleSchema));
+ client.subscribe("topic-avro", "sub-2", consumerConf, consumer)
```
- Or, equivalently:
+</TabItem>
+<TabItem value="Python">
- ```java
- Producer<byte[]> bytesProducer = client.newProducer()
- .topic("some-raw-bytes-topic")
- .create();
- ```
+You can declare an `AvroSchema` using Python through one of the following methods.
-* `String` for normal UTF-8-encoded string data. Apply the schema using `Schema.STRING`:
+**Method 1: Record**
- ```java
- Producer<String> stringProducer = client.newProducer(Schema.STRING)
- .topic("some-string-topic")
- .create();
- ```
+Declare an `AvroSchema` by passing a class that inherits from `pulsar.schema.Record` and defines the fields as class variables.
-* Create JSON schemas for POJOs using `Schema.JSON`. The following is an example.
+```python
+class Example(Record):
+ a = Integer()
+ b = Integer()
- ```java
- Producer<MyPojo> pojoProducer = client.newProducer(Schema.JSON(MyPojo.class))
- .topic("some-pojo-topic")
- .create();
- ```
+producer = client.create_producer(
+ 'avro-schema-topic',
+ schema=AvroSchema(Example))
+r = Example(a=1, b=2)
+producer.send(r)
-* Generate Protobuf schemas using `Schema.PROTOBUF`. The following example shows how to create the Protobuf schema and use it to instantiate a new producer:
+consumer = client.subscribe(
+ 'avro-schema-topic',
+ 'sub',
+ schema=AvroSchema(Example))
+msg = consumer.receive()
+e = msg.value()
+```
- ```java
- Producer<MyProtobuf> protobufProducer = client.newProducer(Schema.PROTOBUF(MyProtobuf.class))
- .topic("some-protobuf-topic")
- .create();
- ```
+**Method 2: JSON definition**
+
+1. Declare an `AvroSchema` using JSON. In this case, Avro schemas are defined using JSON.
+
+ Below is an example of `AvroSchema` defined using a JSON file (`company.avsc`).
+
+ ```json
+ {
+ "doc": "this is doc",
+ "namespace": "example.avro",
+ "type": "record",
+ "name": "Company",
+ "fields": [
+ {"name": "name", "type": ["null", "string"]},
+ {"name": "address", "type": ["null", "string"]},
+ {"name": "employees", "type": ["null", {"type": "array", "items": {
+ "type": "record",
+ "name": "Employee",
+ "fields": [
+ {"name": "name", "type": ["null", "string"]},
+ {"name": "age", "type": ["null", "int"]}
+ ]
+ }}]},
+ {"name": "labels", "type": ["null", {"type": "map", "values": "string"}]}
+ ]
+ }
+ ```
-* Define Avro schemas with `Schema.AVRO`. The following code snippet demonstrates how to create and use Avro schema.
+2. Load a schema definition from a file by using [`avro.schema`](https://avro.apache.org/docs/current/getting-started-python/) or [`fastavro.schema`](https://fastavro.readthedocs.io/en/latest/schema.html#fastavro._schema_py.load_schema).
+
+ If you use the [JSON definition](#method-2-json-definition) method to declare an `AvroSchema`, you need to:
+ - Use [Python dict](https://developers.google.com/edu/python/dict-files) to produce and consume messages, which is different from using the [Record](#method-1-record) method.
+ - Set the value of the `_record_cls` parameter to `None` when generating an `AvroSchema` object.
+
+ **Example**
+
+ ```python
+ from fastavro.schema import load_schema
+ from pulsar.schema import *
+ schema_definition = load_schema("examples/company.avsc")
+ avro_schema = AvroSchema(None, schema_definition=schema_definition)
+ producer = client.create_producer(
+ topic=topic,
+ schema=avro_schema)
+ consumer = client.subscribe(topic, 'test', schema=avro_schema)
+ company = {
+ "name": "company-name" + str(i),
+ "address": 'xxx road xxx street ' + str(i),
+ "employees": [
+ {"name": "user" + str(i), "age": 20 + i},
+ {"name": "user" + str(i), "age": 30 + i},
+ {"name": "user" + str(i), "age": 35 + i},
+ ],
+ "labels": {
+ "industry": "software" + str(i),
+ "scale": ">100",
+ "funds": "1000000.0"
+ }
+ }
+ producer.send(company)
+ msg = consumer.receive()
+ # Users could get a dict object by `value()` method.
+ msg.value()
+ ```
- ```java
- Producer<MyAvro> avroProducer = client.newProducer(Schema.AVRO(MyAvro.class))
- .topic("some-avro-topic")
- .create();
- ```
+</TabItem>
+<TabItem value="Go">
+Suppose you have an `avroExampleStruct` class as follows, and you'd like to transmit it over a Pulsar topic.
-### Avro schema using C++
+```go
+ type avroExampleStruct struct {
+ ID int
+ Name string
+}
+```
-- The following example shows how to create a producer with an Avro schema.
+1. Add an `avroSchemaDef` like this:
- ```cpp
- static const std::string exampleSchema =
- "{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\","
- "\"fields\":[{\"name\":\"a\",\"type\":\"int\"},{\"name\":\"b\",\"type\":\"int\"}]}";
- Producer producer;
- ProducerConfiguration producerConf;
- producerConf.setSchema(SchemaInfo(AVRO, "Avro", exampleSchema));
- client.createProducer("topic-avro", producerConf, producer);
- ```
+ ```go
+ var (
+ exampleSchemaDef = "{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\"," +
+ "\"fields\":[{\"name\":\"ID\",\"type\":\"int\"},{\"name\":\"Name\",\"type\":\"string\"}]}"
+ )
+ ```
-- The following example shows how to create a consumer with an Avro schema.
+2. Create producer and consumer to send/receive messages:
+
+ ```go
+ //Create producer and send message
+ producer, err := client.CreateProducer(pulsar.ProducerOptions{
+ Topic: "my-topic",
+ Schema: pulsar.NewAvroSchema(exampleSchemaDef, nil),
+ })
+
+ msgId, err := producer.Send(context.Background(), &pulsar.ProducerMessage{
+ Value: avroExampleStruct{
+ ID: 10,
+ Name: "avroExampleStruct",
+ },
+ })
+
+ //Create Consumer and receive message
+ consumer, err := client.Subscribe(pulsar.ConsumerOptions{
+ Topic: "my-topic",
+ Schema: pulsar.NewAvroSchema(exampleSchemaDef, nil),
+ SubscriptionName: "my-sub",
+ Type: pulsar.Shared,
+ })
+ message, err := consumer.Receive(context.Background())
+ ```
- ```cpp
- static const std::string exampleSchema =
- "{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\","
- "\"fields\":[{\"name\":\"a\",\"type\":\"int\"},{\"name\":\"b\",\"type\":\"int\"}]}";
- ConsumerConfiguration consumerConf;
- Consumer consumer;
- consumerConf.setSchema(SchemaInfo(AVRO, "Avro", exampleSchema));
- client.subscribe("topic-avro", "sub-2", consumerConf, consumer)
- ```
+</TabItem>
+</Tabs>
+````
+
+### JSON
+
+````mdx-code-block
+<Tabs groupId="api-choice"
+ defaultValue="Java"
+ values={[{"label":"Java","value":"Java"},{"label":"C++","value":"C++"},{"label":"Python","value":"Python"},{"label":"Go","value":"Go"}]}>
+
+<TabItem value="Java">
+
+Similar to using `AvroSchema`, you can declare a `JsonSchema` by passing a class. The only difference is to use `JsonSchema` instead of `AvroSchema` when defining the schema type, as shown below. For how to use `AvroSchema` via record, see [Method 1 - Record](#method-1-record).
+
+```java
+static class SchemaDema {
+ public String name;
+ public int age;
+}
+
+Producer<SchemaDema> producer = pulsarClient.newProducer(Schema.JSON(SchemaDema.class))
+ .topic("my-topic")
+ .create();
+Consumer<SchemaDema> consumer = pulsarClient.newConsumer(Schema.JSON(SchemaDema.class))
+ .topic("my-topic")
+ .subscriptionName("my-sub")
+ .subscribe();
+
+SchemaDema schemaDema = new SchemaDema();
+schemaDema.name = "puslar";
+schemaDema.age = 20;
+producer.newMessage().value(schemaDema).send();
+
+Message<SchemaDema> message = consumer.receive(5, TimeUnit.SECONDS);
Review Comment:
Good question. @liangyepianzhou can you pls take a look? I'm not sure whether it is intentional.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] RobertIndie commented on a diff in pull request #18434: [refactor][doc] Improve schema docs
Posted by GitBox <gi...@apache.org>.
RobertIndie commented on code in PR #18434:
URL: https://github.com/apache/pulsar/pull/18434#discussion_r1045534126
##########
site2/docs/schema-get-started.md:
##########
@@ -11,190 +11,291 @@ import TabItem from '@theme/TabItem';
````
-This hands-on tutorial provides instructions and examples on how to construct and customize schemas.
+This hands-on tutorial provides instructions and examples on how to construct schemas. For instructions on administrative tasks, see [Manage schema](admin-api-schemas.md).
-## Construct a string schema
+## Construct a schema
-This example demonstrates how to construct a [string schema](schema-understand.md#primitive-type) and use it to produce and consume messages in Java.
+- [Construct a bytes schema](#bytes)
+- [Construct a string schema](#string)
+- [Construct a key/value schema](#keyvalue)
+- [Construct an Avro schema](#avro)
+- [Construct a JSON schema](#json)
+- [Construct a Protobuf schema](#protobuf)
+- [Construct a ProtobufNative schema](#protobufnative)
+- [Construct a native Avro schema](#nativeavro)
+- [Construct an AUTO_PRODUCE schema](#auto_produce)
+- [Construct an AUTO_CONSUME schema](#auto_consume)
-1. Create a producer with a string schema and send messages.
+### `bytes`
Review Comment:
```suggestion
### bytes
```
Seems strange:
<img width="250" alt="image" src="https://user-images.githubusercontent.com/16974619/206999532-e4b74a2e-3a43-4db3-b4d8-925c35475264.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] RobertIndie commented on a diff in pull request #18434: [refactor][doc] Improve schema docs
Posted by GitBox <gi...@apache.org>.
RobertIndie commented on code in PR #18434:
URL: https://github.com/apache/pulsar/pull/18434#discussion_r1041963617
##########
site2/docs/schema-understand.md:
##########
@@ -48,92 +46,227 @@ Pulsar supports various schema types, which are mainly divided into two categori
The following table outlines the primitive types that Pulsar schema supports, and the conversions between **schema types** and **language-specific primitive types**.
-| Primitive Type | Description | Java Type| Python Type | Go Type |
-|---|---|---|---|---|
-| `BOOLEAN` | A binary value | boolean | bool | bool |
-| `INT8` | A 8-bit signed integer | int | | int8 |
-| `INT16` | A 16-bit signed integer | int | | int16 |
-| `INT32` | A 32-bit signed integer | int | | int32 |
-| `INT64` | A 64-bit signed integer | int | | int64 |
-| `FLOAT` | A single precision (32-bit) IEEE 754 floating-point number | float | float | float32 |
-| `DOUBLE` | A double-precision (64-bit) IEEE 754 floating-point number | double | float | float64|
-| `BYTES` | A sequence of 8-bit unsigned bytes | byte[], ByteBuffer, ByteBuf | bytes | []byte |
-| `STRING` | A Unicode character sequence | string | str | string|
-| `TIMESTAMP` (`DATE`, `TIME`) | A logic type represents a specific instant in time with millisecond precision. <br />It stores the number of milliseconds since `January 1, 1970, 00:00:00 GMT` as an `INT64` value | java.sql.Timestamp (java.sql.Time, java.util.Date) | | |
-| INSTANT | A single instantaneous point on the time-line with nanoseconds precision| java.time.Instant | | |
-| LOCAL_DATE | An immutable date-time object that represents a date, often viewed as year-month-day| java.time.LocalDate | | |
-| LOCAL_TIME | An immutable date-time object that represents a time, often viewed as hour-minute-second. Time is represented to nanosecond precision.| java.time.LocalDateTime | |
-| LOCAL_DATE_TIME | An immutable date-time object that represents a date-time, often viewed as year-month-day-hour-minute-second | java.time.LocalTime | |
+| Primitive Type | Description | Java Type| Python Type | Go Type | C++ Type | C# Type|
+|---|---|---|---|---|---|---|
+| `BOOLEAN` | A binary value. | boolean | bool | bool | bool | bool |
+| `INT8` | A 8-bit signed integer. | int | int | int8 | int8_t | byte |
+| `INT16` | A 16-bit signed integer. | int | int | int16 | int16_t | short |
+| `INT32` | A 32-bit signed integer. | int | int | int32 | int32_t | int |
+| `INT64` | A 64-bit signed integer. | int | int | int64 | int64_t | long |
+| `FLOAT` | A single precision (32-bit) IEEE 754 floating-point number. | float | float | float32 | float | float |
+| `DOUBLE` | A double-precision (64-bit) IEEE 754 floating-point number. | double | double | float64| double | double |
+| `BYTES` | A sequence of 8-bit unsigned bytes. | byte[], ByteBuffer, ByteBuf | bytes | []byte | void * | byte[], `ReadOnlySequence<byte>` |
Review Comment:
```suggestion
| `BYTES` | A sequence of 8-bit unsigned bytes. | byte[], ByteBuffer, ByteBuf | bytes | []byte | void * | byte[], ReadOnlySequence<byte> |
```
Looks wired in the preview:
<img width="462" alt="image" src="https://user-images.githubusercontent.com/16974619/206140861-5599dec8-87c1-4dc2-86bd-acd61f469217.png">
##########
site2/docs/schema-understand.md:
##########
@@ -48,92 +46,227 @@ Pulsar supports various schema types, which are mainly divided into two categori
The following table outlines the primitive types that Pulsar schema supports, and the conversions between **schema types** and **language-specific primitive types**.
-| Primitive Type | Description | Java Type| Python Type | Go Type |
-|---|---|---|---|---|
-| `BOOLEAN` | A binary value | boolean | bool | bool |
-| `INT8` | A 8-bit signed integer | int | | int8 |
-| `INT16` | A 16-bit signed integer | int | | int16 |
-| `INT32` | A 32-bit signed integer | int | | int32 |
-| `INT64` | A 64-bit signed integer | int | | int64 |
-| `FLOAT` | A single precision (32-bit) IEEE 754 floating-point number | float | float | float32 |
-| `DOUBLE` | A double-precision (64-bit) IEEE 754 floating-point number | double | float | float64|
-| `BYTES` | A sequence of 8-bit unsigned bytes | byte[], ByteBuffer, ByteBuf | bytes | []byte |
-| `STRING` | A Unicode character sequence | string | str | string|
-| `TIMESTAMP` (`DATE`, `TIME`) | A logic type represents a specific instant in time with millisecond precision. <br />It stores the number of milliseconds since `January 1, 1970, 00:00:00 GMT` as an `INT64` value | java.sql.Timestamp (java.sql.Time, java.util.Date) | | |
-| INSTANT | A single instantaneous point on the time-line with nanoseconds precision| java.time.Instant | | |
-| LOCAL_DATE | An immutable date-time object that represents a date, often viewed as year-month-day| java.time.LocalDate | | |
-| LOCAL_TIME | An immutable date-time object that represents a time, often viewed as hour-minute-second. Time is represented to nanosecond precision.| java.time.LocalDateTime | |
-| LOCAL_DATE_TIME | An immutable date-time object that represents a date-time, often viewed as year-month-day-hour-minute-second | java.time.LocalTime | |
+| Primitive Type | Description | Java Type| Python Type | Go Type | C++ Type | C# Type|
+|---|---|---|---|---|---|---|
Review Comment:
Do we also need to explain what `N/A` means here?
##########
site2/docs/schema-understand.md:
##########
@@ -48,92 +46,227 @@ Pulsar supports various schema types, which are mainly divided into two categori
The following table outlines the primitive types that Pulsar schema supports, and the conversions between **schema types** and **language-specific primitive types**.
-| Primitive Type | Description | Java Type| Python Type | Go Type |
-|---|---|---|---|---|
-| `BOOLEAN` | A binary value | boolean | bool | bool |
-| `INT8` | A 8-bit signed integer | int | | int8 |
-| `INT16` | A 16-bit signed integer | int | | int16 |
-| `INT32` | A 32-bit signed integer | int | | int32 |
-| `INT64` | A 64-bit signed integer | int | | int64 |
-| `FLOAT` | A single precision (32-bit) IEEE 754 floating-point number | float | float | float32 |
-| `DOUBLE` | A double-precision (64-bit) IEEE 754 floating-point number | double | float | float64|
-| `BYTES` | A sequence of 8-bit unsigned bytes | byte[], ByteBuffer, ByteBuf | bytes | []byte |
-| `STRING` | A Unicode character sequence | string | str | string|
-| `TIMESTAMP` (`DATE`, `TIME`) | A logic type represents a specific instant in time with millisecond precision. <br />It stores the number of milliseconds since `January 1, 1970, 00:00:00 GMT` as an `INT64` value | java.sql.Timestamp (java.sql.Time, java.util.Date) | | |
-| INSTANT | A single instantaneous point on the time-line with nanoseconds precision| java.time.Instant | | |
-| LOCAL_DATE | An immutable date-time object that represents a date, often viewed as year-month-day| java.time.LocalDate | | |
-| LOCAL_TIME | An immutable date-time object that represents a time, often viewed as hour-minute-second. Time is represented to nanosecond precision.| java.time.LocalDateTime | |
-| LOCAL_DATE_TIME | An immutable date-time object that represents a date-time, often viewed as year-month-day-hour-minute-second | java.time.LocalTime | |
+| Primitive Type | Description | Java Type| Python Type | Go Type | C++ Type | C# Type|
+|---|---|---|---|---|---|---|
+| `BOOLEAN` | A binary value. | boolean | bool | bool | bool | bool |
+| `INT8` | A 8-bit signed integer. | int | int | int8 | int8_t | byte |
+| `INT16` | A 16-bit signed integer. | int | int | int16 | int16_t | short |
+| `INT32` | A 32-bit signed integer. | int | int | int32 | int32_t | int |
+| `INT64` | A 64-bit signed integer. | int | int | int64 | int64_t | long |
+| `FLOAT` | A single precision (32-bit) IEEE 754 floating-point number. | float | float | float32 | float | float |
+| `DOUBLE` | A double-precision (64-bit) IEEE 754 floating-point number. | double | double | float64| double | double |
+| `BYTES` | A sequence of 8-bit unsigned bytes. | byte[], ByteBuffer, ByteBuf | bytes | []byte | void * | byte[], `ReadOnlySequence<byte>` |
+| `STRING` | An Unicode character sequence. | string | str | string| std::string | string |
+| `TIMESTAMP` (`DATE`, `TIME`) | A logic type represents a specific instant in time with millisecond precision. <br />It stores the number of milliseconds since `January 1, 1970, 00:00:00 GMT` as an `INT64` value. | java.sql.Timestamp (java.sql.Time, java.util.Date) | N/A | N/A | N/A | DateTime,TimeSpan |
+| `INSTANT`| A single instantaneous point on the timeline with nanoseconds precision. | java.time.Instant | N/A | N/A | N/A | N/A |
+| `LOCAL_DATE` | An immutable date-time object that represents a date, often viewed as year-month-day. | java.time.LocalDate | N/A | N/A | N/A | N/A |
+| `LOCAL_TIME` | An immutable date-time object that represents a time, often viewed as hour-minute-second. Time is represented to nanosecond precision. | java.time.LocalDateTime | N/A | N/A | N/A | N/A |
+| LOCAL_DATE_TIME | An immutable date-time object that represents a date-time, often viewed as year-month-day-hour-minute-second. | java.time.LocalTime | N/A | N/A | N/A | N/A |
+
+:::note
-For primitive types, Pulsar does not store any schema data in `SchemaInfo`. The `type` in `SchemaInfo` determines how to serialize and deserialize the data.
+Pulsar does not store any schema data in `SchemaInfo` for primitive types. Some of the primitive schema implementations can use the `properties` parameter to store implementation-specific tunable settings. For example, a string schema can use `properties` to store the encoding charset to serialize and deserialize strings.
-Some of the primitive schema implementations can use `properties` to store implementation-specific tunable settings. For example, a `string` schema can use `properties` to store the encoding charset to serialize and deserialize strings.
+:::
-For more instructions, see [Construct a string schema](schema-get-started.md#construct-a-string-schema).
+For more instructions and examples, see [Construct a string schema](schema-get-started.md#string).
### Complex type
-Currently, Pulsar supports the following complex types:
+The following table outlines the complex types that Pulsar schema supports:
| Complex Type | Description |
|---|---|
-| `KeyValue` | Represents a complex type of a key/value pair. |
-| `Struct` | Handles structured data. It supports `AvroBaseStructSchema` and `ProtobufNativeSchema`. |
+| `Keyvalue` | Represents a complex key/value pair. |
Review Comment:
```suggestion
| `KeyValue` | Represents a complex key/value pair. |
```
##########
site2/docs/schema-understand.md:
##########
@@ -48,92 +46,227 @@ Pulsar supports various schema types, which are mainly divided into two categori
The following table outlines the primitive types that Pulsar schema supports, and the conversions between **schema types** and **language-specific primitive types**.
-| Primitive Type | Description | Java Type| Python Type | Go Type |
-|---|---|---|---|---|
-| `BOOLEAN` | A binary value | boolean | bool | bool |
-| `INT8` | A 8-bit signed integer | int | | int8 |
-| `INT16` | A 16-bit signed integer | int | | int16 |
-| `INT32` | A 32-bit signed integer | int | | int32 |
-| `INT64` | A 64-bit signed integer | int | | int64 |
-| `FLOAT` | A single precision (32-bit) IEEE 754 floating-point number | float | float | float32 |
-| `DOUBLE` | A double-precision (64-bit) IEEE 754 floating-point number | double | float | float64|
-| `BYTES` | A sequence of 8-bit unsigned bytes | byte[], ByteBuffer, ByteBuf | bytes | []byte |
-| `STRING` | A Unicode character sequence | string | str | string|
-| `TIMESTAMP` (`DATE`, `TIME`) | A logic type represents a specific instant in time with millisecond precision. <br />It stores the number of milliseconds since `January 1, 1970, 00:00:00 GMT` as an `INT64` value | java.sql.Timestamp (java.sql.Time, java.util.Date) | | |
-| INSTANT | A single instantaneous point on the time-line with nanoseconds precision| java.time.Instant | | |
-| LOCAL_DATE | An immutable date-time object that represents a date, often viewed as year-month-day| java.time.LocalDate | | |
-| LOCAL_TIME | An immutable date-time object that represents a time, often viewed as hour-minute-second. Time is represented to nanosecond precision.| java.time.LocalDateTime | |
-| LOCAL_DATE_TIME | An immutable date-time object that represents a date-time, often viewed as year-month-day-hour-minute-second | java.time.LocalTime | |
+| Primitive Type | Description | Java Type| Python Type | Go Type | C++ Type | C# Type|
+|---|---|---|---|---|---|---|
+| `BOOLEAN` | A binary value. | boolean | bool | bool | bool | bool |
+| `INT8` | A 8-bit signed integer. | int | int | int8 | int8_t | byte |
+| `INT16` | A 16-bit signed integer. | int | int | int16 | int16_t | short |
+| `INT32` | A 32-bit signed integer. | int | int | int32 | int32_t | int |
+| `INT64` | A 64-bit signed integer. | int | int | int64 | int64_t | long |
+| `FLOAT` | A single precision (32-bit) IEEE 754 floating-point number. | float | float | float32 | float | float |
+| `DOUBLE` | A double-precision (64-bit) IEEE 754 floating-point number. | double | double | float64| double | double |
+| `BYTES` | A sequence of 8-bit unsigned bytes. | byte[], ByteBuffer, ByteBuf | bytes | []byte | void * | byte[], `ReadOnlySequence<byte>` |
+| `STRING` | An Unicode character sequence. | string | str | string| std::string | string |
+| `TIMESTAMP` (`DATE`, `TIME`) | A logic type represents a specific instant in time with millisecond precision. <br />It stores the number of milliseconds since `January 1, 1970, 00:00:00 GMT` as an `INT64` value. | java.sql.Timestamp (java.sql.Time, java.util.Date) | N/A | N/A | N/A | DateTime,TimeSpan |
+| `INSTANT`| A single instantaneous point on the timeline with nanoseconds precision. | java.time.Instant | N/A | N/A | N/A | N/A |
+| `LOCAL_DATE` | An immutable date-time object that represents a date, often viewed as year-month-day. | java.time.LocalDate | N/A | N/A | N/A | N/A |
+| `LOCAL_TIME` | An immutable date-time object that represents a time, often viewed as hour-minute-second. Time is represented to nanosecond precision. | java.time.LocalDateTime | N/A | N/A | N/A | N/A |
+| LOCAL_DATE_TIME | An immutable date-time object that represents a date-time, often viewed as year-month-day-hour-minute-second. | java.time.LocalTime | N/A | N/A | N/A | N/A |
+
+:::note
-For primitive types, Pulsar does not store any schema data in `SchemaInfo`. The `type` in `SchemaInfo` determines how to serialize and deserialize the data.
+Pulsar does not store any schema data in `SchemaInfo` for primitive types. Some of the primitive schema implementations can use the `properties` parameter to store implementation-specific tunable settings. For example, a string schema can use `properties` to store the encoding charset to serialize and deserialize strings.
-Some of the primitive schema implementations can use `properties` to store implementation-specific tunable settings. For example, a `string` schema can use `properties` to store the encoding charset to serialize and deserialize strings.
+:::
-For more instructions, see [Construct a string schema](schema-get-started.md#construct-a-string-schema).
+For more instructions and examples, see [Construct a string schema](schema-get-started.md#string).
### Complex type
-Currently, Pulsar supports the following complex types:
+The following table outlines the complex types that Pulsar schema supports:
| Complex Type | Description |
|---|---|
-| `KeyValue` | Represents a complex type of a key/value pair. |
-| `Struct` | Handles structured data. It supports `AvroBaseStructSchema` and `ProtobufNativeSchema`. |
+| `Keyvalue` | Represents a complex key/value pair. |
+| `Struct` | Represents structured data, including `AvroBaseStructSchema`, `ProtobufNativeSchema` and `Schema.NATIVE_AVRO`. |
#### `KeyValue` schema
-`KeyValue` schema helps applications define schemas for both key and value. Pulsar stores the `SchemaInfo` of key schema and the value schema together.
+`KeyValue` schema helps applications define schemas for both key and value. Pulsar stores the `SchemaInfo` of the key schema and the value schema together.
-You can choose the encoding type when constructing the key/value schema.：
+Pulsar provides the following methods to encode a **single** key/value pair in a message：
* `INLINE` - Key/value pairs are encoded together in the message payload.
Review Comment:
```suggestion
* `INLINE` - Key/Value pairs are encoded together in the message payload.
```
Although it's not related to this PR, better to keep it consistent.
##########
site2/docs/schema-understand.md:
##########
@@ -48,92 +46,227 @@ Pulsar supports various schema types, which are mainly divided into two categori
The following table outlines the primitive types that Pulsar schema supports, and the conversions between **schema types** and **language-specific primitive types**.
-| Primitive Type | Description | Java Type| Python Type | Go Type |
-|---|---|---|---|---|
-| `BOOLEAN` | A binary value | boolean | bool | bool |
-| `INT8` | A 8-bit signed integer | int | | int8 |
-| `INT16` | A 16-bit signed integer | int | | int16 |
-| `INT32` | A 32-bit signed integer | int | | int32 |
-| `INT64` | A 64-bit signed integer | int | | int64 |
-| `FLOAT` | A single precision (32-bit) IEEE 754 floating-point number | float | float | float32 |
-| `DOUBLE` | A double-precision (64-bit) IEEE 754 floating-point number | double | float | float64|
-| `BYTES` | A sequence of 8-bit unsigned bytes | byte[], ByteBuffer, ByteBuf | bytes | []byte |
-| `STRING` | A Unicode character sequence | string | str | string|
-| `TIMESTAMP` (`DATE`, `TIME`) | A logic type represents a specific instant in time with millisecond precision. <br />It stores the number of milliseconds since `January 1, 1970, 00:00:00 GMT` as an `INT64` value | java.sql.Timestamp (java.sql.Time, java.util.Date) | | |
-| INSTANT | A single instantaneous point on the time-line with nanoseconds precision| java.time.Instant | | |
-| LOCAL_DATE | An immutable date-time object that represents a date, often viewed as year-month-day| java.time.LocalDate | | |
-| LOCAL_TIME | An immutable date-time object that represents a time, often viewed as hour-minute-second. Time is represented to nanosecond precision.| java.time.LocalDateTime | |
-| LOCAL_DATE_TIME | An immutable date-time object that represents a date-time, often viewed as year-month-day-hour-minute-second | java.time.LocalTime | |
+| Primitive Type | Description | Java Type| Python Type | Go Type | C++ Type | C# Type|
+|---|---|---|---|---|---|---|
+| `BOOLEAN` | A binary value. | boolean | bool | bool | bool | bool |
+| `INT8` | A 8-bit signed integer. | int | int | int8 | int8_t | byte |
+| `INT16` | A 16-bit signed integer. | int | int | int16 | int16_t | short |
+| `INT32` | A 32-bit signed integer. | int | int | int32 | int32_t | int |
+| `INT64` | A 64-bit signed integer. | int | int | int64 | int64_t | long |
+| `FLOAT` | A single precision (32-bit) IEEE 754 floating-point number. | float | float | float32 | float | float |
+| `DOUBLE` | A double-precision (64-bit) IEEE 754 floating-point number. | double | double | float64| double | double |
+| `BYTES` | A sequence of 8-bit unsigned bytes. | byte[], ByteBuffer, ByteBuf | bytes | []byte | void * | byte[], `ReadOnlySequence<byte>` |
+| `STRING` | An Unicode character sequence. | string | str | string| std::string | string |
+| `TIMESTAMP` (`DATE`, `TIME`) | A logic type represents a specific instant in time with millisecond precision. <br />It stores the number of milliseconds since `January 1, 1970, 00:00:00 GMT` as an `INT64` value. | java.sql.Timestamp (java.sql.Time, java.util.Date) | N/A | N/A | N/A | DateTime,TimeSpan |
+| `INSTANT`| A single instantaneous point on the timeline with nanoseconds precision. | java.time.Instant | N/A | N/A | N/A | N/A |
+| `LOCAL_DATE` | An immutable date-time object that represents a date, often viewed as year-month-day. | java.time.LocalDate | N/A | N/A | N/A | N/A |
+| `LOCAL_TIME` | An immutable date-time object that represents a time, often viewed as hour-minute-second. Time is represented to nanosecond precision. | java.time.LocalDateTime | N/A | N/A | N/A | N/A |
+| LOCAL_DATE_TIME | An immutable date-time object that represents a date-time, often viewed as year-month-day-hour-minute-second. | java.time.LocalTime | N/A | N/A | N/A | N/A |
+
+:::note
-For primitive types, Pulsar does not store any schema data in `SchemaInfo`. The `type` in `SchemaInfo` determines how to serialize and deserialize the data.
+Pulsar does not store any schema data in `SchemaInfo` for primitive types. Some of the primitive schema implementations can use the `properties` parameter to store implementation-specific tunable settings. For example, a string schema can use `properties` to store the encoding charset to serialize and deserialize strings.
-Some of the primitive schema implementations can use `properties` to store implementation-specific tunable settings. For example, a `string` schema can use `properties` to store the encoding charset to serialize and deserialize strings.
+:::
-For more instructions, see [Construct a string schema](schema-get-started.md#construct-a-string-schema).
+For more instructions and examples, see [Construct a string schema](schema-get-started.md#string).
### Complex type
-Currently, Pulsar supports the following complex types:
+The following table outlines the complex types that Pulsar schema supports:
| Complex Type | Description |
|---|---|
-| `KeyValue` | Represents a complex type of a key/value pair. |
-| `Struct` | Handles structured data. It supports `AvroBaseStructSchema` and `ProtobufNativeSchema`. |
+| `Keyvalue` | Represents a complex key/value pair. |
+| `Struct` | Represents structured data, including `AvroBaseStructSchema`, `ProtobufNativeSchema` and `Schema.NATIVE_AVRO`. |
#### `KeyValue` schema
-`KeyValue` schema helps applications define schemas for both key and value. Pulsar stores the `SchemaInfo` of key schema and the value schema together.
+`KeyValue` schema helps applications define schemas for both key and value. Pulsar stores the `SchemaInfo` of the key schema and the value schema together.
-You can choose the encoding type when constructing the key/value schema.：
+Pulsar provides the following methods to encode a **single** key/value pair in a message：
* `INLINE` - Key/value pairs are encoded together in the message payload.
-* `SEPARATED` - see [Construct a key/value schema](schema-get-started.md#construct-a-keyvalue-schema).
+* `SEPARATED` - The Key is stored as a message key, while the value is stored as the message payload. See [Construct a key/value schema](schema-get-started.md#keyvalue) for more details.
#### `Struct` schema
-`Struct` schema supports `AvroBaseStructSchema` and `ProtobufNativeSchema`.
+The following table outlines the `struct` types that Pulsar schema supports:
|Type|Description|
---|---|
-`AvroBaseStructSchema`|Pulsar uses [Avro Specification](http://avro.apache.org/docs/current/spec.html) to declare the schema definition for `AvroBaseStructSchema`, which supports `AvroSchema`, `JsonSchema`, and `ProtobufSchema`. <br /><br />This allows Pulsar:<br />- to use the same tools to manage schema definitions<br />- to use different serialization or deserialization methods to handle data|
-`ProtobufNativeSchema`|`ProtobufNativeSchema` is based on protobuf native Descriptor. <br /><br />This allows Pulsar:<br />- to use native protobuf-v3 to serialize or deserialize data<br />- to use `AutoConsume` to deserialize data.
+`AvroBaseStructSchema`|Pulsar uses [Avro Specification](http://avro.apache.org/docs/current/spec.html) to declare the schema definition for `AvroBaseStructSchema`, which supports `AvroSchema`, `JsonSchema`, and `ProtobufSchema`. <br /><br />This allows Pulsar to:<br />- use the same tools to manage schema definitions.<br />- use different serialization or deserialization methods to handle data. |
+`ProtobufNativeSchema`|`ProtobufNativeSchema` is based on protobuf native Descriptor. <br /><br />This allows Pulsar to:<br />- use native protobuf-v3 to serialize or deserialize data<br />- use `AutoConsume` to deserialize data.|
+`Schema.NATIVE_AVRO` | `Schema.NATIVE_AVRO` is used to wrap a native Avro schema type `org.apache.avro.Schema`. The result is a schema instance that accepts a serialized Avro payload without validating it against the wrapped Avro schema. <br /><br />When you migrate or ingest event or messaging data from external systems (such as Kafka and Cassandra), the data is often already serialized in Avro format. The applications producing the data typically have validated the data against their schemas (including compatibility checks) and stored them in a database or a dedicated service (such as schema registry). The schema of each serialized data record is usually retrievable by some metadata attached to that record. In such cases, a Pulsar producer doesn't need to repeat the schema validation when sending the ingested events to a topic. All it needs to do is pass each message or event with its schema to Pulsar. |
Pulsar provides the following methods to use the `struct` schema.
* `static`
* `generic`
* `SchemaDefinition`
-For more examples, see [Construct a struct schema](schema-get-started.md#construct-a-struct-schema).
+This example shows how to construct a `struct` schema with these methods and use it to produce and consume messages.
+
+````mdx-code-block
+<Tabs
+ defaultValue="static"
+ values={[{"label":"static","value":"static"},{"label":"generic","value":"generic"},{"label":"SchemaDefinition","value":"SchemaDefinition"}]}>
+
+<TabItem value="static">
+
+You can predefine the `struct` schema, which can be a POJO in Java, a `struct` in Go, or classes generated by Avro or Protobuf tools.
+
+**Example**
+
+Pulsar gets the schema definition from the predefined `struct` using an Avro library. The schema definition is the schema data stored as a part of the `SchemaInfo`.
+
+1. Create the _User_ class to define the messages sent to Pulsar topics.
+
+ ```java
+ public static class User {
+ public String name;
+ public int age;
+ public User(String name, int age) {
+ this.name = name;
+ this.age = age
+ }
+ public User() {}
+ }
+ ```
+
+2. Create a producer with a `struct` schema and send messages.
+
+ ```java
+ Producer<User> producer = client.newProducer(Schema.AVRO(User.class)).create();
+ producer.newMessage().value(new User("pulsar-user", 1)).send();
+ ```
+
+3. Create a consumer with a `struct` schema and receive messages
+
+ ```java
+ Consumer<User> consumer = client.newConsumer(Schema.AVRO(User.class)).subscribe();
+ User user = consumer.receive().getValue();
+ ```
+
+</TabItem>
+<TabItem value="generic">
+
+Sometimes applications do not have pre-defined structs, and you can use this method to define schema and access data.
+
+You can define the `struct` schema using the `GenericSchemaBuilder`, generate a generic struct using `GenericRecordBuilder`, and consume messages into `GenericRecord`.
+
+**Example**
+
+1. Use `RecordSchemaBuilder` to build a schema.
+
+ ```java
+ RecordSchemaBuilder recordSchemaBuilder = SchemaBuilder.record("schemaName");
+ recordSchemaBuilder.field("intField").type(SchemaType.INT32);
+ SchemaInfo schemaInfo = recordSchemaBuilder.build(SchemaType.AVRO);
+
+ Consumer<GenericRecord> consumer = client.newConsumer(Schema.generic(schemaInfo))
+ .topic(topicName)
+ .subscriptionName(subscriptionName)
+ .subscribe();
+ Producer<GenericRecord> producer = client.newProducer(Schema.generic(schemaInfo))
+ .topic(topicName)
+ .create();
+ ```
+
+2. Use `RecordBuilder` to build the struct records.
+
+ ```java
+ GenericSchemaImpl schema = GenericAvroSchema.of(schemaInfo);
+ // send message
+ GenericRecord record = schema.newRecordBuilder().set("intField", 32).build();
+ producer.newMessage().value(record).send();
+ // receive message
+ Message<GenericRecord> msg = consumer.receive();
+
+ Assert.assertEquals(msg.getValue().getField("intField"), 32);
+ ```
+
+</TabItem>
+<TabItem value="SchemaDefinition">
+
+You can define the `schemaDefinition` to generate a `struct` schema.
+
+**Example**
+
+1. Create the _User_ class to define the messages sent to Pulsar topics.
+
+ ```java
+ public static class User {
+ public String name;
+ public int age;
+ public User(String name, int age) {
+ this.name = name;
+ this.age = age
+ }
+ public User() {}
+ }
+ ```
+
+2. Create a producer with a `SchemaDefinition` and send messages.
+
+ ```java
+ SchemaDefinition<User> schemaDefinition = SchemaDefinition.<User>builder().withPojo(User.class).build();
+ Producer<User> producer = client.newProducer(Schema.AVRO(schemaDefinition)).create();
+ producer.newMessage().value(new User ("pulsar-user", 1)).send();
+ ```
+
+3. Create a consumer with a `SchemaDefinition` schema and receive messages.
+
+ ```java
+ SchemaDefinition<User> schemaDefinition = SchemaDefinition.<User>builder().withPojo(User.class).build();
+ Consumer<User> consumer = client.newConsumer(Schema.AVRO(schemaDefinition)).subscribe();
+ User user = consumer.receive().getValue();
+ ```
+
+</TabItem>
+</Tabs>
+````
### Auto Schema
-If you don't know the schema type of a Pulsar topic in advance, you can use AUTO schema to produce or consume generic records to or from brokers.
+If there is no chance to know the schema type of a Pulsar topic in advance, you can use AUTO schemas to produce/consume generic records to/from brokers.
Auto schema contains two categories:
-* `AUTO_PRODUCE` transfers data from a producer to a Pulsar topic that has a schema and helps the producer validate whether the out-bound bytes are compatible with the schema of the topic. For more instructions, see [Construct an AUTO_PRODUCE schema](schema-get-started.md#construct-an-auto_produce-schema).
-* `AUTO_CONSUME` transfers data from a Pulsar topic that has a schema to a consumer and helps the topic validate whether the out-bound bytes are compatible with the consumer. In other words, the topic deserializes messages into language-specific objects `GenericRecord` using the `SchemaInfo` retrieved from brokers. Currently, `AUTO_CONSUME` supports AVRO, JSON and ProtobufNativeSchema schemas. For more instructions, see [Construct an AUTO_CONSUME schema](schema-get-started.md#construct-an-auto_consume-schema).
+* `AUTO_PRODUCE` transfers data from a producer to a Pulsar topic that has a schema and helps the producer validate whether the outbound bytes are compatible with the schema of the topic. For more instructions, see [Construct an AUTO_PRODUCE schema](schema-get-started.md#auto_produce).
+* `AUTO_CONSUME` transfers data from a Pulsar topic that has a schema to a consumer and helps the topic validate whether the out-bound bytes are compatible with the consumer. In other words, the topic deserializes messages into language-specific objects `GenericRecord` using the `SchemaInfo` retrieved from brokers. For more instructions, see [Construct an AUTO_CONSUME schema](schema-get-started.md#auto_consume).
-### Native Avro Schema
+## Schema validation
-When migrating or ingesting event or message data from external systems (such as Kafka and Cassandra), the events are often already serialized in Avro format. The applications producing the data typically have validated the data against their schemas (including compatibility checks) and stored them in a database or a dedicated service (such as a schema registry). The schema of each serialized data record is usually retrievable by some metadata attached to that record. In such cases, a Pulsar producer doesn't need to repeat the schema validation step when sending the ingested events to a topic. All it needs to do is passing each message or event with its schema to Pulsar.
+Schema validation enables brokers to reject producers/consumers without a schema.
Review Comment:
This would make users confused. I think this section is only focused on `isSchemaValidationEnforced`, and it's used to reject producers/consumers without schema. But the `Schema Validation` is to validate the schema compatibility, which is covered in the next section.
I think we need to change this section to `Schema Validation Enforced`.
##########
site2/docs/schema-understand.md:
##########
@@ -48,92 +46,227 @@ Pulsar supports various schema types, which are mainly divided into two categori
The following table outlines the primitive types that Pulsar schema supports, and the conversions between **schema types** and **language-specific primitive types**.
-| Primitive Type | Description | Java Type| Python Type | Go Type |
-|---|---|---|---|---|
-| `BOOLEAN` | A binary value | boolean | bool | bool |
-| `INT8` | A 8-bit signed integer | int | | int8 |
-| `INT16` | A 16-bit signed integer | int | | int16 |
-| `INT32` | A 32-bit signed integer | int | | int32 |
-| `INT64` | A 64-bit signed integer | int | | int64 |
-| `FLOAT` | A single precision (32-bit) IEEE 754 floating-point number | float | float | float32 |
-| `DOUBLE` | A double-precision (64-bit) IEEE 754 floating-point number | double | float | float64|
-| `BYTES` | A sequence of 8-bit unsigned bytes | byte[], ByteBuffer, ByteBuf | bytes | []byte |
-| `STRING` | A Unicode character sequence | string | str | string|
-| `TIMESTAMP` (`DATE`, `TIME`) | A logic type represents a specific instant in time with millisecond precision. <br />It stores the number of milliseconds since `January 1, 1970, 00:00:00 GMT` as an `INT64` value | java.sql.Timestamp (java.sql.Time, java.util.Date) | | |
-| INSTANT | A single instantaneous point on the time-line with nanoseconds precision| java.time.Instant | | |
-| LOCAL_DATE | An immutable date-time object that represents a date, often viewed as year-month-day| java.time.LocalDate | | |
-| LOCAL_TIME | An immutable date-time object that represents a time, often viewed as hour-minute-second. Time is represented to nanosecond precision.| java.time.LocalDateTime | |
-| LOCAL_DATE_TIME | An immutable date-time object that represents a date-time, often viewed as year-month-day-hour-minute-second | java.time.LocalTime | |
+| Primitive Type | Description | Java Type| Python Type | Go Type | C++ Type | C# Type|
+|---|---|---|---|---|---|---|
+| `BOOLEAN` | A binary value. | boolean | bool | bool | bool | bool |
+| `INT8` | A 8-bit signed integer. | int | int | int8 | int8_t | byte |
+| `INT16` | A 16-bit signed integer. | int | int | int16 | int16_t | short |
+| `INT32` | A 32-bit signed integer. | int | int | int32 | int32_t | int |
+| `INT64` | A 64-bit signed integer. | int | int | int64 | int64_t | long |
+| `FLOAT` | A single precision (32-bit) IEEE 754 floating-point number. | float | float | float32 | float | float |
+| `DOUBLE` | A double-precision (64-bit) IEEE 754 floating-point number. | double | double | float64| double | double |
+| `BYTES` | A sequence of 8-bit unsigned bytes. | byte[], ByteBuffer, ByteBuf | bytes | []byte | void * | byte[], `ReadOnlySequence<byte>` |
+| `STRING` | An Unicode character sequence. | string | str | string| std::string | string |
+| `TIMESTAMP` (`DATE`, `TIME`) | A logic type represents a specific instant in time with millisecond precision. <br />It stores the number of milliseconds since `January 1, 1970, 00:00:00 GMT` as an `INT64` value. | java.sql.Timestamp (java.sql.Time, java.util.Date) | N/A | N/A | N/A | DateTime,TimeSpan |
+| `INSTANT`| A single instantaneous point on the timeline with nanoseconds precision. | java.time.Instant | N/A | N/A | N/A | N/A |
+| `LOCAL_DATE` | An immutable date-time object that represents a date, often viewed as year-month-day. | java.time.LocalDate | N/A | N/A | N/A | N/A |
+| `LOCAL_TIME` | An immutable date-time object that represents a time, often viewed as hour-minute-second. Time is represented to nanosecond precision. | java.time.LocalDateTime | N/A | N/A | N/A | N/A |
+| LOCAL_DATE_TIME | An immutable date-time object that represents a date-time, often viewed as year-month-day-hour-minute-second. | java.time.LocalTime | N/A | N/A | N/A | N/A |
+
+:::note
-For primitive types, Pulsar does not store any schema data in `SchemaInfo`. The `type` in `SchemaInfo` determines how to serialize and deserialize the data.
+Pulsar does not store any schema data in `SchemaInfo` for primitive types. Some of the primitive schema implementations can use the `properties` parameter to store implementation-specific tunable settings. For example, a string schema can use `properties` to store the encoding charset to serialize and deserialize strings.
-Some of the primitive schema implementations can use `properties` to store implementation-specific tunable settings. For example, a `string` schema can use `properties` to store the encoding charset to serialize and deserialize strings.
+:::
-For more instructions, see [Construct a string schema](schema-get-started.md#construct-a-string-schema).
+For more instructions and examples, see [Construct a string schema](schema-get-started.md#string).
### Complex type
-Currently, Pulsar supports the following complex types:
+The following table outlines the complex types that Pulsar schema supports:
| Complex Type | Description |
|---|---|
-| `KeyValue` | Represents a complex type of a key/value pair. |
-| `Struct` | Handles structured data. It supports `AvroBaseStructSchema` and `ProtobufNativeSchema`. |
+| `Keyvalue` | Represents a complex key/value pair. |
+| `Struct` | Represents structured data, including `AvroBaseStructSchema`, `ProtobufNativeSchema` and `Schema.NATIVE_AVRO`. |
#### `KeyValue` schema
-`KeyValue` schema helps applications define schemas for both key and value. Pulsar stores the `SchemaInfo` of key schema and the value schema together.
+`KeyValue` schema helps applications define schemas for both key and value. Pulsar stores the `SchemaInfo` of the key schema and the value schema together.
-You can choose the encoding type when constructing the key/value schema.：
+Pulsar provides the following methods to encode a **single** key/value pair in a message：
* `INLINE` - Key/value pairs are encoded together in the message payload.
-* `SEPARATED` - see [Construct a key/value schema](schema-get-started.md#construct-a-keyvalue-schema).
+* `SEPARATED` - The Key is stored as a message key, while the value is stored as the message payload. See [Construct a key/value schema](schema-get-started.md#keyvalue) for more details.
#### `Struct` schema
-`Struct` schema supports `AvroBaseStructSchema` and `ProtobufNativeSchema`.
+The following table outlines the `struct` types that Pulsar schema supports:
|Type|Description|
---|---|
-`AvroBaseStructSchema`|Pulsar uses [Avro Specification](http://avro.apache.org/docs/current/spec.html) to declare the schema definition for `AvroBaseStructSchema`, which supports `AvroSchema`, `JsonSchema`, and `ProtobufSchema`. <br /><br />This allows Pulsar:<br />- to use the same tools to manage schema definitions<br />- to use different serialization or deserialization methods to handle data|
-`ProtobufNativeSchema`|`ProtobufNativeSchema` is based on protobuf native Descriptor. <br /><br />This allows Pulsar:<br />- to use native protobuf-v3 to serialize or deserialize data<br />- to use `AutoConsume` to deserialize data.
+`AvroBaseStructSchema`|Pulsar uses [Avro Specification](http://avro.apache.org/docs/current/spec.html) to declare the schema definition for `AvroBaseStructSchema`, which supports `AvroSchema`, `JsonSchema`, and `ProtobufSchema`. <br /><br />This allows Pulsar to:<br />- use the same tools to manage schema definitions.<br />- use different serialization or deserialization methods to handle data. |
+`ProtobufNativeSchema`|`ProtobufNativeSchema` is based on protobuf native Descriptor. <br /><br />This allows Pulsar to:<br />- use native protobuf-v3 to serialize or deserialize data<br />- use `AutoConsume` to deserialize data.|
+`Schema.NATIVE_AVRO` | `Schema.NATIVE_AVRO` is used to wrap a native Avro schema type `org.apache.avro.Schema`. The result is a schema instance that accepts a serialized Avro payload without validating it against the wrapped Avro schema. <br /><br />When you migrate or ingest event or messaging data from external systems (such as Kafka and Cassandra), the data is often already serialized in Avro format. The applications producing the data typically have validated the data against their schemas (including compatibility checks) and stored them in a database or a dedicated service (such as schema registry). The schema of each serialized data record is usually retrievable by some metadata attached to that record. In such cases, a Pulsar producer doesn't need to repeat the schema validation when sending the ingested events to a topic. All it needs to do is pass each message or event with its schema to Pulsar. |
Review Comment:
I think it's better to link it to the Get started chapter for users to read more information.
`ProtobufNativeSchema` -> `schema-get-started/#protobufnative`
`Schema.NATIVE_AVRO` -> `schema-get-started/#native-avro`
##########
site2/docs/schema-understand.md:
##########
@@ -48,92 +46,227 @@ Pulsar supports various schema types, which are mainly divided into two categori
The following table outlines the primitive types that Pulsar schema supports, and the conversions between **schema types** and **language-specific primitive types**.
-| Primitive Type | Description | Java Type| Python Type | Go Type |
-|---|---|---|---|---|
-| `BOOLEAN` | A binary value | boolean | bool | bool |
-| `INT8` | A 8-bit signed integer | int | | int8 |
-| `INT16` | A 16-bit signed integer | int | | int16 |
-| `INT32` | A 32-bit signed integer | int | | int32 |
-| `INT64` | A 64-bit signed integer | int | | int64 |
-| `FLOAT` | A single precision (32-bit) IEEE 754 floating-point number | float | float | float32 |
-| `DOUBLE` | A double-precision (64-bit) IEEE 754 floating-point number | double | float | float64|
-| `BYTES` | A sequence of 8-bit unsigned bytes | byte[], ByteBuffer, ByteBuf | bytes | []byte |
-| `STRING` | A Unicode character sequence | string | str | string|
-| `TIMESTAMP` (`DATE`, `TIME`) | A logic type represents a specific instant in time with millisecond precision. <br />It stores the number of milliseconds since `January 1, 1970, 00:00:00 GMT` as an `INT64` value | java.sql.Timestamp (java.sql.Time, java.util.Date) | | |
-| INSTANT | A single instantaneous point on the time-line with nanoseconds precision| java.time.Instant | | |
-| LOCAL_DATE | An immutable date-time object that represents a date, often viewed as year-month-day| java.time.LocalDate | | |
-| LOCAL_TIME | An immutable date-time object that represents a time, often viewed as hour-minute-second. Time is represented to nanosecond precision.| java.time.LocalDateTime | |
-| LOCAL_DATE_TIME | An immutable date-time object that represents a date-time, often viewed as year-month-day-hour-minute-second | java.time.LocalTime | |
+| Primitive Type | Description | Java Type| Python Type | Go Type | C++ Type | C# Type|
+|---|---|---|---|---|---|---|
+| `BOOLEAN` | A binary value. | boolean | bool | bool | bool | bool |
+| `INT8` | A 8-bit signed integer. | int | int | int8 | int8_t | byte |
+| `INT16` | A 16-bit signed integer. | int | int | int16 | int16_t | short |
+| `INT32` | A 32-bit signed integer. | int | int | int32 | int32_t | int |
+| `INT64` | A 64-bit signed integer. | int | int | int64 | int64_t | long |
+| `FLOAT` | A single precision (32-bit) IEEE 754 floating-point number. | float | float | float32 | float | float |
+| `DOUBLE` | A double-precision (64-bit) IEEE 754 floating-point number. | double | double | float64| double | double |
+| `BYTES` | A sequence of 8-bit unsigned bytes. | byte[], ByteBuffer, ByteBuf | bytes | []byte | void * | byte[], `ReadOnlySequence<byte>` |
+| `STRING` | An Unicode character sequence. | string | str | string| std::string | string |
+| `TIMESTAMP` (`DATE`, `TIME`) | A logic type represents a specific instant in time with millisecond precision. <br />It stores the number of milliseconds since `January 1, 1970, 00:00:00 GMT` as an `INT64` value. | java.sql.Timestamp (java.sql.Time, java.util.Date) | N/A | N/A | N/A | DateTime,TimeSpan |
+| `INSTANT`| A single instantaneous point on the timeline with nanoseconds precision. | java.time.Instant | N/A | N/A | N/A | N/A |
+| `LOCAL_DATE` | An immutable date-time object that represents a date, often viewed as year-month-day. | java.time.LocalDate | N/A | N/A | N/A | N/A |
+| `LOCAL_TIME` | An immutable date-time object that represents a time, often viewed as hour-minute-second. Time is represented to nanosecond precision. | java.time.LocalDateTime | N/A | N/A | N/A | N/A |
+| LOCAL_DATE_TIME | An immutable date-time object that represents a date-time, often viewed as year-month-day-hour-minute-second. | java.time.LocalTime | N/A | N/A | N/A | N/A |
+
+:::note
-For primitive types, Pulsar does not store any schema data in `SchemaInfo`. The `type` in `SchemaInfo` determines how to serialize and deserialize the data.
+Pulsar does not store any schema data in `SchemaInfo` for primitive types. Some of the primitive schema implementations can use the `properties` parameter to store implementation-specific tunable settings. For example, a string schema can use `properties` to store the encoding charset to serialize and deserialize strings.
-Some of the primitive schema implementations can use `properties` to store implementation-specific tunable settings. For example, a `string` schema can use `properties` to store the encoding charset to serialize and deserialize strings.
+:::
-For more instructions, see [Construct a string schema](schema-get-started.md#construct-a-string-schema).
+For more instructions and examples, see [Construct a string schema](schema-get-started.md#string).
### Complex type
-Currently, Pulsar supports the following complex types:
+The following table outlines the complex types that Pulsar schema supports:
| Complex Type | Description |
|---|---|
-| `KeyValue` | Represents a complex type of a key/value pair. |
-| `Struct` | Handles structured data. It supports `AvroBaseStructSchema` and `ProtobufNativeSchema`. |
+| `Keyvalue` | Represents a complex key/value pair. |
+| `Struct` | Represents structured data, including `AvroBaseStructSchema`, `ProtobufNativeSchema` and `Schema.NATIVE_AVRO`. |
#### `KeyValue` schema
-`KeyValue` schema helps applications define schemas for both key and value. Pulsar stores the `SchemaInfo` of key schema and the value schema together.
+`KeyValue` schema helps applications define schemas for both key and value. Pulsar stores the `SchemaInfo` of the key schema and the value schema together.
-You can choose the encoding type when constructing the key/value schema.：
+Pulsar provides the following methods to encode a **single** key/value pair in a message：
* `INLINE` - Key/value pairs are encoded together in the message payload.
-* `SEPARATED` - see [Construct a key/value schema](schema-get-started.md#construct-a-keyvalue-schema).
+* `SEPARATED` - The Key is stored as a message key, while the value is stored as the message payload. See [Construct a key/value schema](schema-get-started.md#keyvalue) for more details.
#### `Struct` schema
-`Struct` schema supports `AvroBaseStructSchema` and `ProtobufNativeSchema`.
+The following table outlines the `struct` types that Pulsar schema supports:
|Type|Description|
---|---|
-`AvroBaseStructSchema`|Pulsar uses [Avro Specification](http://avro.apache.org/docs/current/spec.html) to declare the schema definition for `AvroBaseStructSchema`, which supports `AvroSchema`, `JsonSchema`, and `ProtobufSchema`. <br /><br />This allows Pulsar:<br />- to use the same tools to manage schema definitions<br />- to use different serialization or deserialization methods to handle data|
-`ProtobufNativeSchema`|`ProtobufNativeSchema` is based on protobuf native Descriptor. <br /><br />This allows Pulsar:<br />- to use native protobuf-v3 to serialize or deserialize data<br />- to use `AutoConsume` to deserialize data.
+`AvroBaseStructSchema`|Pulsar uses [Avro Specification](http://avro.apache.org/docs/current/spec.html) to declare the schema definition for `AvroBaseStructSchema`, which supports `AvroSchema`, `JsonSchema`, and `ProtobufSchema`. <br /><br />This allows Pulsar to:<br />- use the same tools to manage schema definitions.<br />- use different serialization or deserialization methods to handle data. |
+`ProtobufNativeSchema`|`ProtobufNativeSchema` is based on protobuf native Descriptor. <br /><br />This allows Pulsar to:<br />- use native protobuf-v3 to serialize or deserialize data<br />- use `AutoConsume` to deserialize data.|
+`Schema.NATIVE_AVRO` | `Schema.NATIVE_AVRO` is used to wrap a native Avro schema type `org.apache.avro.Schema`. The result is a schema instance that accepts a serialized Avro payload without validating it against the wrapped Avro schema. <br /><br />When you migrate or ingest event or messaging data from external systems (such as Kafka and Cassandra), the data is often already serialized in Avro format. The applications producing the data typically have validated the data against their schemas (including compatibility checks) and stored them in a database or a dedicated service (such as schema registry). The schema of each serialized data record is usually retrievable by some metadata attached to that record. In such cases, a Pulsar producer doesn't need to repeat the schema validation when sending the ingested events to a topic. All it needs to do is pass each message or event with its schema to Pulsar. |
Review Comment:
We can use `NativeAvroBytesSchema` instead of `Schema.NATIVE_AVRO` to make it consistent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] momo-jun commented on a diff in pull request #18434: [refactor][doc] Improve schema docs
Posted by GitBox <gi...@apache.org>.
momo-jun commented on code in PR #18434:
URL: https://github.com/apache/pulsar/pull/18434#discussion_r1042921538
##########
site2/docs/schema-understand.md:
##########
@@ -48,92 +46,227 @@ Pulsar supports various schema types, which are mainly divided into two categori
The following table outlines the primitive types that Pulsar schema supports, and the conversions between **schema types** and **language-specific primitive types**.
-| Primitive Type | Description | Java Type| Python Type | Go Type |
-|---|---|---|---|---|
-| `BOOLEAN` | A binary value | boolean | bool | bool |
-| `INT8` | A 8-bit signed integer | int | | int8 |
-| `INT16` | A 16-bit signed integer | int | | int16 |
-| `INT32` | A 32-bit signed integer | int | | int32 |
-| `INT64` | A 64-bit signed integer | int | | int64 |
-| `FLOAT` | A single precision (32-bit) IEEE 754 floating-point number | float | float | float32 |
-| `DOUBLE` | A double-precision (64-bit) IEEE 754 floating-point number | double | float | float64|
-| `BYTES` | A sequence of 8-bit unsigned bytes | byte[], ByteBuffer, ByteBuf | bytes | []byte |
-| `STRING` | A Unicode character sequence | string | str | string|
-| `TIMESTAMP` (`DATE`, `TIME`) | A logic type represents a specific instant in time with millisecond precision. <br />It stores the number of milliseconds since `January 1, 1970, 00:00:00 GMT` as an `INT64` value | java.sql.Timestamp (java.sql.Time, java.util.Date) | | |
-| INSTANT | A single instantaneous point on the time-line with nanoseconds precision| java.time.Instant | | |
-| LOCAL_DATE | An immutable date-time object that represents a date, often viewed as year-month-day| java.time.LocalDate | | |
-| LOCAL_TIME | An immutable date-time object that represents a time, often viewed as hour-minute-second. Time is represented to nanosecond precision.| java.time.LocalDateTime | |
-| LOCAL_DATE_TIME | An immutable date-time object that represents a date-time, often viewed as year-month-day-hour-minute-second | java.time.LocalTime | |
+| Primitive Type | Description | Java Type| Python Type | Go Type | C++ Type | C# Type|
+|---|---|---|---|---|---|---|
Review Comment:
`N/A` is an abbreviation for `not applicable`. I think it's a common sense and can be easily googled :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] momo-jun commented on a diff in pull request #18434: [refactor][doc] Improve schema docs
Posted by GitBox <gi...@apache.org>.
momo-jun commented on code in PR #18434:
URL: https://github.com/apache/pulsar/pull/18434#discussion_r1034227178
##########
site2/docs/schema-get-started.md:
##########
@@ -11,80 +11,283 @@ import TabItem from '@theme/TabItem';
````
-This hands-on tutorial provides instructions and examples on how to construct and customize schemas.
+This hands-on tutorial provides instructions and examples on how to construct schemas. For the instructions on administrative tasks, see [Manage schema](admin-api-schemas.md).
-## Construct a string schema
+## Construct a schema
-This example demonstrates how to construct a [string schema](schema-understand.md#primitive-type) and use it to produce and consume messages in Java.
+- [Construct a bytes schema](#bytes)
+- [Construct a string schema](#string)
+- [Construct a key/value schema](#keyvalue)
+- [Construct an Avro schema](#avro)
+- [Construct a JSON schema](#json)
+- [Construct a Protobuf schema](#protobuf)
+- [Construct a ProtobufNative schema](#protobufnative)
+- [Construct a native Avro schema](#nativeavro)
+- [Construct an AUTO_PRODUCE schema](#auto_produce)
+- [Construct an AUTO_CONSUME schema](#auto_consume)
-1. Create a producer with a string schema and send messages.
+### `bytes`
+
+This example demonstrates how to construct a [bytes schema](schema-understand.md#primitive-type) using language-specific clients and use it to produce and consume messages.
+
+````mdx-code-block
+<Tabs groupId="api-choice"
+ defaultValue="Java"
+ values={[{"label":"Java","value":"Java"},{"label":"C++","value":"C++"},{"label":"Python","value":"Python"},{"label":"Go","value":"Go"}]}>
+
+<TabItem value="Java">
+
+```java
+ .topic("my-topic")
+ .create();
+Consumer<byte[]> consumer = pulsarClient.newConsumer(Schema.BYTES)
Review Comment:
@liangyepianzhou can you pls revisit this code snippet?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18434: [refactor][doc] Improve schema docs
Posted by GitBox <gi...@apache.org>.
liangyepianzhou commented on code in PR #18434:
URL: https://github.com/apache/pulsar/pull/18434#discussion_r1046610904
##########
site2/docs/schema-get-started.md:
##########
@@ -368,15 +872,13 @@ This example shows how to construct an [AUTO_PRODUCE](schema-understand.md#auto-
Produce<byte[]> pulsarProducer = client.newProducer(Schema.AUTO_PRODUCE())
Review Comment:
Good catch! The old `Schema.AUTO_PRODUCE()` is obsolete.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] momo-jun merged pull request #18434: [refactor][doc] Improve schema docs
Posted by GitBox <gi...@apache.org>.
momo-jun merged PR #18434:
URL: https://github.com/apache/pulsar/pull/18434
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] RobertIndie commented on a diff in pull request #18434: [refactor][doc] Improve schema docs
Posted by GitBox <gi...@apache.org>.
RobertIndie commented on code in PR #18434:
URL: https://github.com/apache/pulsar/pull/18434#discussion_r1044280700
##########
site2/docs/schema-understand.md:
##########
@@ -48,92 +46,227 @@ Pulsar supports various schema types, which are mainly divided into two categori
The following table outlines the primitive types that Pulsar schema supports, and the conversions between **schema types** and **language-specific primitive types**.
-| Primitive Type | Description | Java Type| Python Type | Go Type |
-|---|---|---|---|---|
-| `BOOLEAN` | A binary value | boolean | bool | bool |
-| `INT8` | A 8-bit signed integer | int | | int8 |
-| `INT16` | A 16-bit signed integer | int | | int16 |
-| `INT32` | A 32-bit signed integer | int | | int32 |
-| `INT64` | A 64-bit signed integer | int | | int64 |
-| `FLOAT` | A single precision (32-bit) IEEE 754 floating-point number | float | float | float32 |
-| `DOUBLE` | A double-precision (64-bit) IEEE 754 floating-point number | double | float | float64|
-| `BYTES` | A sequence of 8-bit unsigned bytes | byte[], ByteBuffer, ByteBuf | bytes | []byte |
-| `STRING` | A Unicode character sequence | string | str | string|
-| `TIMESTAMP` (`DATE`, `TIME`) | A logic type represents a specific instant in time with millisecond precision. <br />It stores the number of milliseconds since `January 1, 1970, 00:00:00 GMT` as an `INT64` value | java.sql.Timestamp (java.sql.Time, java.util.Date) | | |
-| INSTANT | A single instantaneous point on the time-line with nanoseconds precision| java.time.Instant | | |
-| LOCAL_DATE | An immutable date-time object that represents a date, often viewed as year-month-day| java.time.LocalDate | | |
-| LOCAL_TIME | An immutable date-time object that represents a time, often viewed as hour-minute-second. Time is represented to nanosecond precision.| java.time.LocalDateTime | |
-| LOCAL_DATE_TIME | An immutable date-time object that represents a date-time, often viewed as year-month-day-hour-minute-second | java.time.LocalTime | |
+| Primitive Type | Description | Java Type| Python Type | Go Type | C++ Type | C# Type|
+|---|---|---|---|---|---|---|
+| `BOOLEAN` | A binary value. | boolean | bool | bool | bool | bool |
+| `INT8` | A 8-bit signed integer. | int | int | int8 | int8_t | byte |
+| `INT16` | A 16-bit signed integer. | int | int | int16 | int16_t | short |
+| `INT32` | A 32-bit signed integer. | int | int | int32 | int32_t | int |
+| `INT64` | A 64-bit signed integer. | int | int | int64 | int64_t | long |
+| `FLOAT` | A single precision (32-bit) IEEE 754 floating-point number. | float | float | float32 | float | float |
+| `DOUBLE` | A double-precision (64-bit) IEEE 754 floating-point number. | double | double | float64| double | double |
+| `BYTES` | A sequence of 8-bit unsigned bytes. | byte[], ByteBuffer, ByteBuf | bytes | []byte | void * | byte[], `ReadOnlySequence<byte>` |
Review Comment:
@momo-jun Hi, I have changed my code suggestions. And it works for me now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] momo-jun commented on a diff in pull request #18434: [refactor][doc] Improve schema docs
Posted by GitBox <gi...@apache.org>.
momo-jun commented on code in PR #18434:
URL: https://github.com/apache/pulsar/pull/18434#discussion_r1042922337
##########
site2/docs/schema-understand.md:
##########
@@ -48,92 +46,227 @@ Pulsar supports various schema types, which are mainly divided into two categori
The following table outlines the primitive types that Pulsar schema supports, and the conversions between **schema types** and **language-specific primitive types**.
-| Primitive Type | Description | Java Type| Python Type | Go Type |
-|---|---|---|---|---|
-| `BOOLEAN` | A binary value | boolean | bool | bool |
-| `INT8` | A 8-bit signed integer | int | | int8 |
-| `INT16` | A 16-bit signed integer | int | | int16 |
-| `INT32` | A 32-bit signed integer | int | | int32 |
-| `INT64` | A 64-bit signed integer | int | | int64 |
-| `FLOAT` | A single precision (32-bit) IEEE 754 floating-point number | float | float | float32 |
-| `DOUBLE` | A double-precision (64-bit) IEEE 754 floating-point number | double | float | float64|
-| `BYTES` | A sequence of 8-bit unsigned bytes | byte[], ByteBuffer, ByteBuf | bytes | []byte |
-| `STRING` | A Unicode character sequence | string | str | string|
-| `TIMESTAMP` (`DATE`, `TIME`) | A logic type represents a specific instant in time with millisecond precision. <br />It stores the number of milliseconds since `January 1, 1970, 00:00:00 GMT` as an `INT64` value | java.sql.Timestamp (java.sql.Time, java.util.Date) | | |
-| INSTANT | A single instantaneous point on the time-line with nanoseconds precision| java.time.Instant | | |
-| LOCAL_DATE | An immutable date-time object that represents a date, often viewed as year-month-day| java.time.LocalDate | | |
-| LOCAL_TIME | An immutable date-time object that represents a time, often viewed as hour-minute-second. Time is represented to nanosecond precision.| java.time.LocalDateTime | |
-| LOCAL_DATE_TIME | An immutable date-time object that represents a date-time, often viewed as year-month-day-hour-minute-second | java.time.LocalTime | |
+| Primitive Type | Description | Java Type| Python Type | Go Type | C++ Type | C# Type|
+|---|---|---|---|---|---|---|
+| `BOOLEAN` | A binary value. | boolean | bool | bool | bool | bool |
+| `INT8` | A 8-bit signed integer. | int | int | int8 | int8_t | byte |
+| `INT16` | A 16-bit signed integer. | int | int | int16 | int16_t | short |
+| `INT32` | A 32-bit signed integer. | int | int | int32 | int32_t | int |
+| `INT64` | A 64-bit signed integer. | int | int | int64 | int64_t | long |
+| `FLOAT` | A single precision (32-bit) IEEE 754 floating-point number. | float | float | float32 | float | float |
+| `DOUBLE` | A double-precision (64-bit) IEEE 754 floating-point number. | double | double | float64| double | double |
+| `BYTES` | A sequence of 8-bit unsigned bytes. | byte[], ByteBuffer, ByteBuf | bytes | []byte | void * | byte[], `ReadOnlySequence<byte>` |
+| `STRING` | An Unicode character sequence. | string | str | string| std::string | string |
+| `TIMESTAMP` (`DATE`, `TIME`) | A logic type represents a specific instant in time with millisecond precision. <br />It stores the number of milliseconds since `January 1, 1970, 00:00:00 GMT` as an `INT64` value. | java.sql.Timestamp (java.sql.Time, java.util.Date) | N/A | N/A | N/A | DateTime,TimeSpan |
+| `INSTANT`| A single instantaneous point on the timeline with nanoseconds precision. | java.time.Instant | N/A | N/A | N/A | N/A |
+| `LOCAL_DATE` | An immutable date-time object that represents a date, often viewed as year-month-day. | java.time.LocalDate | N/A | N/A | N/A | N/A |
+| `LOCAL_TIME` | An immutable date-time object that represents a time, often viewed as hour-minute-second. Time is represented to nanosecond precision. | java.time.LocalDateTime | N/A | N/A | N/A | N/A |
+| LOCAL_DATE_TIME | An immutable date-time object that represents a date-time, often viewed as year-month-day-hour-minute-second. | java.time.LocalTime | N/A | N/A | N/A | N/A |
+
+:::note
-For primitive types, Pulsar does not store any schema data in `SchemaInfo`. The `type` in `SchemaInfo` determines how to serialize and deserialize the data.
+Pulsar does not store any schema data in `SchemaInfo` for primitive types. Some of the primitive schema implementations can use the `properties` parameter to store implementation-specific tunable settings. For example, a string schema can use `properties` to store the encoding charset to serialize and deserialize strings.
-Some of the primitive schema implementations can use `properties` to store implementation-specific tunable settings. For example, a `string` schema can use `properties` to store the encoding charset to serialize and deserialize strings.
+:::
-For more instructions, see [Construct a string schema](schema-get-started.md#construct-a-string-schema).
+For more instructions and examples, see [Construct a string schema](schema-get-started.md#string).
### Complex type
-Currently, Pulsar supports the following complex types:
+The following table outlines the complex types that Pulsar schema supports:
| Complex Type | Description |
|---|---|
-| `KeyValue` | Represents a complex type of a key/value pair. |
-| `Struct` | Handles structured data. It supports `AvroBaseStructSchema` and `ProtobufNativeSchema`. |
+| `Keyvalue` | Represents a complex key/value pair. |
+| `Struct` | Represents structured data, including `AvroBaseStructSchema`, `ProtobufNativeSchema` and `Schema.NATIVE_AVRO`. |
#### `KeyValue` schema
-`KeyValue` schema helps applications define schemas for both key and value. Pulsar stores the `SchemaInfo` of key schema and the value schema together.
+`KeyValue` schema helps applications define schemas for both key and value. Pulsar stores the `SchemaInfo` of the key schema and the value schema together.
-You can choose the encoding type when constructing the key/value schema.：
+Pulsar provides the following methods to encode a **single** key/value pair in a message：
* `INLINE` - Key/value pairs are encoded together in the message payload.
-* `SEPARATED` - see [Construct a key/value schema](schema-get-started.md#construct-a-keyvalue-schema).
+* `SEPARATED` - The Key is stored as a message key, while the value is stored as the message payload. See [Construct a key/value schema](schema-get-started.md#keyvalue) for more details.
#### `Struct` schema
-`Struct` schema supports `AvroBaseStructSchema` and `ProtobufNativeSchema`.
+The following table outlines the `struct` types that Pulsar schema supports:
|Type|Description|
---|---|
-`AvroBaseStructSchema`|Pulsar uses [Avro Specification](http://avro.apache.org/docs/current/spec.html) to declare the schema definition for `AvroBaseStructSchema`, which supports `AvroSchema`, `JsonSchema`, and `ProtobufSchema`. <br /><br />This allows Pulsar:<br />- to use the same tools to manage schema definitions<br />- to use different serialization or deserialization methods to handle data|
-`ProtobufNativeSchema`|`ProtobufNativeSchema` is based on protobuf native Descriptor. <br /><br />This allows Pulsar:<br />- to use native protobuf-v3 to serialize or deserialize data<br />- to use `AutoConsume` to deserialize data.
+`AvroBaseStructSchema`|Pulsar uses [Avro Specification](http://avro.apache.org/docs/current/spec.html) to declare the schema definition for `AvroBaseStructSchema`, which supports `AvroSchema`, `JsonSchema`, and `ProtobufSchema`. <br /><br />This allows Pulsar to:<br />- use the same tools to manage schema definitions.<br />- use different serialization or deserialization methods to handle data. |
+`ProtobufNativeSchema`|`ProtobufNativeSchema` is based on protobuf native Descriptor. <br /><br />This allows Pulsar to:<br />- use native protobuf-v3 to serialize or deserialize data<br />- use `AutoConsume` to deserialize data.|
+`Schema.NATIVE_AVRO` | `Schema.NATIVE_AVRO` is used to wrap a native Avro schema type `org.apache.avro.Schema`. The result is a schema instance that accepts a serialized Avro payload without validating it against the wrapped Avro schema. <br /><br />When you migrate or ingest event or messaging data from external systems (such as Kafka and Cassandra), the data is often already serialized in Avro format. The applications producing the data typically have validated the data against their schemas (including compatibility checks) and stored them in a database or a dedicated service (such as schema registry). The schema of each serialized data record is usually retrievable by some metadata attached to that record. In such cases, a Pulsar producer doesn't need to repeat the schema validation when sending the ingested events to a topic. All it needs to do is pass each message or event with its schema to Pulsar. |
Review Comment:
Makes sense. I've added these links.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] momo-jun commented on a diff in pull request #18434: [refactor][doc] Improve schema docs
Posted by GitBox <gi...@apache.org>.
momo-jun commented on code in PR #18434:
URL: https://github.com/apache/pulsar/pull/18434#discussion_r1046633060
##########
site2/docs/schema-understand.md:
##########
@@ -48,92 +46,227 @@ Pulsar supports various schema types, which are mainly divided into two categori
The following table outlines the primitive types that Pulsar schema supports, and the conversions between **schema types** and **language-specific primitive types**.
-| Primitive Type | Description | Java Type| Python Type | Go Type |
-|---|---|---|---|---|
-| `BOOLEAN` | A binary value | boolean | bool | bool |
-| `INT8` | A 8-bit signed integer | int | | int8 |
-| `INT16` | A 16-bit signed integer | int | | int16 |
-| `INT32` | A 32-bit signed integer | int | | int32 |
-| `INT64` | A 64-bit signed integer | int | | int64 |
-| `FLOAT` | A single precision (32-bit) IEEE 754 floating-point number | float | float | float32 |
-| `DOUBLE` | A double-precision (64-bit) IEEE 754 floating-point number | double | float | float64|
-| `BYTES` | A sequence of 8-bit unsigned bytes | byte[], ByteBuffer, ByteBuf | bytes | []byte |
-| `STRING` | A Unicode character sequence | string | str | string|
-| `TIMESTAMP` (`DATE`, `TIME`) | A logic type represents a specific instant in time with millisecond precision. <br />It stores the number of milliseconds since `January 1, 1970, 00:00:00 GMT` as an `INT64` value | java.sql.Timestamp (java.sql.Time, java.util.Date) | | |
-| INSTANT | A single instantaneous point on the time-line with nanoseconds precision| java.time.Instant | | |
-| LOCAL_DATE | An immutable date-time object that represents a date, often viewed as year-month-day| java.time.LocalDate | | |
-| LOCAL_TIME | An immutable date-time object that represents a time, often viewed as hour-minute-second. Time is represented to nanosecond precision.| java.time.LocalDateTime | |
-| LOCAL_DATE_TIME | An immutable date-time object that represents a date-time, often viewed as year-month-day-hour-minute-second | java.time.LocalTime | |
+| Primitive Type | Description | Java Type| Python Type | Go Type | C++ Type | C# Type|
+|---|---|---|---|---|---|---|
+| `BOOLEAN` | A binary value. | boolean | bool | bool | bool | bool |
+| `INT8` | A 8-bit signed integer. | int | int | int8 | int8_t | byte |
+| `INT16` | A 16-bit signed integer. | int | int | int16 | int16_t | short |
+| `INT32` | A 32-bit signed integer. | int | int | int32 | int32_t | int |
+| `INT64` | A 64-bit signed integer. | int | int | int64 | int64_t | long |
+| `FLOAT` | A single precision (32-bit) IEEE 754 floating-point number. | float | float | float32 | float | float |
+| `DOUBLE` | A double-precision (64-bit) IEEE 754 floating-point number. | double | double | float64| double | double |
+| `BYTES` | A sequence of 8-bit unsigned bytes. | byte[], ByteBuffer, ByteBuf | bytes | []byte | void * | byte[], `ReadOnlySequence<byte>` |
Review Comment:
Great! It works! Thank you so much!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] labuladong commented on a diff in pull request #18434: [refactor][doc] Improve schema docs
Posted by GitBox <gi...@apache.org>.
labuladong commented on code in PR #18434:
URL: https://github.com/apache/pulsar/pull/18434#discussion_r1044226904
##########
site2/docs/schema-get-started.md:
##########
@@ -368,15 +872,13 @@ This example shows how to construct an [AUTO_PRODUCE](schema-understand.md#auto-
Produce<byte[]> pulsarProducer = client.newProducer(Schema.AUTO_PRODUCE())
Review Comment:
```suggestion
Produce<byte[]> pulsarProducer = client.newProducer(Schema.AUTO_PRODUCE_BYTES())
```
This is the right name:
<img width="838" alt="image" src="https://user-images.githubusercontent.com/37220920/206664349-9f44f4ad-b714-4436-9ab2-0cdeb275c79a.png">
##########
site2/docs/schema-get-started.md:
##########
@@ -228,88 +329,350 @@ Producer<SensorReading> producer = client.newProducer(AvroSchema.of(SensorReadin
.create();
```
-### Avro-based schema using Java
-
-The following schema formats are currently available for Java:
+</TabItem>
+<TabItem value="C++">
-* No schema or the byte array schema (which can be applied using `Schema.BYTES`):
+ ```cpp
+ // Send messages
+ static const std::string exampleSchema =
+ "{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\","
+ "\"fields\":[{\"name\":\"a\",\"type\":\"int\"},{\"name\":\"b\",\"type\":\"int\"}]}";
+ Producer producer;
+ ProducerConfiguration producerConf;
+ producerConf.setSchema(SchemaInfo(AVRO, "Avro", exampleSchema));
+ client.createProducer("topic-avro", producerConf, producer);
- ```java
- Producer<byte[]> bytesProducer = client.newProducer(Schema.BYTES)
- .topic("some-raw-bytes-topic")
- .create();
+ // Receive messages
+ static const std::string exampleSchema =
+ "{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\","
+ "\"fields\":[{\"name\":\"a\",\"type\":\"int\"},{\"name\":\"b\",\"type\":\"int\"}]}";
+ ConsumerConfiguration consumerConf;
+ Consumer consumer;
+ consumerConf.setSchema(SchemaInfo(AVRO, "Avro", exampleSchema));
+ client.subscribe("topic-avro", "sub-2", consumerConf, consumer)
```
- Or, equivalently:
+</TabItem>
+<TabItem value="Python">
- ```java
- Producer<byte[]> bytesProducer = client.newProducer()
- .topic("some-raw-bytes-topic")
- .create();
- ```
+You can declare an `AvroSchema` using Python through one of the following methods.
-* `String` for normal UTF-8-encoded string data. Apply the schema using `Schema.STRING`:
+**Method 1: Record**
- ```java
- Producer<String> stringProducer = client.newProducer(Schema.STRING)
- .topic("some-string-topic")
- .create();
- ```
+Declare an `AvroSchema` by passing a class that inherits from `pulsar.schema.Record` and defines the fields as class variables.
-* Create JSON schemas for POJOs using `Schema.JSON`. The following is an example.
+```python
+class Example(Record):
+ a = Integer()
+ b = Integer()
- ```java
- Producer<MyPojo> pojoProducer = client.newProducer(Schema.JSON(MyPojo.class))
- .topic("some-pojo-topic")
- .create();
- ```
+producer = client.create_producer(
+ 'avro-schema-topic',
+ schema=AvroSchema(Example))
+r = Example(a=1, b=2)
+producer.send(r)
-* Generate Protobuf schemas using `Schema.PROTOBUF`. The following example shows how to create the Protobuf schema and use it to instantiate a new producer:
+consumer = client.subscribe(
+ 'avro-schema-topic',
+ 'sub',
+ schema=AvroSchema(Example))
+msg = consumer.receive()
+e = msg.value()
+```
- ```java
- Producer<MyProtobuf> protobufProducer = client.newProducer(Schema.PROTOBUF(MyProtobuf.class))
- .topic("some-protobuf-topic")
- .create();
- ```
+**Method 2: JSON definition**
+
+1. Declare an `AvroSchema` using JSON. In this case, Avro schemas are defined using JSON.
+
+ Below is an example of `AvroSchema` defined using a JSON file (`company.avsc`).
+
+ ```json
+ {
+ "doc": "this is doc",
+ "namespace": "example.avro",
+ "type": "record",
+ "name": "Company",
+ "fields": [
+ {"name": "name", "type": ["null", "string"]},
+ {"name": "address", "type": ["null", "string"]},
+ {"name": "employees", "type": ["null", {"type": "array", "items": {
+ "type": "record",
+ "name": "Employee",
+ "fields": [
+ {"name": "name", "type": ["null", "string"]},
+ {"name": "age", "type": ["null", "int"]}
+ ]
+ }}]},
+ {"name": "labels", "type": ["null", {"type": "map", "values": "string"}]}
+ ]
+ }
+ ```
-* Define Avro schemas with `Schema.AVRO`. The following code snippet demonstrates how to create and use Avro schema.
+2. Load a schema definition from a file by using [`avro.schema`](https://avro.apache.org/docs/current/getting-started-python/) or [`fastavro.schema`](https://fastavro.readthedocs.io/en/latest/schema.html#fastavro._schema_py.load_schema).
+
+ If you use the [JSON definition](#method-2-json-definition) method to declare an `AvroSchema`, you need to:
+ - Use [Python dict](https://developers.google.com/edu/python/dict-files) to produce and consume messages, which is different from using the [Record](#method-1-record) method.
+ - Set the value of the `_record_cls` parameter to `None` when generating an `AvroSchema` object.
+
+ **Example**
+
+ ```python
+ from fastavro.schema import load_schema
+ from pulsar.schema import *
+ schema_definition = load_schema("examples/company.avsc")
+ avro_schema = AvroSchema(None, schema_definition=schema_definition)
+ producer = client.create_producer(
+ topic=topic,
+ schema=avro_schema)
+ consumer = client.subscribe(topic, 'test', schema=avro_schema)
+ company = {
+ "name": "company-name" + str(i),
+ "address": 'xxx road xxx street ' + str(i),
+ "employees": [
+ {"name": "user" + str(i), "age": 20 + i},
+ {"name": "user" + str(i), "age": 30 + i},
+ {"name": "user" + str(i), "age": 35 + i},
+ ],
+ "labels": {
+ "industry": "software" + str(i),
+ "scale": ">100",
+ "funds": "1000000.0"
+ }
+ }
+ producer.send(company)
+ msg = consumer.receive()
+ # Users could get a dict object by `value()` method.
+ msg.value()
+ ```
- ```java
- Producer<MyAvro> avroProducer = client.newProducer(Schema.AVRO(MyAvro.class))
- .topic("some-avro-topic")
- .create();
- ```
+</TabItem>
+<TabItem value="Go">
+Suppose you have an `avroExampleStruct` class as follows, and you'd like to transmit it over a Pulsar topic.
-### Avro schema using C++
+```go
+ type avroExampleStruct struct {
+ ID int
+ Name string
+}
+```
-- The following example shows how to create a producer with an Avro schema.
+1. Add an `avroSchemaDef` like this:
- ```cpp
- static const std::string exampleSchema =
- "{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\","
- "\"fields\":[{\"name\":\"a\",\"type\":\"int\"},{\"name\":\"b\",\"type\":\"int\"}]}";
- Producer producer;
- ProducerConfiguration producerConf;
- producerConf.setSchema(SchemaInfo(AVRO, "Avro", exampleSchema));
- client.createProducer("topic-avro", producerConf, producer);
- ```
+ ```go
+ var (
+ exampleSchemaDef = "{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\"," +
+ "\"fields\":[{\"name\":\"ID\",\"type\":\"int\"},{\"name\":\"Name\",\"type\":\"string\"}]}"
+ )
+ ```
-- The following example shows how to create a consumer with an Avro schema.
+2. Create producer and consumer to send/receive messages:
+
+ ```go
+ //Create producer and send message
+ producer, err := client.CreateProducer(pulsar.ProducerOptions{
+ Topic: "my-topic",
+ Schema: pulsar.NewAvroSchema(exampleSchemaDef, nil),
+ })
+
+ msgId, err := producer.Send(context.Background(), &pulsar.ProducerMessage{
+ Value: avroExampleStruct{
+ ID: 10,
+ Name: "avroExampleStruct",
+ },
+ })
+
+ //Create Consumer and receive message
+ consumer, err := client.Subscribe(pulsar.ConsumerOptions{
+ Topic: "my-topic",
+ Schema: pulsar.NewAvroSchema(exampleSchemaDef, nil),
+ SubscriptionName: "my-sub",
+ Type: pulsar.Shared,
+ })
+ message, err := consumer.Receive(context.Background())
+ ```
- ```cpp
- static const std::string exampleSchema =
- "{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\","
- "\"fields\":[{\"name\":\"a\",\"type\":\"int\"},{\"name\":\"b\",\"type\":\"int\"}]}";
- ConsumerConfiguration consumerConf;
- Consumer consumer;
- consumerConf.setSchema(SchemaInfo(AVRO, "Avro", exampleSchema));
- client.subscribe("topic-avro", "sub-2", consumerConf, consumer)
- ```
+</TabItem>
+</Tabs>
+````
+
+### JSON
+
+````mdx-code-block
+<Tabs groupId="api-choice"
+ defaultValue="Java"
+ values={[{"label":"Java","value":"Java"},{"label":"C++","value":"C++"},{"label":"Python","value":"Python"},{"label":"Go","value":"Go"}]}>
+
+<TabItem value="Java">
+
+Similar to using `AvroSchema`, you can declare a `JsonSchema` by passing a class. The only difference is to use `JsonSchema` instead of `AvroSchema` when defining the schema type, as shown below. For how to use `AvroSchema` via record, see [Method 1 - Record](#method-1-record).
+
+```java
+static class SchemaDema {
+ public String name;
+ public int age;
+}
+
+Producer<SchemaDema> producer = pulsarClient.newProducer(Schema.JSON(SchemaDema.class))
+ .topic("my-topic")
+ .create();
+Consumer<SchemaDema> consumer = pulsarClient.newConsumer(Schema.JSON(SchemaDema.class))
+ .topic("my-topic")
+ .subscriptionName("my-sub")
+ .subscribe();
+
+SchemaDema schemaDema = new SchemaDema();
+schemaDema.name = "puslar";
+schemaDema.age = 20;
+producer.newMessage().value(schemaDema).send();
+
+Message<SchemaDema> message = consumer.receive(5, TimeUnit.SECONDS);
Review Comment:
What is `Dema`? Do you mean `Demo`?
##########
site2/docs/schema-get-started.md:
##########
@@ -228,88 +329,350 @@ Producer<SensorReading> producer = client.newProducer(AvroSchema.of(SensorReadin
.create();
```
-### Avro-based schema using Java
-
-The following schema formats are currently available for Java:
+</TabItem>
+<TabItem value="C++">
-* No schema or the byte array schema (which can be applied using `Schema.BYTES`):
+ ```cpp
+ // Send messages
+ static const std::string exampleSchema =
+ "{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\","
+ "\"fields\":[{\"name\":\"a\",\"type\":\"int\"},{\"name\":\"b\",\"type\":\"int\"}]}";
+ Producer producer;
+ ProducerConfiguration producerConf;
+ producerConf.setSchema(SchemaInfo(AVRO, "Avro", exampleSchema));
+ client.createProducer("topic-avro", producerConf, producer);
- ```java
- Producer<byte[]> bytesProducer = client.newProducer(Schema.BYTES)
- .topic("some-raw-bytes-topic")
- .create();
+ // Receive messages
+ static const std::string exampleSchema =
+ "{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\","
+ "\"fields\":[{\"name\":\"a\",\"type\":\"int\"},{\"name\":\"b\",\"type\":\"int\"}]}";
+ ConsumerConfiguration consumerConf;
+ Consumer consumer;
+ consumerConf.setSchema(SchemaInfo(AVRO, "Avro", exampleSchema));
+ client.subscribe("topic-avro", "sub-2", consumerConf, consumer)
```
- Or, equivalently:
+</TabItem>
+<TabItem value="Python">
- ```java
- Producer<byte[]> bytesProducer = client.newProducer()
- .topic("some-raw-bytes-topic")
- .create();
- ```
+You can declare an `AvroSchema` using Python through one of the following methods.
-* `String` for normal UTF-8-encoded string data. Apply the schema using `Schema.STRING`:
+**Method 1: Record**
- ```java
- Producer<String> stringProducer = client.newProducer(Schema.STRING)
- .topic("some-string-topic")
- .create();
- ```
+Declare an `AvroSchema` by passing a class that inherits from `pulsar.schema.Record` and defines the fields as class variables.
-* Create JSON schemas for POJOs using `Schema.JSON`. The following is an example.
+```python
+class Example(Record):
+ a = Integer()
+ b = Integer()
- ```java
- Producer<MyPojo> pojoProducer = client.newProducer(Schema.JSON(MyPojo.class))
- .topic("some-pojo-topic")
- .create();
- ```
+producer = client.create_producer(
+ 'avro-schema-topic',
+ schema=AvroSchema(Example))
+r = Example(a=1, b=2)
+producer.send(r)
-* Generate Protobuf schemas using `Schema.PROTOBUF`. The following example shows how to create the Protobuf schema and use it to instantiate a new producer:
+consumer = client.subscribe(
+ 'avro-schema-topic',
+ 'sub',
+ schema=AvroSchema(Example))
+msg = consumer.receive()
+e = msg.value()
+```
- ```java
- Producer<MyProtobuf> protobufProducer = client.newProducer(Schema.PROTOBUF(MyProtobuf.class))
- .topic("some-protobuf-topic")
- .create();
- ```
+**Method 2: JSON definition**
+
+1. Declare an `AvroSchema` using JSON. In this case, Avro schemas are defined using JSON.
+
+ Below is an example of `AvroSchema` defined using a JSON file (`company.avsc`).
+
+ ```json
+ {
+ "doc": "this is doc",
+ "namespace": "example.avro",
+ "type": "record",
+ "name": "Company",
+ "fields": [
+ {"name": "name", "type": ["null", "string"]},
+ {"name": "address", "type": ["null", "string"]},
+ {"name": "employees", "type": ["null", {"type": "array", "items": {
+ "type": "record",
+ "name": "Employee",
+ "fields": [
+ {"name": "name", "type": ["null", "string"]},
+ {"name": "age", "type": ["null", "int"]}
+ ]
+ }}]},
+ {"name": "labels", "type": ["null", {"type": "map", "values": "string"}]}
+ ]
+ }
+ ```
-* Define Avro schemas with `Schema.AVRO`. The following code snippet demonstrates how to create and use Avro schema.
+2. Load a schema definition from a file by using [`avro.schema`](https://avro.apache.org/docs/current/getting-started-python/) or [`fastavro.schema`](https://fastavro.readthedocs.io/en/latest/schema.html#fastavro._schema_py.load_schema).
+
+ If you use the [JSON definition](#method-2-json-definition) method to declare an `AvroSchema`, you need to:
+ - Use [Python dict](https://developers.google.com/edu/python/dict-files) to produce and consume messages, which is different from using the [Record](#method-1-record) method.
+ - Set the value of the `_record_cls` parameter to `None` when generating an `AvroSchema` object.
+
+ **Example**
+
+ ```python
+ from fastavro.schema import load_schema
+ from pulsar.schema import *
+ schema_definition = load_schema("examples/company.avsc")
+ avro_schema = AvroSchema(None, schema_definition=schema_definition)
+ producer = client.create_producer(
+ topic=topic,
+ schema=avro_schema)
+ consumer = client.subscribe(topic, 'test', schema=avro_schema)
+ company = {
+ "name": "company-name" + str(i),
+ "address": 'xxx road xxx street ' + str(i),
+ "employees": [
+ {"name": "user" + str(i), "age": 20 + i},
+ {"name": "user" + str(i), "age": 30 + i},
+ {"name": "user" + str(i), "age": 35 + i},
+ ],
+ "labels": {
+ "industry": "software" + str(i),
+ "scale": ">100",
+ "funds": "1000000.0"
+ }
+ }
+ producer.send(company)
+ msg = consumer.receive()
+ # Users could get a dict object by `value()` method.
+ msg.value()
+ ```
- ```java
- Producer<MyAvro> avroProducer = client.newProducer(Schema.AVRO(MyAvro.class))
- .topic("some-avro-topic")
- .create();
- ```
+</TabItem>
+<TabItem value="Go">
+Suppose you have an `avroExampleStruct` class as follows, and you'd like to transmit it over a Pulsar topic.
-### Avro schema using C++
+```go
+ type avroExampleStruct struct {
+ ID int
+ Name string
+}
+```
-- The following example shows how to create a producer with an Avro schema.
+1. Add an `avroSchemaDef` like this:
- ```cpp
- static const std::string exampleSchema =
- "{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\","
- "\"fields\":[{\"name\":\"a\",\"type\":\"int\"},{\"name\":\"b\",\"type\":\"int\"}]}";
- Producer producer;
- ProducerConfiguration producerConf;
- producerConf.setSchema(SchemaInfo(AVRO, "Avro", exampleSchema));
- client.createProducer("topic-avro", producerConf, producer);
- ```
+ ```go
+ var (
+ exampleSchemaDef = "{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\"," +
+ "\"fields\":[{\"name\":\"ID\",\"type\":\"int\"},{\"name\":\"Name\",\"type\":\"string\"}]}"
+ )
+ ```
-- The following example shows how to create a consumer with an Avro schema.
+2. Create producer and consumer to send/receive messages:
+
+ ```go
+ //Create producer and send message
+ producer, err := client.CreateProducer(pulsar.ProducerOptions{
+ Topic: "my-topic",
+ Schema: pulsar.NewAvroSchema(exampleSchemaDef, nil),
+ })
+
+ msgId, err := producer.Send(context.Background(), &pulsar.ProducerMessage{
+ Value: avroExampleStruct{
+ ID: 10,
+ Name: "avroExampleStruct",
+ },
+ })
+
+ //Create Consumer and receive message
+ consumer, err := client.Subscribe(pulsar.ConsumerOptions{
+ Topic: "my-topic",
+ Schema: pulsar.NewAvroSchema(exampleSchemaDef, nil),
+ SubscriptionName: "my-sub",
+ Type: pulsar.Shared,
+ })
+ message, err := consumer.Receive(context.Background())
+ ```
- ```cpp
- static const std::string exampleSchema =
- "{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\","
- "\"fields\":[{\"name\":\"a\",\"type\":\"int\"},{\"name\":\"b\",\"type\":\"int\"}]}";
- ConsumerConfiguration consumerConf;
- Consumer consumer;
- consumerConf.setSchema(SchemaInfo(AVRO, "Avro", exampleSchema));
- client.subscribe("topic-avro", "sub-2", consumerConf, consumer)
- ```
+</TabItem>
+</Tabs>
+````
+
+### JSON
+
+````mdx-code-block
+<Tabs groupId="api-choice"
+ defaultValue="Java"
+ values={[{"label":"Java","value":"Java"},{"label":"C++","value":"C++"},{"label":"Python","value":"Python"},{"label":"Go","value":"Go"}]}>
+
+<TabItem value="Java">
+
+Similar to using `AvroSchema`, you can declare a `JsonSchema` by passing a class. The only difference is to use `JsonSchema` instead of `AvroSchema` when defining the schema type, as shown below. For how to use `AvroSchema` via record, see [Method 1 - Record](#method-1-record).
+
+```java
+static class SchemaDema {
+ public String name;
+ public int age;
+}
+
+Producer<SchemaDema> producer = pulsarClient.newProducer(Schema.JSON(SchemaDema.class))
+ .topic("my-topic")
+ .create();
+Consumer<SchemaDema> consumer = pulsarClient.newConsumer(Schema.JSON(SchemaDema.class))
+ .topic("my-topic")
+ .subscriptionName("my-sub")
+ .subscribe();
+
+SchemaDema schemaDema = new SchemaDema();
+schemaDema.name = "puslar";
+schemaDema.age = 20;
+producer.newMessage().value(schemaDema).send();
+
+Message<SchemaDema> message = consumer.receive(5, TimeUnit.SECONDS);
Review Comment:
ping @momo-jun, this seems to to be a typo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] RobertIndie commented on a diff in pull request #18434: [refactor][doc] Improve schema docs
Posted by GitBox <gi...@apache.org>.
RobertIndie commented on code in PR #18434:
URL: https://github.com/apache/pulsar/pull/18434#discussion_r1045569927
##########
site2/docs/admin-api-schemas.md:
##########
@@ -489,42 +403,92 @@ To get the topic-level schema compatibility check strategy, use one of the follo
<TabItem value="Admin CLI">
-Use the [`pulsar-admin topicPolicies get-schema-compatibility-strategy`](/tools/pulsar-admin/) command.
+Use the `set-is-allow-auto-update-schema` subcommand.
Review Comment:
```suggestion
Use the `set-schema-validation-enforce` subcommand.
```
##########
site2/docs/schema-get-started.md:
##########
@@ -11,190 +11,291 @@ import TabItem from '@theme/TabItem';
````
-This hands-on tutorial provides instructions and examples on how to construct and customize schemas.
+This hands-on tutorial provides instructions and examples on how to construct schemas. For instructions on administrative tasks, see [Manage schema](admin-api-schemas.md).
-## Construct a string schema
+## Construct a schema
-This example demonstrates how to construct a [string schema](schema-understand.md#primitive-type) and use it to produce and consume messages in Java.
+- [Construct a bytes schema](#bytes)
+- [Construct a string schema](#string)
+- [Construct a key/value schema](#keyvalue)
+- [Construct an Avro schema](#avro)
+- [Construct a JSON schema](#json)
+- [Construct a Protobuf schema](#protobuf)
+- [Construct a ProtobufNative schema](#protobufnative)
+- [Construct a native Avro schema](#nativeavro)
+- [Construct an AUTO_PRODUCE schema](#auto_produce)
+- [Construct an AUTO_CONSUME schema](#auto_consume)
-1. Create a producer with a string schema and send messages.
+### `bytes`
Review Comment:
```suggestion
### bytes
```
Seems wired:
<img width="250" alt="image" src="https://user-images.githubusercontent.com/16974619/206999532-e4b74a2e-3a43-4db3-b4d8-925c35475264.png">
##########
site2/docs/admin-api-schemas.md:
##########
@@ -489,42 +403,92 @@ To get the topic-level schema compatibility check strategy, use one of the follo
<TabItem value="Admin CLI">
-Use the [`pulsar-admin topicPolicies get-schema-compatibility-strategy`](/tools/pulsar-admin/) command.
+Use the `set-is-allow-auto-update-schema` subcommand.
-```shell
-pulsar-admin topicPolicies get-schema-compatibility-strategy <topicName>
+```bash
+bin/pulsar-admin namespaces set-schema-validation-enforce --enable tenant/namespace
```
</TabItem>
<TabItem value="REST API">
-Send a `GET` request to this endpoint: {@inject: endpoint|GET|/admin/v2/topics/:tenant/:namespace/:topic|operation/schemaCompatibilityStrategy?version=@pulsar:version_number@}
+Send a `POST` request to a namespace endpoint: {@inject: endpoint|POST|/admin/v2/namespaces/:tenant/:namespace/schemaValidationEnforced|operation/schemaValidationEnforced?version=@pulsar:version_number@}
+
+The post payload is in JSON format.
+
+```json
+{
+“schemaValidationEnforced”: “true”
+}
+```
</TabItem>
<TabItem value="Java Admin API">
+Here is an example to enable schema validation enforcement for a tenant/namespace.
+
```java
-SchemaCompatibilityStrategy getSchemaCompatibilityStrategy(String topic, boolean applied)
+admin.namespaces().setSchemaValidationEnforced("my-namspace", true);
```
-Here is an example of getting the topic-level schema compatibility check strategy.
+</TabItem>
+</Tabs>
+````
-```java
-PulsarAdmin admin = …;
+### Disable schema validation enforcement
-// get the current applied schema compatibility strategy
-admin.topicPolicies().getSchemaCompatibilityStrategy("my-tenant/my-ns/my-topic", true);
+To disable schema validation enforcement at the **namespace** level, you can use one of the following commands.
-// only get the schema compatibility strategy from topic policies
-admin.topicPolicies().getSchemaCompatibilityStrategy("my-tenant/my-ns/my-topic", false);
+````mdx-code-block
+<Tabs groupId="api-choice"
+ defaultValue="Admin CLI"
+ values={[{"label":"Admin CLI","value":"Admin CLI"},{"label":"REST API","value":"REST API"},{"label":"Java Admin API","value":"Java Admin API"}]}>
+
+<TabItem value="Admin CLI">
+
+Use the `set-is-allow-auto-update-schema` subcommand.
Review Comment:
```suggestion
Use the `set-schema-validation-enforce` subcommand.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18434: [refactor][doc] Improve schema docs
Posted by GitBox <gi...@apache.org>.
liangyepianzhou commented on code in PR #18434:
URL: https://github.com/apache/pulsar/pull/18434#discussion_r1046608415
##########
site2/docs/schema-get-started.md:
##########
@@ -228,88 +329,350 @@ Producer<SensorReading> producer = client.newProducer(AvroSchema.of(SensorReadin
.create();
```
-### Avro-based schema using Java
-
-The following schema formats are currently available for Java:
+</TabItem>
+<TabItem value="C++">
-* No schema or the byte array schema (which can be applied using `Schema.BYTES`):
+ ```cpp
+ // Send messages
+ static const std::string exampleSchema =
+ "{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\","
+ "\"fields\":[{\"name\":\"a\",\"type\":\"int\"},{\"name\":\"b\",\"type\":\"int\"}]}";
+ Producer producer;
+ ProducerConfiguration producerConf;
+ producerConf.setSchema(SchemaInfo(AVRO, "Avro", exampleSchema));
+ client.createProducer("topic-avro", producerConf, producer);
- ```java
- Producer<byte[]> bytesProducer = client.newProducer(Schema.BYTES)
- .topic("some-raw-bytes-topic")
- .create();
+ // Receive messages
+ static const std::string exampleSchema =
+ "{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\","
+ "\"fields\":[{\"name\":\"a\",\"type\":\"int\"},{\"name\":\"b\",\"type\":\"int\"}]}";
+ ConsumerConfiguration consumerConf;
+ Consumer consumer;
+ consumerConf.setSchema(SchemaInfo(AVRO, "Avro", exampleSchema));
+ client.subscribe("topic-avro", "sub-2", consumerConf, consumer)
```
- Or, equivalently:
+</TabItem>
+<TabItem value="Python">
- ```java
- Producer<byte[]> bytesProducer = client.newProducer()
- .topic("some-raw-bytes-topic")
- .create();
- ```
+You can declare an `AvroSchema` using Python through one of the following methods.
-* `String` for normal UTF-8-encoded string data. Apply the schema using `Schema.STRING`:
+**Method 1: Record**
- ```java
- Producer<String> stringProducer = client.newProducer(Schema.STRING)
- .topic("some-string-topic")
- .create();
- ```
+Declare an `AvroSchema` by passing a class that inherits from `pulsar.schema.Record` and defines the fields as class variables.
-* Create JSON schemas for POJOs using `Schema.JSON`. The following is an example.
+```python
+class Example(Record):
+ a = Integer()
+ b = Integer()
- ```java
- Producer<MyPojo> pojoProducer = client.newProducer(Schema.JSON(MyPojo.class))
- .topic("some-pojo-topic")
- .create();
- ```
+producer = client.create_producer(
+ 'avro-schema-topic',
+ schema=AvroSchema(Example))
+r = Example(a=1, b=2)
+producer.send(r)
-* Generate Protobuf schemas using `Schema.PROTOBUF`. The following example shows how to create the Protobuf schema and use it to instantiate a new producer:
+consumer = client.subscribe(
+ 'avro-schema-topic',
+ 'sub',
+ schema=AvroSchema(Example))
+msg = consumer.receive()
+e = msg.value()
+```
- ```java
- Producer<MyProtobuf> protobufProducer = client.newProducer(Schema.PROTOBUF(MyProtobuf.class))
- .topic("some-protobuf-topic")
- .create();
- ```
+**Method 2: JSON definition**
+
+1. Declare an `AvroSchema` using JSON. In this case, Avro schemas are defined using JSON.
+
+ Below is an example of `AvroSchema` defined using a JSON file (`company.avsc`).
+
+ ```json
+ {
+ "doc": "this is doc",
+ "namespace": "example.avro",
+ "type": "record",
+ "name": "Company",
+ "fields": [
+ {"name": "name", "type": ["null", "string"]},
+ {"name": "address", "type": ["null", "string"]},
+ {"name": "employees", "type": ["null", {"type": "array", "items": {
+ "type": "record",
+ "name": "Employee",
+ "fields": [
+ {"name": "name", "type": ["null", "string"]},
+ {"name": "age", "type": ["null", "int"]}
+ ]
+ }}]},
+ {"name": "labels", "type": ["null", {"type": "map", "values": "string"}]}
+ ]
+ }
+ ```
-* Define Avro schemas with `Schema.AVRO`. The following code snippet demonstrates how to create and use Avro schema.
+2. Load a schema definition from a file by using [`avro.schema`](https://avro.apache.org/docs/current/getting-started-python/) or [`fastavro.schema`](https://fastavro.readthedocs.io/en/latest/schema.html#fastavro._schema_py.load_schema).
+
+ If you use the [JSON definition](#method-2-json-definition) method to declare an `AvroSchema`, you need to:
+ - Use [Python dict](https://developers.google.com/edu/python/dict-files) to produce and consume messages, which is different from using the [Record](#method-1-record) method.
+ - Set the value of the `_record_cls` parameter to `None` when generating an `AvroSchema` object.
+
+ **Example**
+
+ ```python
+ from fastavro.schema import load_schema
+ from pulsar.schema import *
+ schema_definition = load_schema("examples/company.avsc")
+ avro_schema = AvroSchema(None, schema_definition=schema_definition)
+ producer = client.create_producer(
+ topic=topic,
+ schema=avro_schema)
+ consumer = client.subscribe(topic, 'test', schema=avro_schema)
+ company = {
+ "name": "company-name" + str(i),
+ "address": 'xxx road xxx street ' + str(i),
+ "employees": [
+ {"name": "user" + str(i), "age": 20 + i},
+ {"name": "user" + str(i), "age": 30 + i},
+ {"name": "user" + str(i), "age": 35 + i},
+ ],
+ "labels": {
+ "industry": "software" + str(i),
+ "scale": ">100",
+ "funds": "1000000.0"
+ }
+ }
+ producer.send(company)
+ msg = consumer.receive()
+ # Users could get a dict object by `value()` method.
+ msg.value()
+ ```
- ```java
- Producer<MyAvro> avroProducer = client.newProducer(Schema.AVRO(MyAvro.class))
- .topic("some-avro-topic")
- .create();
- ```
+</TabItem>
+<TabItem value="Go">
+Suppose you have an `avroExampleStruct` class as follows, and you'd like to transmit it over a Pulsar topic.
-### Avro schema using C++
+```go
+ type avroExampleStruct struct {
+ ID int
+ Name string
+}
+```
-- The following example shows how to create a producer with an Avro schema.
+1. Add an `avroSchemaDef` like this:
- ```cpp
- static const std::string exampleSchema =
- "{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\","
- "\"fields\":[{\"name\":\"a\",\"type\":\"int\"},{\"name\":\"b\",\"type\":\"int\"}]}";
- Producer producer;
- ProducerConfiguration producerConf;
- producerConf.setSchema(SchemaInfo(AVRO, "Avro", exampleSchema));
- client.createProducer("topic-avro", producerConf, producer);
- ```
+ ```go
+ var (
+ exampleSchemaDef = "{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\"," +
+ "\"fields\":[{\"name\":\"ID\",\"type\":\"int\"},{\"name\":\"Name\",\"type\":\"string\"}]}"
+ )
+ ```
-- The following example shows how to create a consumer with an Avro schema.
+2. Create producer and consumer to send/receive messages:
+
+ ```go
+ //Create producer and send message
+ producer, err := client.CreateProducer(pulsar.ProducerOptions{
+ Topic: "my-topic",
+ Schema: pulsar.NewAvroSchema(exampleSchemaDef, nil),
+ })
+
+ msgId, err := producer.Send(context.Background(), &pulsar.ProducerMessage{
+ Value: avroExampleStruct{
+ ID: 10,
+ Name: "avroExampleStruct",
+ },
+ })
+
+ //Create Consumer and receive message
+ consumer, err := client.Subscribe(pulsar.ConsumerOptions{
+ Topic: "my-topic",
+ Schema: pulsar.NewAvroSchema(exampleSchemaDef, nil),
+ SubscriptionName: "my-sub",
+ Type: pulsar.Shared,
+ })
+ message, err := consumer.Receive(context.Background())
+ ```
- ```cpp
- static const std::string exampleSchema =
- "{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\","
- "\"fields\":[{\"name\":\"a\",\"type\":\"int\"},{\"name\":\"b\",\"type\":\"int\"}]}";
- ConsumerConfiguration consumerConf;
- Consumer consumer;
- consumerConf.setSchema(SchemaInfo(AVRO, "Avro", exampleSchema));
- client.subscribe("topic-avro", "sub-2", consumerConf, consumer)
- ```
+</TabItem>
+</Tabs>
+````
+
+### JSON
+
+````mdx-code-block
+<Tabs groupId="api-choice"
+ defaultValue="Java"
+ values={[{"label":"Java","value":"Java"},{"label":"C++","value":"C++"},{"label":"Python","value":"Python"},{"label":"Go","value":"Go"}]}>
+
+<TabItem value="Java">
+
+Similar to using `AvroSchema`, you can declare a `JsonSchema` by passing a class. The only difference is to use `JsonSchema` instead of `AvroSchema` when defining the schema type, as shown below. For how to use `AvroSchema` via record, see [Method 1 - Record](#method-1-record).
+
+```java
+static class SchemaDema {
+ public String name;
+ public int age;
+}
+
+Producer<SchemaDema> producer = pulsarClient.newProducer(Schema.JSON(SchemaDema.class))
+ .topic("my-topic")
+ .create();
+Consumer<SchemaDema> consumer = pulsarClient.newConsumer(Schema.JSON(SchemaDema.class))
+ .topic("my-topic")
+ .subscriptionName("my-sub")
+ .subscribe();
+
+SchemaDema schemaDema = new SchemaDema();
+schemaDema.name = "puslar";
+schemaDema.age = 20;
+producer.newMessage().value(schemaDema).send();
+
+Message<SchemaDema> message = consumer.receive(5, TimeUnit.SECONDS);
Review Comment:
There is a mistake. we can change it to `SchemaDemo`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] momo-jun commented on pull request #18434: [refactor][doc] Improve schema docs
Posted by GitBox <gi...@apache.org>.
momo-jun commented on PR #18434:
URL: https://github.com/apache/pulsar/pull/18434#issuecomment-1331794217
@labuladong thanks for your review. Comments have been addressed. PTAL.
@congbobo184 @liangyepianzhou @RobertIndie Do you have any other review comments?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] RobertIndie commented on a diff in pull request #18434: [refactor][doc] Improve schema docs
Posted by GitBox <gi...@apache.org>.
RobertIndie commented on code in PR #18434:
URL: https://github.com/apache/pulsar/pull/18434#discussion_r1044274797
##########
site2/docs/schema-understand.md:
##########
@@ -48,92 +46,227 @@ Pulsar supports various schema types, which are mainly divided into two categori
The following table outlines the primitive types that Pulsar schema supports, and the conversions between **schema types** and **language-specific primitive types**.
-| Primitive Type | Description | Java Type| Python Type | Go Type |
-|---|---|---|---|---|
-| `BOOLEAN` | A binary value | boolean | bool | bool |
-| `INT8` | A 8-bit signed integer | int | | int8 |
-| `INT16` | A 16-bit signed integer | int | | int16 |
-| `INT32` | A 32-bit signed integer | int | | int32 |
-| `INT64` | A 64-bit signed integer | int | | int64 |
-| `FLOAT` | A single precision (32-bit) IEEE 754 floating-point number | float | float | float32 |
-| `DOUBLE` | A double-precision (64-bit) IEEE 754 floating-point number | double | float | float64|
-| `BYTES` | A sequence of 8-bit unsigned bytes | byte[], ByteBuffer, ByteBuf | bytes | []byte |
-| `STRING` | A Unicode character sequence | string | str | string|
-| `TIMESTAMP` (`DATE`, `TIME`) | A logic type represents a specific instant in time with millisecond precision. <br />It stores the number of milliseconds since `January 1, 1970, 00:00:00 GMT` as an `INT64` value | java.sql.Timestamp (java.sql.Time, java.util.Date) | | |
-| INSTANT | A single instantaneous point on the time-line with nanoseconds precision| java.time.Instant | | |
-| LOCAL_DATE | An immutable date-time object that represents a date, often viewed as year-month-day| java.time.LocalDate | | |
-| LOCAL_TIME | An immutable date-time object that represents a time, often viewed as hour-minute-second. Time is represented to nanosecond precision.| java.time.LocalDateTime | |
-| LOCAL_DATE_TIME | An immutable date-time object that represents a date-time, often viewed as year-month-day-hour-minute-second | java.time.LocalTime | |
+| Primitive Type | Description | Java Type| Python Type | Go Type | C++ Type | C# Type|
+|---|---|---|---|---|---|---|
Review Comment:
Get it. Make sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] RobertIndie commented on a diff in pull request #18434: [refactor][doc] Improve schema docs
Posted by GitBox <gi...@apache.org>.
RobertIndie commented on code in PR #18434:
URL: https://github.com/apache/pulsar/pull/18434#discussion_r1041963617
##########
site2/docs/schema-understand.md:
##########
@@ -48,92 +46,227 @@ Pulsar supports various schema types, which are mainly divided into two categori
The following table outlines the primitive types that Pulsar schema supports, and the conversions between **schema types** and **language-specific primitive types**.
-| Primitive Type | Description | Java Type| Python Type | Go Type |
-|---|---|---|---|---|
-| `BOOLEAN` | A binary value | boolean | bool | bool |
-| `INT8` | A 8-bit signed integer | int | | int8 |
-| `INT16` | A 16-bit signed integer | int | | int16 |
-| `INT32` | A 32-bit signed integer | int | | int32 |
-| `INT64` | A 64-bit signed integer | int | | int64 |
-| `FLOAT` | A single precision (32-bit) IEEE 754 floating-point number | float | float | float32 |
-| `DOUBLE` | A double-precision (64-bit) IEEE 754 floating-point number | double | float | float64|
-| `BYTES` | A sequence of 8-bit unsigned bytes | byte[], ByteBuffer, ByteBuf | bytes | []byte |
-| `STRING` | A Unicode character sequence | string | str | string|
-| `TIMESTAMP` (`DATE`, `TIME`) | A logic type represents a specific instant in time with millisecond precision. <br />It stores the number of milliseconds since `January 1, 1970, 00:00:00 GMT` as an `INT64` value | java.sql.Timestamp (java.sql.Time, java.util.Date) | | |
-| INSTANT | A single instantaneous point on the time-line with nanoseconds precision| java.time.Instant | | |
-| LOCAL_DATE | An immutable date-time object that represents a date, often viewed as year-month-day| java.time.LocalDate | | |
-| LOCAL_TIME | An immutable date-time object that represents a time, often viewed as hour-minute-second. Time is represented to nanosecond precision.| java.time.LocalDateTime | |
-| LOCAL_DATE_TIME | An immutable date-time object that represents a date-time, often viewed as year-month-day-hour-minute-second | java.time.LocalTime | |
+| Primitive Type | Description | Java Type| Python Type | Go Type | C++ Type | C# Type|
+|---|---|---|---|---|---|---|
+| `BOOLEAN` | A binary value. | boolean | bool | bool | bool | bool |
+| `INT8` | A 8-bit signed integer. | int | int | int8 | int8_t | byte |
+| `INT16` | A 16-bit signed integer. | int | int | int16 | int16_t | short |
+| `INT32` | A 32-bit signed integer. | int | int | int32 | int32_t | int |
+| `INT64` | A 64-bit signed integer. | int | int | int64 | int64_t | long |
+| `FLOAT` | A single precision (32-bit) IEEE 754 floating-point number. | float | float | float32 | float | float |
+| `DOUBLE` | A double-precision (64-bit) IEEE 754 floating-point number. | double | double | float64| double | double |
+| `BYTES` | A sequence of 8-bit unsigned bytes. | byte[], ByteBuffer, ByteBuf | bytes | []byte | void * | byte[], `ReadOnlySequence<byte>` |
Review Comment:
```suggestion
| `BYTES` | A sequence of 8-bit unsigned bytes. | byte[], ByteBuffer, ByteBuf | bytes | []byte | void * | byte[], ReadOnlySequence<byte\> |
```
Looks wired in the preview:
<img width="462" alt="image" src="https://user-images.githubusercontent.com/16974619/206140861-5599dec8-87c1-4dc2-86bd-acd61f469217.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] labuladong commented on a diff in pull request #18434: [refactor][doc] Improve schema docs
Posted by GitBox <gi...@apache.org>.
labuladong commented on code in PR #18434:
URL: https://github.com/apache/pulsar/pull/18434#discussion_r1033415078
##########
site2/docs/schema-get-started.md:
##########
@@ -11,80 +11,283 @@ import TabItem from '@theme/TabItem';
````
-This hands-on tutorial provides instructions and examples on how to construct and customize schemas.
+This hands-on tutorial provides instructions and examples on how to construct schemas. For the instructions on administrative tasks, see [Manage schema](admin-api-schemas.md).
-## Construct a string schema
+## Construct a schema
-This example demonstrates how to construct a [string schema](schema-understand.md#primitive-type) and use it to produce and consume messages in Java.
+- [Construct a bytes schema](#bytes)
+- [Construct a string schema](#string)
+- [Construct a key/value schema](#keyvalue)
+- [Construct an Avro schema](#avro)
+- [Construct a JSON schema](#json)
+- [Construct a Protobuf schema](#protobuf)
+- [Construct a ProtobufNative schema](#protobufnative)
+- [Construct a native Avro schema](#nativeavro)
+- [Construct an AUTO_PRODUCE schema](#auto_produce)
+- [Construct an AUTO_CONSUME schema](#auto_consume)
-1. Create a producer with a string schema and send messages.
+### `bytes`
+
+This example demonstrates how to construct a [bytes schema](schema-understand.md#primitive-type) using language-specific clients and use it to produce and consume messages.
+
+````mdx-code-block
+<Tabs groupId="api-choice"
+ defaultValue="Java"
+ values={[{"label":"Java","value":"Java"},{"label":"C++","value":"C++"},{"label":"Python","value":"Python"},{"label":"Go","value":"Go"}]}>
+
+<TabItem value="Java">
+
+```java
+ .topic("my-topic")
+ .create();
+Consumer<byte[]> consumer = pulsarClient.newConsumer(Schema.BYTES)
Review Comment:
Seems missing some code here
##########
site2/docs/schema-understand.md:
##########
@@ -146,67 +159,88 @@ Producer<SensorReading> producer = client.newProducer(JSONSchema.of(SensorReadin
.create();
```
-The table below lists the possible scenarios when this connection attempt occurs and what happens in each scenario:
+The table below outlines the possible scenarios when this connection attempt occurs and the result of each scenario:
-| Scenario | What happens |
-| --- | --- |
-| <li>No schema exists for the topic. </li> | (1) The producer is created using the given schema. (2) Since no existing schema is compatible with the `SensorReading` schema, the schema is transmitted to the broker and stored. (3) Any consumer created using the same schema or topic can consume messages from the `sensor-data` topic. |
-| <li>A schema already exists. </li><li>The producer connects using the same schema that is already stored. </li> | (1) The schema is transmitted to the broker. (2) The broker determines that the schema is compatible. (3) The broker attempts to store the schema in [BookKeeper](concepts-architecture-overview.md#persistent-storage) but then determines that it's already stored, so it is used to tag produced messages. | <li>A schema already exists. </li><li>The producer connects using a new schema that is compatible. </li> | (1) The schema is transmitted to the broker. (2) The broker determines that the schema is compatible and stores the new schema as the current version (with a new version number). |
+| Scenario | Result |
+|-----------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| <li>No schema exists for the topic. </li> | (1) The producer is created with the given schema. <br /> (2) The schema is transmitted to the broker and stored since there is no existing schema. <br /> (3) Any consumer created using the same schema or topic can consume messages from the `sensor-data` topic. |
+| <li>A schema already exists. </li><li>The producer connects using the same schema that is already stored. </li> | (1) The schema is transmitted to the broker.<br /> (2) The broker determines that the schema is compatible. <br /> (3) The broker attempts to store the schema in [BookKeeper](concepts-architecture-overview.md#persistent-storage) but then determines that it's already stored, so it is used to tag produced messages. |
+| <li>A schema already exists. </li><li>The producer connects using a new schema that is compatible. </li> | (1) The schema is transmitted to the broker. <br /> (2) The broker determines that the schema is compatible and stores the new schema as the current version (with a new version number). |
-## Schema AutoUpdate
+### Schema compatibility check
-If a schema passes the schema compatibility check, Pulsar producer automatically updates this schema to the topic it produces by default.
+The purpose of schema compatibility check is to ensure that existing consumers can process the introduced messages.
-### AutoUpdate for producer
+When receiving a `SchemaInfo` from producers, brokers recognize the schema type and deploy the schema compatibility checker ([`schemaRegistryCompatibilityCheckers`](https://github.com/apache/pulsar/blob/bf194b557c48e2d3246e44f1fc28876932d8ecb8/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java)) for that schema type to check if the `SchemaInfo` is compatible with the schema of the topic by applying the configured compatibility check strategy.
-For a producer, the `AutoUpdate` happens in the following cases:
+The default value of `schemaRegistryCompatibilityCheckers` in the `conf/broker.conf` file is as follows.
+
+```properties
+schemaRegistryCompatibilityCheckers=org.apache.pulsar.broker.service.schema.JsonSchemaCompatibilityCheck,org.apache.pulsar.broker.service.schema.AvroSchemaCompatibilityCheck,org.apache.pulsar.broker.service.schema.ProtobufNativeSchemaCompatibilityCheck
+```
-* If a **topic doesn’t have a schema**, Pulsar registers a schema automatically.
+Each schema type corresponds to one instance of the schema compatibility checker. Avro, JSON, and Protobuf schemas have their own compatibility checkers, while all the other schema types share the default compatibility checker that disables the schema evolution.
-* If a **topic has a schema**:
+#### Schema compatibility check strategy
- * If a **producer doesn’t carry a schema**:
+Suppose that you have a topic containing three schemas (V1, V2, and V3). V1 is the oldest and V3 is the latest. The following table outlines 8 schema compatibility strategies and how it works.
- * If `isSchemaValidationEnforced` or `schemaValidationEnforced` is **disabled** in the namespace to which the topic belongs, the producer is allowed to connect to the topic and produce data.
-
- * If `isSchemaValidationEnforced` or `schemaValidationEnforced` is **enabled** in the namespace to which the topic belongs, the producer is rejected and disconnected.
+| Compatibility check strategy | Definition | Changes allowed | Check against which schema |
+| --- | --- | --- | --- |
+| `ALWAYS_COMPATIBLE` | Disable schema compatibility check. | All changes are allowed | All previous versions |
+| `ALWAYS_INCOMPATIBLE` | Disable schema evolution, that is, any schema change is rejected. | No change is allowed | N/A |
+| `BACKWARD` | Consumers using schema V3 can process data written by producers using the **last schema version** V2. | <li>Add optional fields </li><li>Delete fields </li> | Latest version |
+| `BACKWARD_TRANSITIVE` | Consumers using schema V3 can process data written by producers using **all previous schema versions** V2 and V1. | <li>Add optional fields </li><li>Delete fields </li> | All previous versions |
+| `FORWARD` | Consumers using the **last schema version** V2 can process data written by producers using a new schema V3, even though they may not be able to use the full capabilities of the new schema. | <li>Add fields </li><li>Delete optional fields </li> | Latest version |
+| `FORWARD_TRANSITIVE` | Consumers using **all previous schema versions** V2 or V1 can process data written by producers using a new schema V3. | <li>Add fields </li><li>Delete optional fields </li> | All previous versions |
+| `FULL` | Schemas are both backward and forward compatible. <li>Consumers using the last schema V2 can process data written by producers using the new schema V3. </li><li>Consumers using the new schema V3 can process data written by producers using the last schema V2.</li> | <li>Modify optional fields </li> | Latest version |
+| `FULL_TRANSITIVE` | Backward and forward compatible among schema V3, V2, and V1. <li>Consumers using the schema V3 can process data written by producers using schema V2 and V1. </li><li>Consumers using the schema V2 or V1 can process data written by producers using the schema V3.</li> | <li>Modify optional fields </li> | All previous versions |
- * If a **producer carries a schema**:
-
- A broker performs the compatibility check based on the configured compatibility check strategy of the namespace to which the topic belongs.
-
- * If the schema is registered, a producer is connected to a broker.
-
- * If the schema is not registered:
-
- * If `isAllowAutoUpdateSchema` sets to **false**, the producer is rejected to connect to a broker.
+:::tip
+
+* The default schema compatibility check strategy varies depending on schema types.
+ * For Avro and JSON, the default one is `FULL`.
+ * For others, the default one is `ALWAYS_INCOMPATIBLE`.
+* For more instructions, see [Admin API](admin-api-schemas.md#set-schema-compatibility-check-strategy).
+
+:::
+
+### Schema AutoUpdate
+
+By default, schema `AutoUpdate` is enabled. When a schema passes the schema compatibility check, the producer automatically updates this schema to the topic it produces.
+
+#### Producer side
+
+For a producer, the `AutoUpdate` happens in the following cases:
+
+* If a **topic doesn’t have a schema** (meaning the data is in raw bytes), Pulsar registers the schema automatically.
+
+* If a **topic has a schema** and the **producer doesn’t carry any schema** (meaning it produces raw bytes):
+
+ * If [schema validation](#schema-validation) is **disabled** (`schemaValidationEnforced`=`false`) in the namespace that the topic belongs to, the producer is allowed to connect to the topic and produce data.
- * If `isAllowAutoUpdateSchema` sets to **true**:
-
- * If the schema passes the compatibility check, then the broker registers a new schema automatically for the topic and the producer is connected.
-
- * If the schema does not pass the compatibility check, then the broker does not register a schema and the producer is rejected to connect to a broker.
+ * Otherwise, the producer is rejected.
-
+ * If a **topic has a schema** and the **producer carries a schema**, see [How schema works on producer side](schema-overview.md#producer-side) for more information.
-### AutoUpdate for consumer
+#### Consumer side
For a consumer, the `AutoUpdate` happens in the following cases:
-* If a **consumer connects to a topic without a schema** (which means the consumer receiving raw bytes), the consumer can connect to the topic successfully without doing any compatibility check.
+* If a consumer connects to a topic **without a schema** (meaning it consumes raw bytes), the consumer can connect to the topic successfully without doing any compatibility check.
-* If a **consumer connects to a topic with a schema**.
+* If a consumer connects to a topic **with a schema**, see [How schema works on consumer side](schema-overview.md#consumer-side) for more information.
- * If a topic does not have all of them (a schema/data/a local consumer and a local producer):
-
- * If `isAllowAutoUpdateSchema` sets to **true**, then the consumer registers a schema and it is connected to a broker.
-
- * If `isAllowAutoUpdateSchema` sets to **false**, then the consumer is rejected to connect to a broker.
-
- * If a topic has one of them (a schema/data/a local consumer and a local producer), then the schema compatibility check is performed.
-
- * If the schema passes the compatibility check, then the consumer is connected to the broker.
-
- * If the schema does not pass the compatibility check, then the consumer is rejected to connect to the broker.
-
-
\ No newline at end of file
+### Order of upgrading clients
+
+To adapt to schema evolution and auto-update, you need to upgrade your client applications accordingly. The upgrade order may vary depending on the configured [schema compatibility check strategy](#schema-compatibility-check-strategy).
+
+The following table outlines the mapping between the schema compatibility check strategy and the upgrade order of clients.
+
+| Compatibility check strategy | Upgrade order | Description |
+| --- | --- |---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `ALWAYS_COMPATIBLE` | Any order | The compatibility check is disabled. Consequently, you can upgrade the producers and consumers in **any order**. |
+| `ALWAYS_INCOMPATIBLE` | N/A | The schema evolution is disabled. |
+| <li>`BACKWARD` </li><li>`BACKWARD_TRANSITIVE` </li> | Consumer first | There is no guarantee that consumers using the old schema can read data produced using the new schema. Consequently, **upgrade all consumers first**, and then start producing new data. |
+| <li>`FORWARD` </li><li>`FORWARD_TRANSITIVE` </li> | Producer first | There is no guarantee that consumers using the new schema can read data produced using the old schema. Consequently, **upgrade all producers first**<li>to use the new schema and ensure that the data already produced using the old schemas are not available to consumers, and then upgrades the consumers. </li> |
Review Comment:
```suggestion
| <li>`FORWARD` </li><li>`FORWARD_TRANSITIVE` </li> | Producer first | There is no guarantee that consumers using the new schema can read data produced using the old schema. Consequently, **upgrade all producers first** to use the new schema and ensure that the data already produced using the old schemas are not available to consumers, and then upgrades the consumers. |
```
there is a redundant list tag
##########
site2/docs/schema-get-started.md:
##########
@@ -11,80 +11,283 @@ import TabItem from '@theme/TabItem';
````
-This hands-on tutorial provides instructions and examples on how to construct and customize schemas.
+This hands-on tutorial provides instructions and examples on how to construct schemas. For the instructions on administrative tasks, see [Manage schema](admin-api-schemas.md).
-## Construct a string schema
+## Construct a schema
-This example demonstrates how to construct a [string schema](schema-understand.md#primitive-type) and use it to produce and consume messages in Java.
+- [Construct a bytes schema](#bytes)
+- [Construct a string schema](#string)
+- [Construct a key/value schema](#keyvalue)
+- [Construct an Avro schema](#avro)
+- [Construct a JSON schema](#json)
+- [Construct a Protobuf schema](#protobuf)
+- [Construct a ProtobufNative schema](#protobufnative)
+- [Construct a native Avro schema](#nativeavro)
+- [Construct an AUTO_PRODUCE schema](#auto_produce)
+- [Construct an AUTO_CONSUME schema](#auto_consume)
-1. Create a producer with a string schema and send messages.
+### `bytes`
+
+This example demonstrates how to construct a [bytes schema](schema-understand.md#primitive-type) using language-specific clients and use it to produce and consume messages.
+
+````mdx-code-block
+<Tabs groupId="api-choice"
+ defaultValue="Java"
+ values={[{"label":"Java","value":"Java"},{"label":"C++","value":"C++"},{"label":"Python","value":"Python"},{"label":"Go","value":"Go"}]}>
+
+<TabItem value="Java">
+
+```java
+ .topic("my-topic")
+ .create();
+Consumer<byte[]> consumer = pulsarClient.newConsumer(Schema.BYTES)
+ .topic("my-topic")
+ .subscriptionName("my-sub")
+ .subscribe();
+
+producer.newMessage().value("message".getBytes()).send();
+
+Message<byte[]> message = consumer.receive(5, TimeUnit.SECONDS);
+```
+
+</TabItem>
+<TabItem value="C++">
+
+```cpp
+SchemaInfo schemaInfo = SchemaInfo(SchemaType::BYTES, "Bytes", "");
+Producer producer;
+client.createProducer("topic-bytes", ProducerConfiguration().setSchema(schemaInfo), producer);
+std::array<char, 1024> buffer;
+producer.send(MessageBuilder().setContent(buffer.data(), buffer.size()).build());
+Consumer consumer;
+res = client.subscribe("topic-bytes", "my-sub", ConsumerConfiguration().setSchema(schemaInfo), consumer);
+Message msg;
+consumer.receive(msg, 3000);
+```
+
+</TabItem>
+<TabItem value="Python">
+
+```python
+producer = client.create_producer(
+'bytes-schema-topic',
+schema=BytesSchema())
+producer.send(b"Hello")
+
+consumer = client.subscribe(
+'bytes-schema-topic',
+ 'sub',
+ schema=BytesSchema())
+msg = consumer.receive()
+data = msg.value()
+```
Review Comment:
Indent style can be improved here.
##########
site2/docs/schema-get-started.md:
##########
@@ -11,80 +11,283 @@ import TabItem from '@theme/TabItem';
````
-This hands-on tutorial provides instructions and examples on how to construct and customize schemas.
+This hands-on tutorial provides instructions and examples on how to construct schemas. For the instructions on administrative tasks, see [Manage schema](admin-api-schemas.md).
-## Construct a string schema
+## Construct a schema
-This example demonstrates how to construct a [string schema](schema-understand.md#primitive-type) and use it to produce and consume messages in Java.
+- [Construct a bytes schema](#bytes)
+- [Construct a string schema](#string)
+- [Construct a key/value schema](#keyvalue)
+- [Construct an Avro schema](#avro)
+- [Construct a JSON schema](#json)
+- [Construct a Protobuf schema](#protobuf)
+- [Construct a ProtobufNative schema](#protobufnative)
+- [Construct a native Avro schema](#nativeavro)
+- [Construct an AUTO_PRODUCE schema](#auto_produce)
+- [Construct an AUTO_CONSUME schema](#auto_consume)
-1. Create a producer with a string schema and send messages.
+### `bytes`
+
+This example demonstrates how to construct a [bytes schema](schema-understand.md#primitive-type) using language-specific clients and use it to produce and consume messages.
+
+````mdx-code-block
+<Tabs groupId="api-choice"
+ defaultValue="Java"
+ values={[{"label":"Java","value":"Java"},{"label":"C++","value":"C++"},{"label":"Python","value":"Python"},{"label":"Go","value":"Go"}]}>
+
+<TabItem value="Java">
+
+```java
+ .topic("my-topic")
+ .create();
+Consumer<byte[]> consumer = pulsarClient.newConsumer(Schema.BYTES)
+ .topic("my-topic")
+ .subscriptionName("my-sub")
+ .subscribe();
+
+producer.newMessage().value("message".getBytes()).send();
+
+Message<byte[]> message = consumer.receive(5, TimeUnit.SECONDS);
+```
+
+</TabItem>
+<TabItem value="C++">
+
+```cpp
+SchemaInfo schemaInfo = SchemaInfo(SchemaType::BYTES, "Bytes", "");
+Producer producer;
+client.createProducer("topic-bytes", ProducerConfiguration().setSchema(schemaInfo), producer);
+std::array<char, 1024> buffer;
+producer.send(MessageBuilder().setContent(buffer.data(), buffer.size()).build());
+Consumer consumer;
+res = client.subscribe("topic-bytes", "my-sub", ConsumerConfiguration().setSchema(schemaInfo), consumer);
+Message msg;
+consumer.receive(msg, 3000);
+```
+
+</TabItem>
+<TabItem value="Python">
+
+```python
+producer = client.create_producer(
+'bytes-schema-topic',
+schema=BytesSchema())
+producer.send(b"Hello")
+
+consumer = client.subscribe(
+'bytes-schema-topic',
+ 'sub',
+ schema=BytesSchema())
+msg = consumer.receive()
+data = msg.value()
+```
+
+</TabItem>
+<TabItem value="Go">
+
+```go
+producer, err := client.CreateProducer(pulsar.ProducerOptions{
+ Topic: "my-topic",
+ Schema: pulsar.NewBytesSchema(nil),
+})
+id, err := producer.Send(context.Background(), &pulsar.ProducerMessage{
+ Value: []byte("message"),
+})
+
+consumer, err := client.Subscribe(pulsar.ConsumerOptions{
+ Topic: "my-topic",
+ Schema: pulsar.NewBytesSchema(nil),
+ SubscriptionName: "my-sub",
+ Type: pulsar.Exclusive,
+})
+```
+
+</TabItem>
+</Tabs>
+````
+
+### `string`
+
+This example demonstrates how to construct a [string schema](schema-understand.md#primitive-type) using language-specific clients and use it to produce and consume messages.
+
+````mdx-code-block
+<Tabs groupId="api-choice"
+ defaultValue="Java"
+ values={[{"label":"Java","value":"Java"},{"label":"C++","value":"C++"},{"label":"Python","value":"Python"},{"label":"Go","value":"Go"}]}>
+
+<TabItem value="Java">
```java
Producer<String> producer = client.newProducer(Schema.STRING).create();
producer.newMessage().value("Hello Pulsar!").send();
- ```
-2. Create a consumer with a string schema and receive messages.
-
- ```java
Consumer<String> consumer = client.newConsumer(Schema.STRING).subscribe();
- consumer.receive();
+ Message<String> message = consumer.receive();
```
-## Construct a key/value schema
+</TabItem>
+<TabItem value="C++">
+
+```cpp
+SchemaInfo schemaInfo = SchemaInfo(SchemaType::STRING, "String", "");
+Producer producer;
+client.createProducer("topic-string", ProducerConfiguration().setSchema(schemaInfo), producer);
+producer.send(MessageBuilder().setContent("message").build());
+
+Consumer consumer;
+client.subscribe("topic-string", "my-sub", ConsumerConfiguration().setSchema(schemaInfo), consumer);
+Message msg;
+consumer.receive(msg, 3000);
+```
-This example shows how to construct a [key/value schema](schema-understand.md#keyvalue-schema) and use it to produce and consume messages in Java.
+</TabItem>
+<TabItem value="Python">
+
+```python
+producer = client.create_producer(
+ 'string-schema-topic',
+ schema=StringSchema())
+producer.send("Hello")
+
+consumer = client.subscribe(
+ 'string-schema-topic',
+ 'sub',
+ schema=StringSchema())
+msg = consumer.receive()
+str = msg.value()
+```
+
+</TabItem>
+<TabItem value="Go">
+
+```go
+producer, err := client.CreateProducer(pulsar.ProducerOptions{
+ Topic: "my-topic",
+ Schema: pulsar.NewStringSchema(nil),
+})
+id, err := producer.Send(context.Background(), &pulsar.ProducerMessage{
+ Value: "message",
+})
+
+consumer, err := client.Subscribe(pulsar.ConsumerOptions{
+ Topic: "my-topic",
+ Schema: pulsar.NewStringSchema(nil),
+ SubscriptionName: "my-sub",
+ Type: pulsar.Exclusive,
+})
+msg, err := consumer.Receive(context.Background())
Review Comment:
Same suggestion as above.
##########
site2/docs/schema-get-started.md:
##########
@@ -178,10 +396,10 @@ You can define the `schemaDefinition` to generate a `struct` schema.
```java
SchemaDefinition<User> schemaDefinition = SchemaDefinition.<User>builder().withPojo(User.class).build();
Producer<User> producer = client.newProducer(Schema.AVRO(schemaDefinition)).create();
- producer.newMessage().value(User.builder().name("pulsar-user").age(1).build()).send();
+ producer.newMessage().value(new User (“pulsar-user”, 1)).send();
Review Comment:
```suggestion
producer.newMessage().value(new User("pulsar-user", 1)).send();
```
Full-width quotes would be only used in Chinese.
##########
site2/docs/schema-get-started.md:
##########
@@ -164,12 +380,14 @@ You can define the `schemaDefinition` to generate a `struct` schema.
1. Create the _User_ class to define the messages sent to Pulsar topics.
```java
- @Builder
- @AllArgsConstructor
- @NoArgsConstructor
public static class User {
- String name;
- int age;
+ public String name;
+ public int age;
+ public User(String name, int age) {
+ this.name = name;
+ this.age = age
Review Comment:
Suggest all indents to be the same (4 spaces).
##########
site2/docs/schema-get-started.md:
##########
@@ -228,88 +452,349 @@ Producer<SensorReading> producer = client.newProducer(AvroSchema.of(SensorReadin
.create();
```
-### Avro-based schema using Java
-
-The following schema formats are currently available for Java:
+</TabItem>
+<TabItem value="C++">
-* No schema or the byte array schema (which can be applied using `Schema.BYTES`):
+ ```cpp
+ // Send messages
+ static const std::string exampleSchema =
+ "{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\","
+ "\"fields\":[{\"name\":\"a\",\"type\":\"int\"},{\"name\":\"b\",\"type\":\"int\"}]}";
+ Producer producer;
+ ProducerConfiguration producerConf;
+ producerConf.setSchema(SchemaInfo(AVRO, "Avro", exampleSchema));
+ client.createProducer("topic-avro", producerConf, producer);
- ```java
- Producer<byte[]> bytesProducer = client.newProducer(Schema.BYTES)
- .topic("some-raw-bytes-topic")
- .create();
+ // Receive messages
+ static const std::string exampleSchema =
+ "{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\","
+ "\"fields\":[{\"name\":\"a\",\"type\":\"int\"},{\"name\":\"b\",\"type\":\"int\"}]}";
+ ConsumerConfiguration consumerConf;
+ Consumer consumer;
+ consumerConf.setSchema(SchemaInfo(AVRO, "Avro", exampleSchema));
+ client.subscribe("topic-avro", "sub-2", consumerConf, consumer)
```
- Or, equivalently:
+</TabItem>
+<TabItem value="Python">
- ```java
- Producer<byte[]> bytesProducer = client.newProducer()
- .topic("some-raw-bytes-topic")
- .create();
- ```
+You can declare an `AvroSchema` using Python through one of the following methods.
-* `String` for normal UTF-8-encoded string data. Apply the schema using `Schema.STRING`:
+**Method 1: Record**
- ```java
- Producer<String> stringProducer = client.newProducer(Schema.STRING)
- .topic("some-string-topic")
- .create();
- ```
+Declare an `AvroSchema` by passing a class that inherits from `pulsar.schema.Record` and defines the fields as class variables.
-* Create JSON schemas for POJOs using `Schema.JSON`. The following is an example.
+```python
+class Example(Record):
+ a = Integer()
+ b = Integer()
- ```java
- Producer<MyPojo> pojoProducer = client.newProducer(Schema.JSON(MyPojo.class))
- .topic("some-pojo-topic")
- .create();
- ```
+producer = client.create_producer(
+'avro-schema-topic',
+schema=AvroSchema(Example))
+r = Example(a=1, b=2)
+producer.send(r)
-* Generate Protobuf schemas using `Schema.PROTOBUF`. The following example shows how to create the Protobuf schema and use it to instantiate a new producer:
+consumer = client.subscribe(
+'avro-schema-topic',
+ 'sub',
+ schema=AvroSchema(Example))
+msg = consumer.receive()
+e = msg.value()
Review Comment:
Same indent style problem as above.
##########
site2/docs/schema-get-started.md:
##########
@@ -11,80 +11,283 @@ import TabItem from '@theme/TabItem';
````
-This hands-on tutorial provides instructions and examples on how to construct and customize schemas.
+This hands-on tutorial provides instructions and examples on how to construct schemas. For the instructions on administrative tasks, see [Manage schema](admin-api-schemas.md).
-## Construct a string schema
+## Construct a schema
-This example demonstrates how to construct a [string schema](schema-understand.md#primitive-type) and use it to produce and consume messages in Java.
+- [Construct a bytes schema](#bytes)
+- [Construct a string schema](#string)
+- [Construct a key/value schema](#keyvalue)
+- [Construct an Avro schema](#avro)
+- [Construct a JSON schema](#json)
+- [Construct a Protobuf schema](#protobuf)
+- [Construct a ProtobufNative schema](#protobufnative)
+- [Construct a native Avro schema](#nativeavro)
+- [Construct an AUTO_PRODUCE schema](#auto_produce)
+- [Construct an AUTO_CONSUME schema](#auto_consume)
-1. Create a producer with a string schema and send messages.
+### `bytes`
+
+This example demonstrates how to construct a [bytes schema](schema-understand.md#primitive-type) using language-specific clients and use it to produce and consume messages.
+
+````mdx-code-block
+<Tabs groupId="api-choice"
+ defaultValue="Java"
+ values={[{"label":"Java","value":"Java"},{"label":"C++","value":"C++"},{"label":"Python","value":"Python"},{"label":"Go","value":"Go"}]}>
+
+<TabItem value="Java">
+
+```java
+ .topic("my-topic")
+ .create();
+Consumer<byte[]> consumer = pulsarClient.newConsumer(Schema.BYTES)
+ .topic("my-topic")
+ .subscriptionName("my-sub")
+ .subscribe();
+
+producer.newMessage().value("message".getBytes()).send();
+
+Message<byte[]> message = consumer.receive(5, TimeUnit.SECONDS);
+```
+
+</TabItem>
+<TabItem value="C++">
+
+```cpp
+SchemaInfo schemaInfo = SchemaInfo(SchemaType::BYTES, "Bytes", "");
+Producer producer;
+client.createProducer("topic-bytes", ProducerConfiguration().setSchema(schemaInfo), producer);
+std::array<char, 1024> buffer;
+producer.send(MessageBuilder().setContent(buffer.data(), buffer.size()).build());
+Consumer consumer;
+res = client.subscribe("topic-bytes", "my-sub", ConsumerConfiguration().setSchema(schemaInfo), consumer);
+Message msg;
+consumer.receive(msg, 3000);
+```
+
+</TabItem>
+<TabItem value="Python">
+
+```python
+producer = client.create_producer(
+'bytes-schema-topic',
+schema=BytesSchema())
+producer.send(b"Hello")
+
+consumer = client.subscribe(
+'bytes-schema-topic',
+ 'sub',
+ schema=BytesSchema())
+msg = consumer.receive()
+data = msg.value()
+```
+
+</TabItem>
+<TabItem value="Go">
+
+```go
+producer, err := client.CreateProducer(pulsar.ProducerOptions{
+ Topic: "my-topic",
+ Schema: pulsar.NewBytesSchema(nil),
+})
+id, err := producer.Send(context.Background(), &pulsar.ProducerMessage{
+ Value: []byte("message"),
+})
+
+consumer, err := client.Subscribe(pulsar.ConsumerOptions{
+ Topic: "my-topic",
+ Schema: pulsar.NewBytesSchema(nil),
+ SubscriptionName: "my-sub",
+ Type: pulsar.Exclusive,
+})
+```
Review Comment:
Suggest 4 space or a tab as indent for go code.
##########
site2/docs/schema-get-started.md:
##########
@@ -228,88 +452,349 @@ Producer<SensorReading> producer = client.newProducer(AvroSchema.of(SensorReadin
.create();
```
-### Avro-based schema using Java
-
-The following schema formats are currently available for Java:
+</TabItem>
+<TabItem value="C++">
-* No schema or the byte array schema (which can be applied using `Schema.BYTES`):
+ ```cpp
+ // Send messages
+ static const std::string exampleSchema =
+ "{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\","
+ "\"fields\":[{\"name\":\"a\",\"type\":\"int\"},{\"name\":\"b\",\"type\":\"int\"}]}";
+ Producer producer;
+ ProducerConfiguration producerConf;
+ producerConf.setSchema(SchemaInfo(AVRO, "Avro", exampleSchema));
+ client.createProducer("topic-avro", producerConf, producer);
- ```java
- Producer<byte[]> bytesProducer = client.newProducer(Schema.BYTES)
- .topic("some-raw-bytes-topic")
- .create();
+ // Receive messages
+ static const std::string exampleSchema =
+ "{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\","
+ "\"fields\":[{\"name\":\"a\",\"type\":\"int\"},{\"name\":\"b\",\"type\":\"int\"}]}";
+ ConsumerConfiguration consumerConf;
+ Consumer consumer;
+ consumerConf.setSchema(SchemaInfo(AVRO, "Avro", exampleSchema));
+ client.subscribe("topic-avro", "sub-2", consumerConf, consumer)
```
- Or, equivalently:
+</TabItem>
+<TabItem value="Python">
- ```java
- Producer<byte[]> bytesProducer = client.newProducer()
- .topic("some-raw-bytes-topic")
- .create();
- ```
+You can declare an `AvroSchema` using Python through one of the following methods.
-* `String` for normal UTF-8-encoded string data. Apply the schema using `Schema.STRING`:
+**Method 1: Record**
- ```java
- Producer<String> stringProducer = client.newProducer(Schema.STRING)
- .topic("some-string-topic")
- .create();
- ```
+Declare an `AvroSchema` by passing a class that inherits from `pulsar.schema.Record` and defines the fields as class variables.
-* Create JSON schemas for POJOs using `Schema.JSON`. The following is an example.
+```python
+class Example(Record):
+ a = Integer()
+ b = Integer()
- ```java
- Producer<MyPojo> pojoProducer = client.newProducer(Schema.JSON(MyPojo.class))
- .topic("some-pojo-topic")
- .create();
- ```
+producer = client.create_producer(
+'avro-schema-topic',
+schema=AvroSchema(Example))
+r = Example(a=1, b=2)
+producer.send(r)
-* Generate Protobuf schemas using `Schema.PROTOBUF`. The following example shows how to create the Protobuf schema and use it to instantiate a new producer:
+consumer = client.subscribe(
+'avro-schema-topic',
+ 'sub',
+ schema=AvroSchema(Example))
+msg = consumer.receive()
+e = msg.value()
+```
- ```java
- Producer<MyProtobuf> protobufProducer = client.newProducer(Schema.PROTOBUF(MyProtobuf.class))
- .topic("some-protobuf-topic")
- .create();
- ```
+**Method 2: JSON definition**
+
+1. Declare an `AvroSchema` using JSON. In this case, Avro schemas are defined using JSON.
+
+ Below is an example of `AvroSchema` defined using a JSON file (`company.avsc`).
+
+ ```json
+ {
+ "doc": "this is doc",
+ "namespace": "example.avro",
+ "type": "record",
+ "name": "Company",
+ "fields": [
+ {"name": "name", "type": ["null", "string"]},
+ {"name": "address", "type": ["null", "string"]},
+ {"name": "employees", "type": ["null", {"type": "array", "items": {
+ "type": "record",
+ "name": "Employee",
+ "fields": [
+ {"name": "name", "type": ["null", "string"]},
+ {"name": "age", "type": ["null", "int"]}
+ ]
+ }}]},
+ {"name": "labels", "type": ["null", {"type": "map", "values": "string"}]}
+ ]
+ }
+ ```
-* Define Avro schemas with `Schema.AVRO`. The following code snippet demonstrates how to create and use Avro schema.
+2. Load a schema definition from a file by using [`avro.schema`](https://avro.apache.org/docs/current/getting-started-python/) or [`fastavro.schema`](https://fastavro.readthedocs.io/en/latest/schema.html#fastavro._schema_py.load_schema).
+
+ If you use the [JSON definition](#method-2-json-definition) method to declare an `AvroSchema`, you need to:
+ - Use [Python dict](https://developers.google.com/edu/python/dict-files) to produce and consume messages, which is different from using the [Record](#method-1-record) method.
+ - Set the value of the `_record_cls` parameter to `None` when generating an `AvroSchema` object.
+
+ **Example**
+
+ ```python
+ from fastavro.schema import load_schema
+ from pulsar.schema import *
+ schema_definition = load_schema("examples/company.avsc")
+ avro_schema = AvroSchema(None, schema_definition=schema_definition)
+ producer = client.create_producer(
+ topic=topic,
+ schema=avro_schema)
+ consumer = client.subscribe(topic, 'test', schema=avro_schema)
+ company = {
+ "name": "company-name" + str(i),
+ "address": 'xxx road xxx street ' + str(i),
+ "employees": [
+ {"name": "user" + str(i), "age": 20 + i},
+ {"name": "user" + str(i), "age": 30 + i},
+ {"name": "user" + str(i), "age": 35 + i},
+ ],
+ "labels": {
+ "industry": "software" + str(i),
+ "scale": ">100",
+ "funds": "1000000.0"
+ }
+ }
+ producer.send(company)
+ msg = consumer.receive()
+ # Users could get a dict object by `value()` method.
+ msg.value()
+ ```
- ```java
- Producer<MyAvro> avroProducer = client.newProducer(Schema.AVRO(MyAvro.class))
- .topic("some-avro-topic")
- .create();
- ```
+</TabItem>
+<TabItem value="Go">
+Suppose you have an `avroExampleStruct` class as follows, and you'd like to transmit it over a Pulsar topic.
-### Avro schema using C++
+```go
+ type avroExampleStruct struct {
+ ID int
+ Name string
+}
+```
Review Comment:
Same indent style problem as above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] momo-jun commented on pull request #18434: [refactor][doc] Improve schema docs
Posted by GitBox <gi...@apache.org>.
momo-jun commented on PR #18434:
URL: https://github.com/apache/pulsar/pull/18434#issuecomment-1347707311
@RobertIndie @labuladong Thanks very much for your concrete review. Comments have been addressed. PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org