You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2022/05/17 03:44:37 UTC
[GitHub] [apisix] bzp2010 opened a new pull request, #7059: docs: add pubsub develop example for kafka
bzp2010 opened a new pull request, #7059:
URL: https://github.com/apache/apisix/pull/7059
### Description
Add pubsub develop documentation example for Apache Kafka.
### Checklist
- [x] I have explained the need for this PR and the problem it solves
- [x] I have explained the changes or the new features added to this PR
- [x] I have added tests corresponding to this change
- [x] I have updated the documentation to reflect this change
- [x] I have verified that this change is backward compatible (If not, please discuss on the [APISIX mailing list](https://github.com/apache/apisix/tree/master#community) first)
<!--
Note
1. Mark the PR as draft until it's ready to be reviewed.
2. Always add/update tests for any changes unless you have a good reason.
3. Always update the documentation to reflect the changes made in the PR.
4. Make a new commit to resolve conversations instead of `push -f`.
5. To resolve merge conflicts, merge master instead of rebasing.
6. Use "request review" to notify the reviewer after making changes.
7. Only a reviewer can mark a conversation as resolved.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander merged pull request #7059: docs: add pubsub develop example for kafka
Posted by GitBox <gi...@apache.org>.
spacewander merged PR #7059:
URL: https://github.com/apache/apisix/pull/7059
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a diff in pull request #7059: docs: add pubsub develop example for kafka
Posted by GitBox <gi...@apache.org>.
spacewander commented on code in PR #7059:
URL: https://github.com/apache/apisix/pull/7059#discussion_r874478724
##########
docs/en/latest/pubsub.md:
##########
@@ -58,6 +58,192 @@ Apache APISIX implement an extensible pubsub module, which is responsible for st
- Implement the required message system instruction processing functions
- Optional: Create plugins to support advanced configurations of this messaging system
-### Example
+### Example of Apache Kafka
-TODO, an example will be added later to point out how to support other messaging systems.
+#### Add new commands and response body definitions to `pubsub.proto`
+
+The core of the protocol definition in `pubsub.proto` is the two parts `PubSubReq` and `PubSubResp`.
+
+First, create the `CmdKafkaFetch` command and add the required parameters. Then, register this command in the list of commands for `req` in `PubSubReq`, which is named `cmd_kafka_fetch`.
+
+```protobuf
+message CmdKafkaFetch {
+ string topic = 1;
+ int32 partition = 2;
+ int64 offset = 3;
+}
+
+message PubSubReq {
+ int64 sequence = 1;
+ oneof req {
+ CmdKafkaFetch cmd_kafka_fetch = 31;
Review Comment:
Could you update the code?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] bzp2010 commented on a diff in pull request #7059: docs: add pubsub develop example for kafka
Posted by GitBox <gi...@apache.org>.
bzp2010 commented on code in PR #7059:
URL: https://github.com/apache/apisix/pull/7059#discussion_r875446914
##########
docs/en/latest/pubsub.md:
##########
@@ -58,6 +58,85 @@ Apache APISIX implement an extensible pubsub module, which is responsible for st
- Implement the required message system instruction processing functions
- Optional: Create plugins to support advanced configurations of this messaging system
-### Example
+### Example of Apache Kafka
-TODO, an example will be added later to point out how to support other messaging systems.
+#### Add new commands and response body definitions to `pubsub.proto`
+
+The core of the protocol definition in `pubsub.proto` is the two parts `PubSubReq` and `PubSubResp`.
+
+First, create the `CmdKafkaFetch` command and add the required parameters. Then, register this command in the list of commands for `req` in `PubSubReq`, which is named `cmd_kafka_fetch`.
+
+Then create the corresponding response body `KafkaFetchResp` and register it in the `resp` of `PubSubResp`, named `kafka_fetch_resp`.
+
+The protocol definition [pubsub.proto](https://github.com/apache/apisix/blob/master/apisix/include/apisix/model/pubsub.proto).
+
+#### Add a new option to the `scheme` configuration item in upstream
+
+Add a new option `kafka` to the `scheme` field enumeration in the `upstream` of `apisix/schema_def.lua`.
+
+The schema definition [schema_def.lua](https://github.com/apache/apisix/blob/master/apisix/schema_def.lua).
+
+#### Add a new `scheme` judgment branch to `http_access_phase`
+
+Add a `scheme` judgment branch to the `http_access_phase` function in `apisix/init.lua` to support the processing of `kafka` type upstreams. Because of Apache Kafka has its own clustering and partition scheme, we do not need to use the Apache APISIX built-in load balancing algorithm, so we intercept and take over the processing flow before selecting the upstream node, here using the `kafka_access_phase` function.
+
+The APISIX init file [init.lua](https://github.com/apache/apisix/blob/master/apisix/init.lua).
+
+#### Implement the required message system commands processing functions
+
+First, create an instance of the `pubsub` module, which is provided in the `core` package.
+
+Then, an instance of the Apache Kafka client is created, and this code is omitted here.
Review Comment:
fixed
##########
docs/en/latest/pubsub.md:
##########
@@ -58,6 +58,85 @@ Apache APISIX implement an extensible pubsub module, which is responsible for st
- Implement the required message system instruction processing functions
- Optional: Create plugins to support advanced configurations of this messaging system
-### Example
+### Example of Apache Kafka
-TODO, an example will be added later to point out how to support other messaging systems.
+#### Add new commands and response body definitions to `pubsub.proto`
+
+The core of the protocol definition in `pubsub.proto` is the two parts `PubSubReq` and `PubSubResp`.
+
+First, create the `CmdKafkaFetch` command and add the required parameters. Then, register this command in the list of commands for `req` in `PubSubReq`, which is named `cmd_kafka_fetch`.
+
+Then create the corresponding response body `KafkaFetchResp` and register it in the `resp` of `PubSubResp`, named `kafka_fetch_resp`.
+
+The protocol definition [pubsub.proto](https://github.com/apache/apisix/blob/master/apisix/include/apisix/model/pubsub.proto).
+
+#### Add a new option to the `scheme` configuration item in upstream
+
+Add a new option `kafka` to the `scheme` field enumeration in the `upstream` of `apisix/schema_def.lua`.
+
+The schema definition [schema_def.lua](https://github.com/apache/apisix/blob/master/apisix/schema_def.lua).
+
+#### Add a new `scheme` judgment branch to `http_access_phase`
+
+Add a `scheme` judgment branch to the `http_access_phase` function in `apisix/init.lua` to support the processing of `kafka` type upstreams. Because of Apache Kafka has its own clustering and partition scheme, we do not need to use the Apache APISIX built-in load balancing algorithm, so we intercept and take over the processing flow before selecting the upstream node, here using the `kafka_access_phase` function.
Review Comment:
fixed
##########
docs/en/latest/pubsub.md:
##########
@@ -58,6 +58,85 @@ Apache APISIX implement an extensible pubsub module, which is responsible for st
- Implement the required message system instruction processing functions
- Optional: Create plugins to support advanced configurations of this messaging system
-### Example
+### Example of Apache Kafka
-TODO, an example will be added later to point out how to support other messaging systems.
+#### Add new commands and response body definitions to `pubsub.proto`
+
+The core of the protocol definition in `pubsub.proto` is the two parts `PubSubReq` and `PubSubResp`.
+
+First, create the `CmdKafkaFetch` command and add the required parameters. Then, register this command in the list of commands for `req` in `PubSubReq`, which is named `cmd_kafka_fetch`.
+
+Then create the corresponding response body `KafkaFetchResp` and register it in the `resp` of `PubSubResp`, named `kafka_fetch_resp`.
+
+The protocol definition [pubsub.proto](https://github.com/apache/apisix/blob/master/apisix/include/apisix/model/pubsub.proto).
+
+#### Add a new option to the `scheme` configuration item in upstream
+
+Add a new option `kafka` to the `scheme` field enumeration in the `upstream` of `apisix/schema_def.lua`.
+
+The schema definition [schema_def.lua](https://github.com/apache/apisix/blob/master/apisix/schema_def.lua).
+
+#### Add a new `scheme` judgment branch to `http_access_phase`
+
+Add a `scheme` judgment branch to the `http_access_phase` function in `apisix/init.lua` to support the processing of `kafka` type upstreams. Because of Apache Kafka has its own clustering and partition scheme, we do not need to use the Apache APISIX built-in load balancing algorithm, so we intercept and take over the processing flow before selecting the upstream node, here using the `kafka_access_phase` function.
+
+The APISIX init file [init.lua](https://github.com/apache/apisix/blob/master/apisix/init.lua).
+
+#### Implement the required message system commands processing functions
+
+First, create an instance of the `pubsub` module, which is provided in the `core` package.
+
+Then, an instance of the Apache Kafka client is created, and this code is omitted here.
+
+Next, add the command registered in the protocol definition above to the `pubsub` instance, which will provide a callback function that provides the parameters parsed from the communication protocol, in which the developer needs to call the kafka client to get the data and return it to the `pubsub` module as the function return value.
+
+:::note Callback function prototype
+The `params` is the data in the protocol definition; the first return value is the data, which needs to contain the fields in the response body definition, and returns the `nil` value when there is an error; the second return value is the error, and returns the error string when there is an error
+:::
+
+Finally, it enters the loop to wait for client commands and when an error occurs it returns the error and stops the processing flow.
Review Comment:
fixed
##########
docs/en/latest/pubsub.md:
##########
@@ -58,6 +58,85 @@ Apache APISIX implement an extensible pubsub module, which is responsible for st
- Implement the required message system instruction processing functions
- Optional: Create plugins to support advanced configurations of this messaging system
-### Example
+### Example of Apache Kafka
-TODO, an example will be added later to point out how to support other messaging systems.
+#### Add new commands and response body definitions to `pubsub.proto`
+
+The core of the protocol definition in `pubsub.proto` is the two parts `PubSubReq` and `PubSubResp`.
+
+First, create the `CmdKafkaFetch` command and add the required parameters. Then, register this command in the list of commands for `req` in `PubSubReq`, which is named `cmd_kafka_fetch`.
+
+Then create the corresponding response body `KafkaFetchResp` and register it in the `resp` of `PubSubResp`, named `kafka_fetch_resp`.
+
+The protocol definition [pubsub.proto](https://github.com/apache/apisix/blob/master/apisix/include/apisix/model/pubsub.proto).
+
+#### Add a new option to the `scheme` configuration item in upstream
+
+Add a new option `kafka` to the `scheme` field enumeration in the `upstream` of `apisix/schema_def.lua`.
+
+The schema definition [schema_def.lua](https://github.com/apache/apisix/blob/master/apisix/schema_def.lua).
+
+#### Add a new `scheme` judgment branch to `http_access_phase`
+
+Add a `scheme` judgment branch to the `http_access_phase` function in `apisix/init.lua` to support the processing of `kafka` type upstreams. Because of Apache Kafka has its own clustering and partition scheme, we do not need to use the Apache APISIX built-in load balancing algorithm, so we intercept and take over the processing flow before selecting the upstream node, here using the `kafka_access_phase` function.
+
+The APISIX init file [init.lua](https://github.com/apache/apisix/blob/master/apisix/init.lua).
+
+#### Implement the required message system commands processing functions
+
+First, create an instance of the `pubsub` module, which is provided in the `core` package.
+
+Then, an instance of the Apache Kafka client is created, and this code is omitted here.
+
+Next, add the command registered in the protocol definition above to the `pubsub` instance, which will provide a callback function that provides the parameters parsed from the communication protocol, in which the developer needs to call the kafka client to get the data and return it to the `pubsub` module as the function return value.
+
+:::note Callback function prototype
+The `params` is the data in the protocol definition; the first return value is the data, which needs to contain the fields in the response body definition, and returns the `nil` value when there is an error; the second return value is the error, and returns the error string when there is an error
+:::
+
+Finally, it enters the loop to wait for client commands and when an error occurs it returns the error and stops the processing flow.
+
+The kafka pubsub implementation [kafka.lua](https://github.com/apache/apisix/blob/master/apisix/pubsub/kafka.lua).
+
+#### Optional: Create plugins to support advanced configurations of this messaging system
+
+Add the required fields to the plugin schema definition and write them to the context of the current request in the `access` function.
+
+The `kafka-proxy` plugin [kafka-proxy.lua](https://github.com/apache/apisix/blob/master/apisix/plugins/kafka-proxy.lua).
+
+Add this plugin to the list of plugins in the APISIX configuration file.
+
+The plugins list [config-default.yaml](https://github.com/apache/apisix/blob/master/conf/config-default.yaml).
+
+#### Results
+
+After this is done, create a route like the one below to connect to this messaging system via APISIX using the WebSocket.
+
+```shell
+curl -X PUT 'http://127.0.0.1:9080/apisix/admin/routes/kafka' \
+ -H 'X-API-KEY: <api-key>' \
Review Comment:
fixed
##########
docs/en/latest/pubsub.md:
##########
@@ -58,6 +58,85 @@ Apache APISIX implement an extensible pubsub module, which is responsible for st
- Implement the required message system instruction processing functions
- Optional: Create plugins to support advanced configurations of this messaging system
-### Example
+### Example of Apache Kafka
-TODO, an example will be added later to point out how to support other messaging systems.
+#### Add new commands and response body definitions to `pubsub.proto`
+
+The core of the protocol definition in `pubsub.proto` is the two parts `PubSubReq` and `PubSubResp`.
+
+First, create the `CmdKafkaFetch` command and add the required parameters. Then, register this command in the list of commands for `req` in `PubSubReq`, which is named `cmd_kafka_fetch`.
+
+Then create the corresponding response body `KafkaFetchResp` and register it in the `resp` of `PubSubResp`, named `kafka_fetch_resp`.
+
+The protocol definition [pubsub.proto](https://github.com/apache/apisix/blob/master/apisix/include/apisix/model/pubsub.proto).
+
+#### Add a new option to the `scheme` configuration item in upstream
+
+Add a new option `kafka` to the `scheme` field enumeration in the `upstream` of `apisix/schema_def.lua`.
+
+The schema definition [schema_def.lua](https://github.com/apache/apisix/blob/master/apisix/schema_def.lua).
+
+#### Add a new `scheme` judgment branch to `http_access_phase`
+
+Add a `scheme` judgment branch to the `http_access_phase` function in `apisix/init.lua` to support the processing of `kafka` type upstreams. Because of Apache Kafka has its own clustering and partition scheme, we do not need to use the Apache APISIX built-in load balancing algorithm, so we intercept and take over the processing flow before selecting the upstream node, here using the `kafka_access_phase` function.
+
+The APISIX init file [init.lua](https://github.com/apache/apisix/blob/master/apisix/init.lua).
+
+#### Implement the required message system commands processing functions
+
+First, create an instance of the `pubsub` module, which is provided in the `core` package.
+
+Then, an instance of the Apache Kafka client is created, and this code is omitted here.
+
+Next, add the command registered in the protocol definition above to the `pubsub` instance, which will provide a callback function that provides the parameters parsed from the communication protocol, in which the developer needs to call the kafka client to get the data and return it to the `pubsub` module as the function return value.
+
+:::note Callback function prototype
+The `params` is the data in the protocol definition; the first return value is the data, which needs to contain the fields in the response body definition, and returns the `nil` value when there is an error; the second return value is the error, and returns the error string when there is an error
+:::
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] hf400159 commented on a diff in pull request #7059: docs: add pubsub develop example for kafka
Posted by GitBox <gi...@apache.org>.
hf400159 commented on code in PR #7059:
URL: https://github.com/apache/apisix/pull/7059#discussion_r874722133
##########
docs/en/latest/pubsub.md:
##########
@@ -58,6 +58,85 @@ Apache APISIX implement an extensible pubsub module, which is responsible for st
- Implement the required message system instruction processing functions
- Optional: Create plugins to support advanced configurations of this messaging system
-### Example
+### Example of Apache Kafka
-TODO, an example will be added later to point out how to support other messaging systems.
+#### Add new commands and response body definitions to `pubsub.proto`
+
+The core of the protocol definition in `pubsub.proto` is the two parts `PubSubReq` and `PubSubResp`.
+
+First, create the `CmdKafkaFetch` command and add the required parameters. Then, register this command in the list of commands for `req` in `PubSubReq`, which is named `cmd_kafka_fetch`.
+
+Then create the corresponding response body `KafkaFetchResp` and register it in the `resp` of `PubSubResp`, named `kafka_fetch_resp`.
+
+The protocol definition [pubsub.proto](https://github.com/apache/apisix/blob/master/apisix/include/apisix/model/pubsub.proto).
+
+#### Add a new option to the `scheme` configuration item in upstream
+
+Add a new option `kafka` to the `scheme` field enumeration in the `upstream` of `apisix/schema_def.lua`.
+
+The schema definition [schema_def.lua](https://github.com/apache/apisix/blob/master/apisix/schema_def.lua).
+
+#### Add a new `scheme` judgment branch to `http_access_phase`
+
+Add a `scheme` judgment branch to the `http_access_phase` function in `apisix/init.lua` to support the processing of `kafka` type upstreams. Because of Apache Kafka has its own clustering and partition scheme, we do not need to use the Apache APISIX built-in load balancing algorithm, so we intercept and take over the processing flow before selecting the upstream node, here using the `kafka_access_phase` function.
+
+The APISIX init file [init.lua](https://github.com/apache/apisix/blob/master/apisix/init.lua).
+
+#### Implement the required message system commands processing functions
+
+First, create an instance of the `pubsub` module, which is provided in the `core` package.
+
+Then, an instance of the Apache Kafka client is created, and this code is omitted here.
+
+Next, add the command registered in the protocol definition above to the `pubsub` instance, which will provide a callback function that provides the parameters parsed from the communication protocol, in which the developer needs to call the kafka client to get the data and return it to the `pubsub` module as the function return value.
+
+:::note Callback function prototype
+The `params` is the data in the protocol definition; the first return value is the data, which needs to contain the fields in the response body definition, and returns the `nil` value when there is an error; the second return value is the error, and returns the error string when there is an error
+:::
Review Comment:
```suggestion
:::note Callback function prototype
The `params` is the data in the protocol definition; the first return value is the data, which needs to contain the fields in the response body definition, and returns the `nil` value when there is an error; the second return value is the error, and returns the error string when there is an error
:::
```
##########
docs/en/latest/pubsub.md:
##########
@@ -58,6 +58,85 @@ Apache APISIX implement an extensible pubsub module, which is responsible for st
- Implement the required message system instruction processing functions
- Optional: Create plugins to support advanced configurations of this messaging system
-### Example
+### Example of Apache Kafka
-TODO, an example will be added later to point out how to support other messaging systems.
+#### Add new commands and response body definitions to `pubsub.proto`
+
+The core of the protocol definition in `pubsub.proto` is the two parts `PubSubReq` and `PubSubResp`.
+
+First, create the `CmdKafkaFetch` command and add the required parameters. Then, register this command in the list of commands for `req` in `PubSubReq`, which is named `cmd_kafka_fetch`.
+
+Then create the corresponding response body `KafkaFetchResp` and register it in the `resp` of `PubSubResp`, named `kafka_fetch_resp`.
+
+The protocol definition [pubsub.proto](https://github.com/apache/apisix/blob/master/apisix/include/apisix/model/pubsub.proto).
+
+#### Add a new option to the `scheme` configuration item in upstream
+
+Add a new option `kafka` to the `scheme` field enumeration in the `upstream` of `apisix/schema_def.lua`.
+
+The schema definition [schema_def.lua](https://github.com/apache/apisix/blob/master/apisix/schema_def.lua).
+
+#### Add a new `scheme` judgment branch to `http_access_phase`
+
+Add a `scheme` judgment branch to the `http_access_phase` function in `apisix/init.lua` to support the processing of `kafka` type upstreams. Because of Apache Kafka has its own clustering and partition scheme, we do not need to use the Apache APISIX built-in load balancing algorithm, so we intercept and take over the processing flow before selecting the upstream node, here using the `kafka_access_phase` function.
+
+The APISIX init file [init.lua](https://github.com/apache/apisix/blob/master/apisix/init.lua).
+
+#### Implement the required message system commands processing functions
+
+First, create an instance of the `pubsub` module, which is provided in the `core` package.
+
+Then, an instance of the Apache Kafka client is created, and this code is omitted here.
+
+Next, add the command registered in the protocol definition above to the `pubsub` instance, which will provide a callback function that provides the parameters parsed from the communication protocol, in which the developer needs to call the kafka client to get the data and return it to the `pubsub` module as the function return value.
+
+:::note Callback function prototype
+The `params` is the data in the protocol definition; the first return value is the data, which needs to contain the fields in the response body definition, and returns the `nil` value when there is an error; the second return value is the error, and returns the error string when there is an error
+:::
+
+Finally, it enters the loop to wait for client commands and when an error occurs it returns the error and stops the processing flow.
+
+The kafka pubsub implementation [kafka.lua](https://github.com/apache/apisix/blob/master/apisix/pubsub/kafka.lua).
+
+#### Optional: Create plugins to support advanced configurations of this messaging system
+
+Add the required fields to the plugin schema definition and write them to the context of the current request in the `access` function.
+
+The `kafka-proxy` plugin [kafka-proxy.lua](https://github.com/apache/apisix/blob/master/apisix/plugins/kafka-proxy.lua).
+
+Add this plugin to the list of plugins in the APISIX configuration file.
+
+The plugins list [config-default.yaml](https://github.com/apache/apisix/blob/master/conf/config-default.yaml).
+
+#### Results
+
+After this is done, create a route like the one below to connect to this messaging system via APISIX using the WebSocket.
+
+```shell
+curl -X PUT 'http://127.0.0.1:9080/apisix/admin/routes/kafka' \
+ -H 'X-API-KEY: <api-key>' \
Review Comment:
```suggestion
-H 'X-API-KEY: ${api-key}' \
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] bzp2010 commented on a diff in pull request #7059: docs: add pubsub develop example for kafka
Posted by GitBox <gi...@apache.org>.
bzp2010 commented on code in PR #7059:
URL: https://github.com/apache/apisix/pull/7059#discussion_r874502513
##########
docs/en/latest/pubsub.md:
##########
@@ -58,6 +58,192 @@ Apache APISIX implement an extensible pubsub module, which is responsible for st
- Implement the required message system instruction processing functions
- Optional: Create plugins to support advanced configurations of this messaging system
-### Example
+### Example of Apache Kafka
-TODO, an example will be added later to point out how to support other messaging systems.
+#### Add new commands and response body definitions to `pubsub.proto`
+
+The core of the protocol definition in `pubsub.proto` is the two parts `PubSubReq` and `PubSubResp`.
+
+First, create the `CmdKafkaFetch` command and add the required parameters. Then, register this command in the list of commands for `req` in `PubSubReq`, which is named `cmd_kafka_fetch`.
+
+```protobuf
+message CmdKafkaFetch {
+ string topic = 1;
+ int32 partition = 2;
+ int64 offset = 3;
+}
+
+message PubSubReq {
+ int64 sequence = 1;
+ oneof req {
+ CmdKafkaFetch cmd_kafka_fetch = 31;
Review Comment:
updated to source file link
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on a diff in pull request #7059: docs: add pubsub develop example for kafka
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on code in PR #7059:
URL: https://github.com/apache/apisix/pull/7059#discussion_r874795115
##########
docs/en/latest/pubsub.md:
##########
@@ -58,6 +58,85 @@ Apache APISIX implement an extensible pubsub module, which is responsible for st
- Implement the required message system instruction processing functions
- Optional: Create plugins to support advanced configurations of this messaging system
-### Example
+### Example of Apache Kafka
-TODO, an example will be added later to point out how to support other messaging systems.
+#### Add new commands and response body definitions to `pubsub.proto`
+
+The core of the protocol definition in `pubsub.proto` is the two parts `PubSubReq` and `PubSubResp`.
+
+First, create the `CmdKafkaFetch` command and add the required parameters. Then, register this command in the list of commands for `req` in `PubSubReq`, which is named `cmd_kafka_fetch`.
+
+Then create the corresponding response body `KafkaFetchResp` and register it in the `resp` of `PubSubResp`, named `kafka_fetch_resp`.
+
+The protocol definition [pubsub.proto](https://github.com/apache/apisix/blob/master/apisix/include/apisix/model/pubsub.proto).
+
+#### Add a new option to the `scheme` configuration item in upstream
+
+Add a new option `kafka` to the `scheme` field enumeration in the `upstream` of `apisix/schema_def.lua`.
+
+The schema definition [schema_def.lua](https://github.com/apache/apisix/blob/master/apisix/schema_def.lua).
+
+#### Add a new `scheme` judgment branch to `http_access_phase`
+
+Add a `scheme` judgment branch to the `http_access_phase` function in `apisix/init.lua` to support the processing of `kafka` type upstreams. Because of Apache Kafka has its own clustering and partition scheme, we do not need to use the Apache APISIX built-in load balancing algorithm, so we intercept and take over the processing flow before selecting the upstream node, here using the `kafka_access_phase` function.
Review Comment:
```suggestion
Add a `scheme` judgment branch to the `http_access_phase` function in `apisix/init.lua` to support the processing of `kafka` type upstreams. Because Apache Kafka has its clustering and partition scheme, we do not need to use the Apache APISIX built-in load balancing algorithm, so we intercept and take over the processing flow before selecting the upstream node, using the `kafka_access_phase` function.
```
##########
docs/en/latest/pubsub.md:
##########
@@ -58,6 +58,85 @@ Apache APISIX implement an extensible pubsub module, which is responsible for st
- Implement the required message system instruction processing functions
- Optional: Create plugins to support advanced configurations of this messaging system
-### Example
+### Example of Apache Kafka
-TODO, an example will be added later to point out how to support other messaging systems.
+#### Add new commands and response body definitions to `pubsub.proto`
+
+The core of the protocol definition in `pubsub.proto` is the two parts `PubSubReq` and `PubSubResp`.
+
+First, create the `CmdKafkaFetch` command and add the required parameters. Then, register this command in the list of commands for `req` in `PubSubReq`, which is named `cmd_kafka_fetch`.
+
+Then create the corresponding response body `KafkaFetchResp` and register it in the `resp` of `PubSubResp`, named `kafka_fetch_resp`.
+
+The protocol definition [pubsub.proto](https://github.com/apache/apisix/blob/master/apisix/include/apisix/model/pubsub.proto).
+
+#### Add a new option to the `scheme` configuration item in upstream
+
+Add a new option `kafka` to the `scheme` field enumeration in the `upstream` of `apisix/schema_def.lua`.
+
+The schema definition [schema_def.lua](https://github.com/apache/apisix/blob/master/apisix/schema_def.lua).
+
+#### Add a new `scheme` judgment branch to `http_access_phase`
+
+Add a `scheme` judgment branch to the `http_access_phase` function in `apisix/init.lua` to support the processing of `kafka` type upstreams. Because of Apache Kafka has its own clustering and partition scheme, we do not need to use the Apache APISIX built-in load balancing algorithm, so we intercept and take over the processing flow before selecting the upstream node, here using the `kafka_access_phase` function.
+
+The APISIX init file [init.lua](https://github.com/apache/apisix/blob/master/apisix/init.lua).
+
+#### Implement the required message system commands processing functions
+
+First, create an instance of the `pubsub` module, which is provided in the `core` package.
+
+Then, an instance of the Apache Kafka client is created, and this code is omitted here.
Review Comment:
```suggestion
Then, an instance of the Apache Kafka client is created and omitted code here.
```
##########
docs/en/latest/pubsub.md:
##########
@@ -58,6 +58,85 @@ Apache APISIX implement an extensible pubsub module, which is responsible for st
- Implement the required message system instruction processing functions
- Optional: Create plugins to support advanced configurations of this messaging system
-### Example
+### Example of Apache Kafka
-TODO, an example will be added later to point out how to support other messaging systems.
+#### Add new commands and response body definitions to `pubsub.proto`
+
+The core of the protocol definition in `pubsub.proto` is the two parts `PubSubReq` and `PubSubResp`.
+
+First, create the `CmdKafkaFetch` command and add the required parameters. Then, register this command in the list of commands for `req` in `PubSubReq`, which is named `cmd_kafka_fetch`.
+
+Then create the corresponding response body `KafkaFetchResp` and register it in the `resp` of `PubSubResp`, named `kafka_fetch_resp`.
+
+The protocol definition [pubsub.proto](https://github.com/apache/apisix/blob/master/apisix/include/apisix/model/pubsub.proto).
+
+#### Add a new option to the `scheme` configuration item in upstream
+
+Add a new option `kafka` to the `scheme` field enumeration in the `upstream` of `apisix/schema_def.lua`.
+
+The schema definition [schema_def.lua](https://github.com/apache/apisix/blob/master/apisix/schema_def.lua).
+
+#### Add a new `scheme` judgment branch to `http_access_phase`
+
+Add a `scheme` judgment branch to the `http_access_phase` function in `apisix/init.lua` to support the processing of `kafka` type upstreams. Because of Apache Kafka has its own clustering and partition scheme, we do not need to use the Apache APISIX built-in load balancing algorithm, so we intercept and take over the processing flow before selecting the upstream node, here using the `kafka_access_phase` function.
+
+The APISIX init file [init.lua](https://github.com/apache/apisix/blob/master/apisix/init.lua).
+
+#### Implement the required message system commands processing functions
+
+First, create an instance of the `pubsub` module, which is provided in the `core` package.
+
+Then, an instance of the Apache Kafka client is created, and this code is omitted here.
+
+Next, add the command registered in the protocol definition above to the `pubsub` instance, which will provide a callback function that provides the parameters parsed from the communication protocol, in which the developer needs to call the kafka client to get the data and return it to the `pubsub` module as the function return value.
+
+:::note Callback function prototype
+The `params` is the data in the protocol definition; the first return value is the data, which needs to contain the fields in the response body definition, and returns the `nil` value when there is an error; the second return value is the error, and returns the error string when there is an error
+:::
+
+Finally, it enters the loop to wait for client commands and when an error occurs it returns the error and stops the processing flow.
Review Comment:
```suggestion
Finally, it enters the loop to wait for client commands, and when an error occurs, it returns the error and stops the processing flow.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org