You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@dubbo.apache.org by li...@apache.org on 2022/07/19 09:01:40 UTC
[dubbo-samples] branch master updated: Dubbo samples mesh k8s: add envoy healthChecks (#482)
This is an automated email from the ASF dual-hosted git repository.
liujun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/dubbo-samples.git
The following commit(s) were added to refs/heads/master by this push:
new 22f730d1 Dubbo samples mesh k8s: add envoy healthChecks (#482)
22f730d1 is described below
commit 22f730d1a02dc9988a83fb56c997e7f02fc4ca8d
Author: conghuhu <56...@users.noreply.github.com>
AuthorDate: Tue Jul 19 17:01:33 2022 +0800
Dubbo samples mesh k8s: add envoy healthChecks (#482)
---
dubbo-samples-mesh-k8s/README.md | 454 +++++++++++----------
dubbo-samples-mesh-k8s/assets/grpc_load.png | Bin 0 -> 37550 bytes
dubbo-samples-mesh-k8s/deploy/EnvoyFilter.yml | 41 ++
.../resources/k8s => deploy}/ServiceAccount.yml | 0
.../dubbo-samples-mesh-consumer/pom.xml | 5 +-
.../apache/dubbo/samples/ConsumerBootstrap.java | 15 +-
.../samples/action/GreetingServiceConsumer.java | 7 +-
.../src/main/resources/k8s/Deployment.yml | 24 +-
.../src/main/resources/k8s/Service.yml | 5 +-
.../resources/spring/dubbo-consumer.properties | 8 +-
.../dubbo-samples-mesh-provider/pom.xml | 5 +-
.../org/apache/dubbo/samples/impl/GreeterImpl.java | 76 ++--
.../src/main/resources/k8s/Deployment.yml | 3 -
.../src/main/resources/k8s/Service.yml | 10 +-
.../resources/spring/dubbo-provider.properties | 7 +-
15 files changed, 362 insertions(+), 298 deletions(-)
diff --git a/dubbo-samples-mesh-k8s/README.md b/dubbo-samples-mesh-k8s/README.md
index 0499ca2a..8d19388a 100644
--- a/dubbo-samples-mesh-k8s/README.md
+++ b/dubbo-samples-mesh-k8s/README.md
@@ -2,24 +2,42 @@
可以按照下文步骤，将 Demo 部署到本地集群，也可在 [KataCoda 在线快速体验]()。
-## 1 总体目标
+* [1 总体目标](#target)
+* [2 基本流程](#basic)
+* [3 详细步骤](#detail)
+ + [3.1 环境要求](#env)
+ + [3.2 前置条件](#prepare)
+ + [3.3 项目与镜像打包（可跳过）](#image)
+ + [3.4 部署到 Kubernetes](#deploy)
+ - [3.4.1 部署 Provider](#deploy_provider)
+ - [3.4.2 部署 Consumer](#deploy_consumer)
+ + [3.5 检查 Provider 和 Consumer 正常通信](#check)
+* [4 健康检查](#health_check)
+ + [4.1 Istio 服务的健康检查](#health_check_istio)
+ + [4.2 Envoy 主动健康检查](#health_check_envoy)
+* [5 下一步计划](#next)
+* [6 常用命令](#common)
+
+<h2 id="target">1 总体目标</h2>
* 部署 Dubbo 应用到 Kubernetes + Istio
-* 基于 Kubernetes 内置 Service 实现服务发现,Pilot 监听api-server，通过XDS下发给Envoy实例
+* 基于 Kubernetes 内置 Service 实现服务发现, Pilot 监听 api-server ，通过 XDS 下发给 Envoy 实例，去掉注册中心
* 将 Dubbo 应用对接到 Kubernetes 生命周期
-* 利用 istio 的envoy代理实现服务调用
+* 利用 Envoy 代理实现服务调用、负载均衡
+* 基于 EnvoyFilter 对 consumer 的上游集群做主动健康检查
-## 2 基本流程
+<h2 id="basic">2 基本流程</h2>
+
+1. 创建一个 Dubbo 应用( [dubbo-samples-mesh-k8s](https://github.com/apache/dubbo-samples/tree/master/dubbo-samples-mesh-k8s) )
+2. 构建容器镜像并推送到镜像仓库（ [dubbomesh 示例镜像](https://hub.docker.com/u/15841721425) ）
-1. 创建一个 Dubbo
- 应用( [dubbo-samples-kubernetes](https://github.com/apache/dubbo-samples/tree/master/dubbo-samples-mesh-k8s) )
-2. 构建容器镜像并推送到镜像仓库（ [dubbomesh 示例例镜像](https://hub.docker.com/u/15841721425) ）
3. 分别部署 Dubbo Provider 与 Dubbo Consumer 到 Kubernetes
4. 验证服务发现与调用正常
+5. 观测负载均衡与主动健康检查
-## 3 详细步骤
+<h2 id="detail">3 详细步骤</h2>
-### 3.1 环境要求
+<h3 id="env">3.1 环境要求</h3>
请确保本地安装如下环境，以提供容器运行时、Kubernetes集群及访问工具
@@ -41,7 +59,7 @@ minikube start
kubectl cluster-info
```
-### 3.2 前置条件
+<h3 id="prepare">3.2 前置条件</h3>
由于示例 Dubbo 项目均部署在 Pod 中且与 API-SERVER 有交互，因此有相应的权限要求，我们这里创建独立 ServiceAccount 并绑定必须的 Roles，后面所有的 Dubbo Kubernetes
资源都将使用这里新建的 ServiceAccount。
@@ -50,7 +68,7 @@ kubectl cluster-info
```shell
# 初始化命名空间和账号
-kubectl apply -f https://raw.githubusercontent.com/apache/dubbo-samples/master/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/src/main/resources/k8s/ServiceAccount.yml
+kubectl apply -f https://raw.githubusercontent.com/apache/dubbo-samples/master/dubbo-samples-mesh-k8s/deploy/ServiceAccount.yml
# 切换命名空间
kubens dubbo-demo
@@ -61,7 +79,7 @@ kubectl label namespace dubbo-demo istio-injection=enabled
# 注意项目路径一定是英文，否则protobuf编译失败
```
-### 3.3 项目与镜像打包（可跳过）
+<h3 id="image">3.3 项目与镜像打包（可跳过）</h3>
示例项目及相关镜像均已就绪，以下仅为指引说明，可直接跳过此步骤直接查看 3.4 小节。
@@ -80,20 +98,36 @@ kubectl label namespace dubbo-demo istio-injection=enabled
前提是本地需要对 dubbo 3.1.0-SNAPSHOT mvn install 后。
```xml
+
<dubbo.version>3.1.0-SNAPSHOT</dubbo.version>
-<!--<dubbo.version>3.0.7</dubbo.version>-->
+ <!--<dubbo.version>3.0.7</dubbo.version>-->
```
-设置 Dubbo 项目使用 Kubernetes 作为注册中心，这里通过 DEFAULT_MASTER_HOST指定使用默认 API-SERVER 集群地址 kubernetes.default.srv，同时还指定了
-namespace、trustCerts 两个参数
+设置 Dubbo 项目配置：
```properties
+# provider
dubbo.application.name=dubbo-samples-mesh-provider
dubbo.application.metadataServicePort=20885
-dubbo.registry.address=kubernetes://DEFAULT_MASTER_HOST?registry-type=service&duplicate=false&namespace=dubbo-demo&trustCerts=true
+dubbo.registry.address=N/A
dubbo.protocol.name=tri
dubbo.protocol.port=50052
dubbo.application.qosEnable=true
+# 为了使 Kubernetes 集群能够正常访问到探针，需要开启 QOS 允许远程访问，此操作有可能带来安全风险，请仔细评估后再打开
+dubbo.application.qosAcceptForeignIp=true
+
+```
+
+```properties
+# consumer
+dubbo.application.name=dubbo-samples-mesh-consumer
+dubbo.application.metadataServicePort=20885
+dubbo.registry.address=N/A
+dubbo.protocol.name=tri
+dubbo.protocol.port=20880
+dubbo.consumer.timeout=30000
+dubbo.application.qosEnable=true
+# 为了使 Kubernetes 集群能够正常访问到探针，需要开启 QOS 允许远程访问，此操作有可能带来安全风险，请仔细评估后再打开
dubbo.application.qosAcceptForeignIp=true
```
@@ -121,9 +155,9 @@ docker push 15841721425/dubbo-samples-mesh-provider
docker push 15841721425/dubbo-samples-mesh-consumer
```
-### 3.4 部署到 Kubernetes
+<h3 id="deploy">3.4 部署到 Kubernetes</h3>
-#### 3.4.1 部署 Provider
+<h4 id="deploy_provider">3.4.1 部署 Provider</h3>
```shell
# 部署 Service
@@ -146,7 +180,7 @@ kubectl get pods -l app=dubbo-samples-mesh-provider
kubectl logs your-pod-id
```
-#### 3.4.2 部署 Consumer
+<h4 id="deploy_consumer">3.4.2 部署 Consumer</h3>
```shell
# 部署 Service
@@ -158,7 +192,28 @@ kubectl apply -f https://raw.githubusercontent.com/apache/dubbo-samples/master/d
部署 consumer 与 provider 是一样的，这里也保持了 K8S Service 与 Dubbo consumer 名字一致： dubbo-samples-mesh-consumer。
-检查启动日志，consumer 完成对 provider 服务的消费。
+<h3 id="check">3.5 检查 Provider 和 Consumer 正常通信</h3>
+
+**注：**
+
+grpc 通过 HTTP/2 运行，与通过 HTTP/1.1 运行相比具有若干优点，例如高效的二进制编码显著降低序列化成本，通过单个连接复用请求和响应减少 TCP
+管理开销，以及自动类型检查。但是kubernetes的默认负载平衡通常不能对grpc起到作用，需要添加其他grpc负载均衡服务。
+
+- 为什么 grpc 需要特殊的负载均衡
+
+因为 grpc 构建在HTTP/2上，而 HTTP/2 被设计为具有单个长期 TCP
+连接，所有请求都被多路复用，意味着多个请求可以在任何时间点在同一连接上处于活动状态。这减少了连接管理开销，但也意味着连接级的均衡没有作用。一旦建立连接，就不再需要进行平衡。所有的请求都将固定到单个目标实例上，如下所示：
+
+<div style="text-align: center"><img src="./assets/grpc_load.png" width="600" alt="grpc负载不均"></div>
+
+Kubernetes 的 kube-proxy 本质上是一个L4负载平衡器，因此我们不能依靠它来平衡微服务之间的gRPC调用。
+
+使用 Envoy 就可以优雅的解决上诉问题。 详细配置在 Service.yml 中，其中：
+
+- grpc对应的服务端口，name要加 **grpc-** 前缀；
+- Service不能配置成Headless服务。
+
+继执行 3.4 步骤后， 检查启动日志，查看 consumer 完成对 provider 服务的消费。
```shell
# 查看 pod 列表
@@ -171,21 +226,30 @@ kubectl logs your-pod-id
kubectl logs your-pod-id -c istio-proxy
```
-### 3.5 检查 Provider 和 Consumer 正常通信
-
-可以看到 consumer pod 日志输出如下:
+可以看到 consumer pod 日志输出如下( Triple 协议被 Envoy 代理负载均衡):
```bash
-[22/04/22 01:10:24:024UTC]main INFO deploy.DefaultApplicationDeployer:[DUBBO]Dubbo Application[1.1](dubbo-samples-apiserver-consumer)is ready.,dubbo version:3.0.7,current host:172.17.0.6
- result:hello,Kubernetes Api Server
+[15/07/22 11:37:50:050 UTC] main INFO action.GreetingServiceConsumer: consumer Unary reply <-message:
+"hello,service mesh, response from provider: 172.17.0.11:50052, client: 172.17.0.11, local: dubbo-samples-mesh-provider, remote: null, isProviderSide: true"
+
+==================== dubbo invoke 1513 end ====================
+[15/07/22 11:38:00:000 UTC] main INFO action.GreetingServiceConsumer: consumer Unary reply <-message:
+"hello,service mesh, response from provider: 172.17.0.8:50052, client: 172.17.0.8, local: dubbo-samples-mesh-provider, remote: null, isProviderSide: true"
+
+==================== dubbo invoke 1514 end ====================
+[15/07/22 11:38:10:010 UTC] main INFO action.GreetingServiceConsumer: consumer Unary reply <-message:
+"hello,service mesh, response from provider: 172.17.0.6:50052, client: 172.17.0.6, local: dubbo-samples-mesh-provider, remote: null, isProviderSide: true"
+
+==================== dubbo invoke 1515 end ====================
```
consumer istio-proxy 日志输出如下:
```shell
-[2022-06-22T08:25:02.647Z] "POST /org.apache.dubbo.samples.Greeter/greet HTTP/2" 200
-- via_upstream - "-" 28 34 25 3 "-" "-" "11830c24-e1d0-997f-9f40-0237f272cd37" "dubbo-samples-mesh-provider:50052" "172.17.0.9:50052"
-outbound|50052||dubbo-samples-mesh-provider.dubbo-demo.svc.cluster.local 172.17.0.5:34826 172.17.0.9:50052 172.17.0.5:34824 - default
+[2022-07-15T05:35:14.418Z] "POST /org.apache.dubbo.samples.Greeter/greet HTTP/2" 200
+- via_upstream - "-" 19 160 2 1 "-" "-" "6b8a5a03-5783-98bf-9bee-f93ea6e3d68e"
+"dubbo-samples-mesh-provider:50052" "172.17.0.4:50052"
+outbound|50052||dubbo-samples-mesh-provider.dubbo-demo.svc.cluster.local 172.17.0.7:52768 10.101.172.129:50052 172.17.0.7:38488 - default
```
可以看到 provider pod 日志输出如下:
@@ -198,215 +262,175 @@ Server test dubbo tri k8s received greet request name: "Kubernetes Api Server"
provider istio-proxy 日志输出如下:
```shell
-[2022-06-22T08:23:28.440Z] "POST /org.apache.dubbo.samples.Greeter/greet HTTP/2" 200
-- via_upstream - "-" 28 34 15 2 "-" "-" "4ed30165-83d5-9c58-8458-9d558ba3bc5a" "dubbo-samples-mesh-provider:50052" "172.17.0.9:50052"
-inbound|50052|| 127.0.0.6:40761 172.17.0.9:50052 172.17.0.5:33626 outbound_.50052_._.dubbo-samples-mesh-provider.dubbo-demo.svc.cluster.local default
+[2022-07-15T05:25:34.061Z] "POST /org.apache.dubbo.samples.Greeter/greet HTTP/2" 200
+- via_upstream - "-" 19 162 1 1 "-" "-" "201e6976-da10-96e1-8da7-ad032e58db47"
+"dubbo-samples-mesh-provider:50052" "172.17.0.10:50052"
+ inbound|50052|| 127.0.0.6:47013 172.17.0.10:50052 172.17.0.7:60244
+ outbound_.50052_._.dubbo-samples-mesh-provider.dubbo-demo.svc.cluster.local default
```
-## 5 下一步计划
+<h2 id="health_check">4 健康检查</h2>
-TODO
+<h3 id="health_check_istio">4.1 Istio 服务的健康检查</h3>
-* 基于Triple的health服务，envoy探测应用的健康状况，然后模拟应用不健康后envoy的行为是否正常。
-* 探索envoy和istio支持的服务治理的内容，比如开发者需要实现重试，API处要传什么值。
+Pod 的生命周期 与服务调度息息相关，通过对 Kubernetes 官方探针的实现，能够使 Dubbo 乃至整个应用的生命周期与 Pod 的生命周期对齐。
-## 6 附录 k8s manifests
+存活检测
+对于 livenessProbe 存活检测，由于 Dubbo 框架本身无法获取到应用的存活状态，因此本接口无默认实现，且默认返回成功。开发者可以根据 SPI 定义对此 SPI 接口进行拓展，从应用层次对是否存活进行判断。
-ServiceAccount.yml
+就绪检测
+对于 readinessProbe 就绪检测，目前 Dubbo 默认提供了两个检测维度，一是对 Dubbo 服务自身是否启停做判断，另外是对所有服务是否存在已注册接口，如果所有服务均已从注册中心下线（可以通过 QOS
+运维进行操作）将返回未就绪的状态。
-```yaml
-apiVersion: v1
-kind: Namespace
-metadata:
- name: dubbo-demo
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: Role
-metadata:
- namespace: dubbo-demo
- name: dubbo-role
-rules:
- - apiGroups: [ "" ]
- resources: [ "pods" ]
- verbs: [ "get", "watch", "list", "update", "patch" ]
- - apiGroups: [ "", "service.dubbo.apache.org" ]
- resources: [ "services", "endpoints", "virtualservices", "destinationrules" ]
- verbs: [ "get", "watch", "list" ]
----
-apiVersion: v1
-kind: ServiceAccount
-metadata:
- name: dubbo-sa
- namespace: dubbo-demo
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: RoleBinding
-metadata:
- name: dubbo-sa-bind
- namespace: dubbo-demo
-roleRef:
- apiGroup: rbac.authorization.k8s.io
- kind: Role
- name: dubbo-role
-subjects:
- - kind: ServiceAccount
- name: dubbo-sa
-```
+启动检测
+对于 startupProbe 启动检测，目前Dubbo 默认提供了一个检测维度，即是在所有启动流程（接口暴露、注册中心写入等）均结束后返回已就绪状态。
-Service.yml
+**使用方法：**
+参考配置(具体可以参考 dubbo-samples-mesh-provider 的配置文件)
```yaml
-apiVersion: v1
-kind: Service
-metadata:
- name: dubbo-samples-mesh-provider
- namespace: dubbo-demo
-spec:
- clusterIP: None
- selector:
- app: dubbo-samples-mesh-provider
- ports:
- - name: tri
- protocol: TCP
- port: 50052
- targetPort: 50052
- - name: dubbo
- protocol: TCP
- port: 20880
- targetPort: 20880
- - name: qos
- protocol: TCP
- port: 22222
- targetPort: 22222
- - name: test
- protocol: TCP
- port: 31000
- targetPort: 31000
+livenessProbe:
+ httpGet:
+ path: /live
+ port: 22222
+ initialDelaySeconds: 5
+ periodSeconds: 5
+readinessProbe:
+ httpGet:
+ path: /ready
+ port: 22222
+ initialDelaySeconds: 5
+ periodSeconds: 5
+startupProbe:
+ httpGet:
+ path: /startup
+ port: 22222
+ failureThreshold: 30
+ periodSeconds: 10
```
-Deployment.yml
+<h3 id="health_check_envoy">4.2 Envoy 主动健康检查</h3>
+
+本次从 consumer 侧的 Envoy ，对其上游集群 provider 做主动健康检查(GRPC)。
+
+配置EnvoyFilter如下：
```yaml
-apiVersion: apps/v1
-kind: Deployment
+apiVersion: networking.istio.io/v1alpha3
+kind: EnvoyFilter
metadata:
- name: dubbo-samples-mesh-provider
+ name: cluster
namespace: dubbo-demo
spec:
- replicas: 2
- selector:
- matchLabels:
- app: dubbo-samples-mesh-provider
- template:
- metadata:
- labels:
- app: dubbo-samples-mesh-provider
- spec:
- serviceAccountName: dubbo-sa
- containers:
- - name: server
- image: 15841721425/dubbo-samples-mesh-provider
- imagePullPolicy: IfNotPresent
- ports:
- - containerPort: 50052
- protocol: TCP
- - containerPort: 22222
- protocol: TCP
- - containerPort: 31000
- protocol: TCP
- livenessProbe:
- httpGet:
- path: /live
- port: 22222
- initialDelaySeconds: 5
- periodSeconds: 5
- readinessProbe:
- httpGet:
- path: /ready
- port: 22222
- initialDelaySeconds: 5
- periodSeconds: 5
- startupProbe:
- httpGet:
- path: /startup
- port: 22222
- failureThreshold: 30
- periodSeconds: 10
+ workloadSelector:
+ labels:
+ app: dubbo-samples-mesh-consumer
+ configPatches:
+ - applyTo: CLUSTER
+ match:
+ cluster:
+ name: outbound|50052||dubbo-samples-mesh-provider.dubbo-demo.svc.cluster.local
+ patch:
+ operation: MERGE
+ value:
+ health_checks:
+ - timeout: 5s
+ interval: 5s
+ initial_jitter: 1s
+ interval_jitter: 1s
+ interval_jitter_percent: 50
+ unhealthy_threshold: 1
+ healthy_threshold: 1
+ reuse_connection: true
+ no_traffic_interval: 2s
+ no_traffic_healthy_interval: 4s
+ unhealthy_interval: 5s
+ unhealthy_edge_interval: 10s
+ healthy_edge_interval: 10s
+ tls_options:
+ alpn_protocols:
+ - http1.1
+ - h2
+ transport_socket_match_criteria:
+ useMTLS: true
+ grpc_health_check:
+ authority: dubbo-samples-mesh-provider.dubbo-demo.svc.cluster.local
```
-Service.yml
+Envoy健康检查的配置说明(
+详见[Envoy 健康检查文档](https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/core/v3/health_check.proto#envoy-v3-api-msg-config-core-v3-healthcheck)):
```yaml
-apiVersion: v1
-kind: Service
-metadata:
- name: dubbo-samples-mesh-consumer
- namespace: dubbo-demo
-spec:
- clusterIP: None
- selector:
- app: dubbo-samples-mesh-consumer
- ports:
- - name: dubbo
- protocol: TCP
- port: 20880
- targetPort: 20880
- - name: qos
- protocol: TCP
- port: 22222
- targetPort: 22222
- - name: test
- protocol: TCP
- port: 31000
- targetPort: 31000
+{
+ "timeout": "{...}",超时时间
+ "interval": "{...}",检测间隔
+ "initial_jitter": "{...}",初始抖动
+ "interval_jitter": "{...}",间隔抖动
+ "interval_jitter_percent": "...",间隔抖动比例
+ "unhealthy_threshold": "{...}",不健康阈值
+ "healthy_threshold": "{...}",健康阈值
+ "reuse_connection": "{...}",重用连接
+ "http_health_check": "{...}",http类型健康检测
+ "tcp_health_check": "{...}",tcp类型健康检测
+ "grpc_health_check": "{...}",grpc类型健康检测
+ "custom_health_check": "{...}",自定义健康检测
+ "no_traffic_interval": "{...}",没有流量时的间隔
+ "no_traffic_healthy_interval": "{...}",没有流量健康后的间隔
+ "unhealthy_interval": "{...}",不健康的间隔
+ "unhealthy_edge_interval": "{...}",不健康边缘间隔
+ "healthy_edge_interval": "{...}",健康边缘间隔
+ "event_log_path": "...",日志路径
+ "always_log_health_check_failures": "...",失败总是记录日志
+ "tls_options": "{...}",选项
+ "transport_socket_match_criteria": "{...}"trasport_socket匹配条件
+}
```
-Deployment.yml
+**实验结果**
-```yaml
-apiVersion: apps/v1
-kind: Deployment
-metadata:
- name: dubbo-samples-mesh-consumer
- namespace: dubbo-demo
-spec:
- replicas: 1
- selector:
- matchLabels:
- app: dubbo-samples-mesh-consumer
- template:
- metadata:
- labels:
- app: dubbo-samples-mesh-consumer
- spec:
- serviceAccountName: dubbo-sa
- containers:
- - name: server
- image: 15841721425/dubbo-samples-mesh-consumer
- imagePullPolicy: IfNotPresent
- ports:
- - containerPort: 20880
- protocol: TCP
- - containerPort: 22222
- protocol: TCP
- - containerPort: 31000
- protocol: TCP
- livenessProbe:
- httpGet:
- path: /live
- port: 22222
- initialDelaySeconds: 5
- periodSeconds: 5
- readinessProbe:
- httpGet:
- path: /ready
- port: 22222
- initialDelaySeconds: 5
- periodSeconds: 5
- startupProbe:
- httpGet:
- path: /startup
- port: 22222
- failureThreshold: 30
- periodSeconds: 10
+- 首先按照步骤 3.4 启动好 provider 和 consumer。
+-
+
+运行 `kubectl apply -f https://raw.githubusercontent.com/apache/dubbo-samples/master/dubbo-samples-mesh-k8s/deploy/EnvoyFilter.yml`
+
+- 随后查看 provider 侧的 istio-proxy 日志，可观察到 provider 接收到 GRPC 健康检查：
+
+```bash
+[2022-07-08T09:03:34.152Z] "POST /grpc.health.v1.Health/Check HTTP/2" 200 - via_upstream
+ - "-" 5 7 5 5 "-" "Envoy/HC" "b33a9988-2c15-99b0-be0c-e085b64d77fa" "dubbo-samples-mesh-provider.dubbo-demo.svc.cluster.local" "172.17.0.7:50052"
+ inbound|50052|| 127.0.0.6:44823 172.17.0.7:50052 172.17.0.8:53858 - default
```
+
+**参考：**
+
+- [Dubbo 探针文档](https://dubbo.apache.org/zh/docs/v3.0/references/lifecycle/brief/)
+- [Istio 服务的健康检查官方文档](https://istio.io/latest/zh/docs/ops/configuration/mesh/app-health-check/)
+- [Envoy 健康检查文档](https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/core/v3/health_check.proto#envoy-v3-api-msg-config-core-v3-healthcheck)
+
+<h2 id="next">5 下一步计划</h2>
+
+TODO
+
+* 探索envoy和istio支持的服务治理的内容，比如开发者需要实现重试，API处要传什么值。
+* 精简SDK。
+
+<h2 id="common">6 常用命令</h2>
+
+```shell
+# dump current Envoy configs
+kubectl exec -it ${your pod id} -c istio-proxy curl http://127.0.0.1:15000/config_dump > config_dump
+
+# 进入 istio-proxy 容器
+kubectl exec -it ${your pod id} -c istio-proxy -- /bin/bash
+
+# 查看容器日志
+kubectl logs ${your pod id} -n ${your namespace}
+
+kubectl logs ${your pod id} -n ${your namespace} -c istio-proxy
+
+# 开启自动注入sidecar
+kubectl label namespace ${your namespace} istio-injection=enabled --overwrite
+
+# 关闭自动注入sidecar
+kubectl label namespace ${your namespace} istio-injection=disabled --overwrite
+```
\ No newline at end of file
diff --git a/dubbo-samples-mesh-k8s/assets/grpc_load.png b/dubbo-samples-mesh-k8s/assets/grpc_load.png
new file mode 100644
index 00000000..cb37eca6
Binary files /dev/null and b/dubbo-samples-mesh-k8s/assets/grpc_load.png differ
diff --git a/dubbo-samples-mesh-k8s/deploy/EnvoyFilter.yml b/dubbo-samples-mesh-k8s/deploy/EnvoyFilter.yml
new file mode 100644
index 00000000..716a37c4
--- /dev/null
+++ b/dubbo-samples-mesh-k8s/deploy/EnvoyFilter.yml
@@ -0,0 +1,41 @@
+apiVersion: networking.istio.io/v1alpha3
+kind: EnvoyFilter
+metadata:
+ name: cluster
+ namespace: dubbo-demo
+spec:
+ workloadSelector:
+ labels:
+ app: dubbo-samples-mesh-consumer
+ configPatches:
+ - applyTo: CLUSTER
+ match:
+ cluster:
+ name: outbound|50052||dubbo-samples-mesh-provider.dubbo-demo.svc.cluster.local
+ patch:
+ operation: MERGE
+ value:
+ health_checks:
+ - timeout: 5s
+ interval: 5s
+ initial_jitter: 1s
+ interval_jitter: 1s
+ interval_jitter_percent: 50
+ unhealthy_threshold: 1
+ healthy_threshold: 20
+ reuse_connection: true
+ no_traffic_interval: 2s
+ no_traffic_healthy_interval: 4s
+ unhealthy_interval: 5s
+ unhealthy_edge_interval: 10s
+ healthy_edge_interval: 10s
+ # event_log_path: /tmp
+ # always_log_health_check_failures: true
+ tls_options:
+ alpn_protocols:
+ - http1.1
+ - h2
+ transport_socket_match_criteria:
+ useMTLS: true
+ grpc_health_check:
+ authority: dubbo-samples-mesh-provider.dubbo-demo.svc.cluster.local
\ No newline at end of file
diff --git a/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/src/main/resources/k8s/ServiceAccount.yml b/dubbo-samples-mesh-k8s/deploy/ServiceAccount.yml
similarity index 100%
rename from dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/src/main/resources/k8s/ServiceAccount.yml
rename to dubbo-samples-mesh-k8s/deploy/ServiceAccount.yml
diff --git a/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/pom.xml b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/pom.xml
index a2589363..861733a6 100644
--- a/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/pom.xml
+++ b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/pom.xml
@@ -30,11 +30,11 @@
<properties>
<source.level>1.8</source.level>
<target.level>1.8</target.level>
- <dubbo.version>3.1.0-SNAPSHOT</dubbo.version>
+ <dubbo.version>3.0.7</dubbo.version>
<spring.version>4.3.16.RELEASE</spring.version>
<junit.version>4.12</junit.version>
<maven-compiler-plugin.version>3.7.0</maven-compiler-plugin.version>
- <grpc.version>1.44.0</grpc.version>
+ <grpc.version>1.31.1</grpc.version>
<protoc.version>3.19.4</protoc.version>
<dubbo.compiler.version>0.0.4.1</dubbo.compiler.version>
</properties>
@@ -63,6 +63,7 @@
<groupId>org.apache.dubbo</groupId>
<artifactId>dubbo</artifactId>
<version>${dubbo.version}</version>
+ <!-- <version>3.1.0-SNAPSHOT</version>-->
</dependency>
<dependency>
diff --git a/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/src/main/java/org/apache/dubbo/samples/ConsumerBootstrap.java b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/src/main/java/org/apache/dubbo/samples/ConsumerBootstrap.java
index 52ee0ae4..742255bd 100644
--- a/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/src/main/java/org/apache/dubbo/samples/ConsumerBootstrap.java
+++ b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/src/main/java/org/apache/dubbo/samples/ConsumerBootstrap.java
@@ -19,9 +19,9 @@
package org.apache.dubbo.samples;
+import java.util.concurrent.atomic.AtomicInteger;
import org.apache.dubbo.config.spring.context.annotation.EnableDubbo;
import org.apache.dubbo.samples.action.GreetingServiceConsumer;
-
import org.springframework.context.annotation.AnnotationConfigApplicationContext;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.context.annotation.Configuration;
@@ -29,13 +29,18 @@ import org.springframework.context.annotation.PropertySource;
public class ConsumerBootstrap {
- public static void main(String[] args) {
+ public static void main(String[] args) throws InterruptedException {
AnnotationConfigApplicationContext context = new AnnotationConfigApplicationContext(ConsumerConfiguration.class);
context.start();
- System.out.println("==================== dubbo invoke started ====================");
+ System.out.println("==================== dubbo invoke loop started ====================");
GreetingServiceConsumer greetingServiceConsumer = context.getBean(GreetingServiceConsumer.class);
- greetingServiceConsumer.doSayHello("Kubernetes Api Server");
- System.out.println("==================== dubbo invoke end ====================");
+ AtomicInteger count = new AtomicInteger(0);
+ while (true) {
+ greetingServiceConsumer.doSayHello("service mesh");
+ System.out.println("==================== dubbo invoke " + count.get() + " end ====================");
+ count.getAndIncrement();
+ Thread.sleep(10000);
+ }
}
@Configuration
diff --git a/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/src/main/java/org/apache/dubbo/samples/action/GreetingServiceConsumer.java b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/src/main/java/org/apache/dubbo/samples/action/GreetingServiceConsumer.java
index ac04d876..b2841361 100644
--- a/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/src/main/java/org/apache/dubbo/samples/action/GreetingServiceConsumer.java
+++ b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/src/main/java/org/apache/dubbo/samples/action/GreetingServiceConsumer.java
@@ -33,13 +33,14 @@ public class GreetingServiceConsumer {
private static final Logger LOGGER = LoggerFactory.getLogger(GreetingServiceConsumer.class);
- @DubboReference(version = "1.0.0", providedBy = "dubbo-samples-mesh-provider", url = "tri://dubbo-samples-mesh-provider:50052")
+ @DubboReference(version = "1.0.0", providedBy = "dubbo-samples-mesh-provider", url = "tri://dubbo-samples-mesh-provider:50052", lazy = true)
+// @DubboReference(version = "1.0.0", providedBy = "dubbo-samples-mesh-provider", url = "tri://localhost:50052")
private Greeter greeter;
public void doSayHello(String name) {
final GreeterReply reply = greeter.greet(GreeterRequest.newBuilder()
- .setName(name)
- .build());
+ .setName(name)
+ .build());
LOGGER.info("consumer Unary reply <-{}", reply);
}
diff --git a/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/src/main/resources/k8s/Deployment.yml b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/src/main/resources/k8s/Deployment.yml
index 50eb7bdd..0ff54cc5 100644
--- a/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/src/main/resources/k8s/Deployment.yml
+++ b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/src/main/resources/k8s/Deployment.yml
@@ -26,26 +26,20 @@ spec:
- containerPort: 31000
protocol: TCP
livenessProbe:
- # httpGet:
- # path: /live
- # port: 22222
- grpc:
- port: 20880
+ httpGet:
+ path: /live
+ port: 22222
initialDelaySeconds: 5
periodSeconds: 5
readinessProbe:
- # httpGet:
- # path: /ready
- # port: 22222
- grpc:
- port: 20880
+ httpGet:
+ path: /ready
+ port: 22222
initialDelaySeconds: 5
periodSeconds: 5
startupProbe:
- # httpGet:
- # path: /startup
- # port: 22222
- grpc:
- port: 20880
+ httpGet:
+ path: /startup
+ port: 22222
failureThreshold: 30
periodSeconds: 10
\ No newline at end of file
diff --git a/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/src/main/resources/k8s/Service.yml b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/src/main/resources/k8s/Service.yml
index a1eb11b6..f03d9adf 100644
--- a/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/src/main/resources/k8s/Service.yml
+++ b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/src/main/resources/k8s/Service.yml
@@ -4,11 +4,12 @@ metadata:
name: dubbo-samples-mesh-consumer
namespace: dubbo-demo
spec:
- clusterIP: None
+ type: ClusterIP
+ sessionAffinity: None
selector:
app: dubbo-samples-mesh-consumer
ports:
- - name: dubbo
+ - name: grpc-dubbo
protocol: TCP
port: 20880
targetPort: 20880
diff --git a/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/src/main/resources/spring/dubbo-consumer.properties b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/src/main/resources/spring/dubbo-consumer.properties
index aef4b9cb..6e63f37e 100644
--- a/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/src/main/resources/spring/dubbo-consumer.properties
+++ b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/src/main/resources/spring/dubbo-consumer.properties
@@ -18,11 +18,11 @@
#
dubbo.application.name=dubbo-samples-mesh-consumer
dubbo.application.metadataServicePort=20885
-#dubbo.registry.address=N/A
-dubbo.registry.address=kubernetes://DEFAULT_MASTER_HOST?registry-type=service&duplicate=false&namespace=dubbo-demo&trustCerts=true&subscribed-services=dubbo-samples-mesh-provider
+#dubbo.application.metadataServicePort=20886
+dubbo.registry.address=N/A
#dubbo.registry.address=zookeeper://101.42.243.67:2181?timeout=60000
-#dubbo.protocol.name=tri
-#dubbo.protocol.port=-1
+dubbo.protocol.name=tri
+dubbo.protocol.port=20880
dubbo.consumer.timeout=30000
dubbo.application.qosEnable=true
dubbo.application.qosAcceptForeignIp=true
\ No newline at end of file
diff --git a/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/pom.xml b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/pom.xml
index 862dbbdc..9b175214 100644
--- a/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/pom.xml
+++ b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/pom.xml
@@ -11,8 +11,8 @@
<properties>
<source.level>1.8</source.level>
<target.level>1.8</target.level>
- <dubbo.version>3.1.0-SNAPSHOT</dubbo.version>
- <grpc.version>1.44.0</grpc.version>
+ <dubbo.version>3.0.7</dubbo.version>
+ <grpc.version>1.31.1</grpc.version>
<spring.version>4.3.16.RELEASE</spring.version>
<junit.version>4.12</junit.version>
<maven-compiler-plugin.version>3.7.0</maven-compiler-plugin.version>
@@ -44,6 +44,7 @@
<groupId>org.apache.dubbo</groupId>
<artifactId>dubbo</artifactId>
<version>${dubbo.version}</version>
+ <!-- <version>3.1.0-SNAPSHOT</version>-->
</dependency>
<dependency>
diff --git a/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/src/main/java/org/apache/dubbo/samples/impl/GreeterImpl.java b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/src/main/java/org/apache/dubbo/samples/impl/GreeterImpl.java
index 162d0130..7ee92830 100644
--- a/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/src/main/java/org/apache/dubbo/samples/impl/GreeterImpl.java
+++ b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/src/main/java/org/apache/dubbo/samples/impl/GreeterImpl.java
@@ -44,7 +44,7 @@ public class GreeterImpl extends DubboGreeterTriple.GreeterImplBase {
public static final Map<String, Boolean> cancelResultMap = new HashMap<>();
public GreeterImpl() {
- this.serverName = "test dubbo tri k8s";
+ this.serverName = "test dubbo tri mesh";
}
public GreeterImpl(String serverName) {
@@ -54,9 +54,16 @@ public class GreeterImpl extends DubboGreeterTriple.GreeterImplBase {
@Override
public GreeterReply greet(GreeterRequest request) {
LOGGER.info("Server {} received greet request {}", serverName, request);
+ boolean isProviderSide = RpcContext.getContext().isProviderSide();
+ String clientIP = RpcContext.getContext().getRemoteHost();
+ String remoteApplication = RpcContext.getContext().getRemoteApplicationName();
+ String application = RpcContext.getContext().getUrl().getParameter("application");
+ String protocol = RpcContext.getContext().getProtocol();
return GreeterReply.newBuilder()
- .setMessage("hello," + request.getName())
- .build();
+ .setMessage("hello," + request.getName() + ", response from provider: " + RpcContext.getContext().getLocalAddress() +
+ ", client: " + clientIP + ", local: " + application + ", remote: " + remoteApplication +
+ ", isProviderSide: " + isProviderSide)
+ .build();
}
@Override
@@ -81,32 +88,32 @@ public class GreeterImpl extends DubboGreeterTriple.GreeterImplBase {
@Override
public void cancelServerStream(GreeterRequest request,
- StreamObserver<GreeterReply> replyStream) {
+ StreamObserver<GreeterReply> replyStream) {
RpcContext.getCancellationContext().addListener(context -> {
LOGGER.info("cancel--cancelServerStream");
cancelResultMap.put("cancelServerStream", true);
});
for (int i = 0; i < 10; i++) {
replyStream.onNext(GreeterReply.newBuilder()
- .setMessage(request.getName() + "--" + i)
- .build());
+ .setMessage(request.getName() + "--" + i)
+ .build());
}
}
@Override
public StreamObserver<GreeterRequest> cancelBiStream(StreamObserver<GreeterReply> replyStream) {
RpcContext.getCancellationContext()
- .addListener(context -> {
- LOGGER.info("cancel--cancelBiStream");
- cancelResultMap.put("cancelBiStream", true);
- });
+ .addListener(context -> {
+ LOGGER.info("cancel--cancelBiStream");
+ cancelResultMap.put("cancelBiStream", true);
+ });
return new StreamObserver<GreeterRequest>() {
@Override
public void onNext(GreeterRequest data) {
LOGGER.info("Bi-Stream-Request:" + data.getName());
replyStream.onNext(GreeterReply.newBuilder()
- .setMessage(data.getName())
- .build());
+ .setMessage(data.getName())
+ .build());
}
@Override
@@ -124,19 +131,19 @@ public class GreeterImpl extends DubboGreeterTriple.GreeterImplBase {
@Override
public StreamObserver<GreeterRequest> cancelBiStream2(
- StreamObserver<GreeterReply> replyStream) {
+ StreamObserver<GreeterReply> replyStream) {
RpcContext.getCancellationContext()
- .addListener(context -> {
- LOGGER.info("cancel--cancelBiStream2");
- cancelResultMap.put("cancelBiStream2", true);
- });
+ .addListener(context -> {
+ LOGGER.info("cancel--cancelBiStream2");
+ cancelResultMap.put("cancelBiStream2", true);
+ });
return new StreamObserver<GreeterRequest>() {
@Override
public void onNext(GreeterRequest data) {
LOGGER.info("Bi-Stream-Request:" + data.getName());
replyStream.onNext(GreeterReply.newBuilder()
- .setMessage(data.getName())
- .build());
+ .setMessage(data.getName())
+ .build());
}
@Override
@@ -154,20 +161,20 @@ public class GreeterImpl extends DubboGreeterTriple.GreeterImplBase {
@Override
public StreamObserver<GreeterRequest> compressorBiStream(
- StreamObserver<GreeterReply> replyStream) {
+ StreamObserver<GreeterReply> replyStream) {
ServerStreamObserver<GreeterReply> replyServerStreamObserver = (ServerStreamObserver<GreeterReply>) replyStream;
replyServerStreamObserver.setCompression("gzip");
return getGreeterRequestStreamObserver(replyServerStreamObserver);
}
private StreamObserver<GreeterRequest> getGreeterRequestStreamObserver(
- StreamObserver<GreeterReply> streamObserver) {
+ StreamObserver<GreeterReply> streamObserver) {
return new StreamObserver<GreeterRequest>() {
@Override
public void onNext(GreeterRequest data) {
streamObserver.onNext(GreeterReply.newBuilder()
- .setMessage(data.getName())
- .build());
+ .setMessage(data.getName())
+ .build());
}
@Override
@@ -185,14 +192,14 @@ public class GreeterImpl extends DubboGreeterTriple.GreeterImplBase {
@Override
public StreamObserver<GreeterRequest> clientCompressorBiStream(
- StreamObserver<GreeterReply> replyStream) {
+ StreamObserver<GreeterReply> replyStream) {
ServerStreamObserver<GreeterReply> replyServerStreamObserver = (ServerStreamObserver<GreeterReply>) replyStream;
return getGreeterRequestStreamObserver(replyServerStreamObserver);
}
@Override
public StreamObserver<GreeterRequest> serverCompressorBiStream(
- StreamObserver<GreeterReply> replyStream) {
+ StreamObserver<GreeterReply> replyStream) {
ServerStreamObserver<GreeterReply> replyServerStreamObserver = (ServerStreamObserver<GreeterReply>) replyStream;
replyServerStreamObserver.setCompression("gzip");
return getGreeterRequestStreamObserver(replyServerStreamObserver);
@@ -202,15 +209,14 @@ public class GreeterImpl extends DubboGreeterTriple.GreeterImplBase {
public GreeterReply queryCancelResult(GreeterRequest request) {
boolean canceled = cancelResultMap.getOrDefault(request.getName(), false);
return GreeterReply.newBuilder()
- .setMessage(String.valueOf(canceled))
- .build();
+ .setMessage(String.valueOf(canceled))
+ .build();
}
-
public GreeterReply greetException(GreeterRequest request) {
RpcContext.getServerContext().setAttachment("str", "str")
- .setAttachment("integer", 1)
- .setAttachment("raw", new byte[]{1, 2, 3, 4});
+ .setAttachment("integer", 1)
+ .setAttachment("raw", new byte[] {1, 2, 3, 4});
throw new RuntimeException("Biz Exception");
}
@@ -224,8 +230,8 @@ public class GreeterImpl extends DubboGreeterTriple.GreeterImplBase {
n++;
LOGGER.info(data.getName() + " " + n);
replyStream.onNext(GreeterReply.newBuilder()
- .setMessage(data.getName() + " " + n)
- .build());
+ .setMessage(data.getName() + " " + n)
+ .build());
}
@Override
@@ -244,11 +250,11 @@ public class GreeterImpl extends DubboGreeterTriple.GreeterImplBase {
@Override
public void greetServerStream(GreeterRequest request,
- StreamObserver<GreeterReply> replyStream) {
+ StreamObserver<GreeterReply> replyStream) {
for (int i = 0; i < 10; i++) {
replyStream.onNext(GreeterReply.newBuilder()
- .setMessage(request.getName() + "--" + i)
- .build());
+ .setMessage(request.getName() + "--" + i)
+ .build());
}
replyStream.onCompleted();
}
diff --git a/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/src/main/resources/k8s/Deployment.yml b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/src/main/resources/k8s/Deployment.yml
index f5da9869..39a0d9df 100644
--- a/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/src/main/resources/k8s/Deployment.yml
+++ b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/src/main/resources/k8s/Deployment.yml
@@ -31,7 +31,6 @@ spec:
# port: 22222
grpc:
port: 50052
-# service: Health
initialDelaySeconds: 5
periodSeconds: 5
readinessProbe:
@@ -40,7 +39,6 @@ spec:
# port: 22222
grpc:
port: 50052
-# service: Health
initialDelaySeconds: 5
periodSeconds: 5
startupProbe:
@@ -49,6 +47,5 @@ spec:
# port: 22222
grpc:
port: 50052
-# service: Health
failureThreshold: 30
periodSeconds: 10
\ No newline at end of file
diff --git a/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/src/main/resources/k8s/Service.yml b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/src/main/resources/k8s/Service.yml
index a2c403ba..b0f81eb3 100644
--- a/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/src/main/resources/k8s/Service.yml
+++ b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/src/main/resources/k8s/Service.yml
@@ -4,18 +4,14 @@ metadata:
name: dubbo-samples-mesh-provider
namespace: dubbo-demo
spec:
- clusterIP: None
+ type: ClusterIP
+ sessionAffinity: None
selector:
app: dubbo-samples-mesh-provider
ports:
- - name: tri
- protocol: TCP
+ - name: grpc-tri
port: 50052
targetPort: 50052
- - name: dubbo
- protocol: TCP
- port: 20880
- targetPort: 20880
- name: qos
protocol: TCP
port: 22222
diff --git a/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/src/main/resources/spring/dubbo-provider.properties b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/src/main/resources/spring/dubbo-provider.properties
index a4ec19f4..8ddf71ca 100644
--- a/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/src/main/resources/spring/dubbo-provider.properties
+++ b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/src/main/resources/spring/dubbo-provider.properties
@@ -18,13 +18,10 @@
#
dubbo.application.name=dubbo-samples-mesh-provider
dubbo.application.metadataServicePort=20885
-#dubbo.registry.address=N/A
-dubbo.registry.address=kubernetes://DEFAULT_MASTER_HOST?registry-type=service&duplicate=false&namespace=dubbo-demo&trustCerts=true
+dubbo.registry.address=N/A
#dubbo.registry.address=zookeeper://101.42.243.67:2181?timeout=60000
dubbo.protocol.name=tri
dubbo.protocol.port=50052
# qos??
dubbo.application.qosEnable=true
-dubbo.application.qosAcceptForeignIp=true
-# token??
-#dubbo.provider.token=true
\ No newline at end of file
+dubbo.application.qosAcceptForeignIp=true
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org