You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@dubbo.apache.org by li...@apache.org on 2022/06/28 11:56:50 UTC

[dubbo-samples] branch master updated: Dubbo samples mesh k8s (#463)

This is an automated email from the ASF dual-hosted git repository.

liujun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/dubbo-samples.git


The following commit(s) were added to refs/heads/master by this push:
     new 4bda16e2 Dubbo samples mesh k8s (#463)
4bda16e2 is described below

commit 4bda16e2c00a00fe55c9d879c0147138ac4b2053
Author: conghuhu <56...@users.noreply.github.com>
AuthorDate: Tue Jun 28 19:56:42 2022 +0800

    Dubbo samples mesh k8s (#463)
---
 dubbo-samples-mesh-k8s/.gitignore                  |  33 ++
 dubbo-samples-mesh-k8s/README.md                   | 415 +++++++++++++++++++++
 .../dubbo-samples-mesh-consumer/Dockerfile         |   6 +
 .../dubbo-samples-mesh-consumer/pom.xml            | 217 +++++++++++
 .../apache/dubbo/samples/ConsumerBootstrap.java    |  48 +++
 .../samples/action/GreetingServiceConsumer.java    |  46 +++
 .../src/main/proto/greeter.proto                   |  54 +++
 .../src/main/resources/k8s/Deployment.yml          |  45 +++
 .../src/main/resources/k8s/Service.yml             |  22 ++
 .../src/main/resources/log4j.properties            |  23 ++
 .../resources/spring/dubbo-consumer.properties     |  28 ++
 .../dubbo-samples-mesh-provider/Dockerfile         |   6 +
 .../dubbo-samples-mesh-provider/pom.xml            | 198 ++++++++++
 .../apache/dubbo/samples/ProviderBootstrap.java    |  44 +++
 .../org/apache/dubbo/samples/impl/GreeterImpl.java | 256 +++++++++++++
 .../src/main/proto/greeter.proto                   |  54 +++
 .../src/main/resources/k8s/Deployment.yml          |  45 +++
 .../src/main/resources/k8s/Service.yml             |  26 ++
 .../src/main/resources/k8s/ServiceAccount.yml      |  36 ++
 .../src/main/resources/log4j.properties            |  23 ++
 .../resources/spring/dubbo-provider.properties     |  30 ++
 dubbo-samples-mesh-k8s/pom.xml                     |  50 +++
 pom.xml                                            |   1 +
 23 files changed, 1706 insertions(+)

diff --git a/dubbo-samples-mesh-k8s/.gitignore b/dubbo-samples-mesh-k8s/.gitignore
new file mode 100644
index 00000000..549e00a2
--- /dev/null
+++ b/dubbo-samples-mesh-k8s/.gitignore
@@ -0,0 +1,33 @@
+HELP.md
+target/
+!.mvn/wrapper/maven-wrapper.jar
+!**/src/main/**/target/
+!**/src/test/**/target/
+
+### STS ###
+.apt_generated
+.classpath
+.factorypath
+.project
+.settings
+.springBeans
+.sts4-cache
+
+### IntelliJ IDEA ###
+.idea
+*.iws
+*.iml
+*.ipr
+
+### NetBeans ###
+/nbproject/private/
+/nbbuild/
+/dist/
+/nbdist/
+/.nb-gradle/
+build/
+!**/src/main/**/build/
+!**/src/test/**/build/
+
+### VS Code ###
+.vscode/
diff --git a/dubbo-samples-mesh-k8s/README.md b/dubbo-samples-mesh-k8s/README.md
new file mode 100644
index 00000000..79c5ccdc
--- /dev/null
+++ b/dubbo-samples-mesh-k8s/README.md
@@ -0,0 +1,415 @@
+# Dubbo mesh using Istio
+
+可以按照下文步骤,将 Demo 部署到本地集群,也可在 [KataCoda 在线快速体验]()。
+
+## 1 总体目标
+
+* 部署 Dubbo 应用到 Kubernetes + Istio
+* 基于 Kubernetes 内置 Service 实现服务发现,Pilot 监听api-server,通过XDS下发给Envoy实例
+* 将 Dubbo 应用对接到 Kubernetes 生命周期
+* 利用 istio 的envoy代理实现服务调用
+
+## 2 基本流程
+
+1. 创建一个 Dubbo
+   应用( [dubbo-samples-kubernetes](https://github.com/apache/dubbo-samples/tree/master/dubbo-samples-kubernetes) )
+2. 构建容器镜像并推送到镜像仓库( [dubboteam 示例例镜像](https://hub.docker.com/u/dubboteam) )
+3. 分别部署 Dubbo Provider 与 Dubbo Consumer 到 Kubernetes
+4. 验证服务发现与调用正常
+
+## 3 详细步骤
+
+### 3.1 环境要求
+
+请确保本地安装如下环境,以提供容器运行时、Kubernetes集群及访问工具
+
+* [Docker](https://www.docker.com/get-started/)
+* [Minikube](https://minikube.sigs.k8s.io/docs/start/)
+* [Kubectl](https://kubernetes.io/docs/tasks/tools/)
+* [Kubens(optional)](https://github.com/ahmetb/kubectx)
+* [Istio](https://istio.io/latest/zh/)
+
+通过以下命令启动本地 Kubernetes 集群
+
+```shell
+minikube start
+```
+
+通过 kubectl 检查集群正常运行,且 kubectl 绑定到默认本地集群
+
+```shell
+kubectl cluster-info
+```
+
+### 3.2 前置条件
+
+由于示例 Dubbo 项目均部署在 Pod 中且与 API-SERVER 有交互,因此有相应的权限要求,我们这里创建独立 ServiceAccount 并绑定必须的 Roles,后面所有的 Dubbo Kubernetes
+资源都将使用这里新建的 ServiceAccount。
+
+通过以下命令我们创建了独立的 Namespace `dubbo-demo` 与 ServiceAccount `dubbo-sa`。
+
+```shell
+# 初始化命名空间和账号
+kubectl apply -f https://raw.githubusercontent.com/apache/dubbo-samples/master/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/src/main/resources/k8s/ServiceAccount.yml
+
+# 切换命名空间
+kubens dubbo-demo
+
+# dubbo-demo 开启自动注入
+kubectl label namespace dubbo-demo istio-injection=enabled
+
+# 注意项目路径一定是英文,否则protobuf编译失败
+```
+
+### 3.3 项目与镜像打包(可跳过)
+
+示例项目及相关镜像均已就绪,以下仅为指引说明,可直接跳过此步骤直接查看 3.4 小节。
+
+注意,由于 kubernetes 为独立扩展项目,开启 Kubernetes 支持前请添加如下依赖到 pom.xml
+
+```xml
+
+<dependency>
+    <groupId>org.apache.dubbo.extensions</groupId>
+    <artifactId>dubbo-registry-kubernetes</artifactId>
+    <version>1.0.2-SNAPSHOT</version>
+</dependency>
+```
+
+如果为 dubbo 3.1 的贡献者,在dubbo-samples-mesh-consumer和dubbo-samples-mesh-provider的pom.xml中,可作如下修改,
+前提是本地需要对 dubbo 3.1.0-SNAPSHOT mvn install 后。
+
+```xml
+
+<dependency>
+    <groupId>org.apache.dubbo</groupId>
+    <artifactId>dubbo</artifactId>
+    <!--            <version>${dubbo.version}</version>-->
+    <version>3.1.0-SNAPSHOT</version>
+</dependency>
+```
+
+设置 Dubbo 项目使用 Kubernetes 作为注册中心,这里通过 DEFAULT_MASTER_HOST指定使用默认 API-SERVER 集群地址 kubernetes.default.srv,同时还指定了
+namespace、trustCerts 两个参数
+
+```properties
+dubbo.application.name=dubbo-samples-mesh-provider
+dubbo.application.metadataServicePort=20885
+dubbo.registry.address=kubernetes://DEFAULT_MASTER_HOST?registry-type=service&duplicate=false&namespace=dubbo-demo&trustCerts=true
+dubbo.protocol.name=tri
+dubbo.protocol.port=50052
+dubbo.application.qosEnable=true
+dubbo.application.qosAcceptForeignIp=true
+
+```
+
+如果要在本地打包镜像,可通过提供的Dockerfile打包镜像(也可以直接使用示例提供好的镜像包)
+
+```shell
+# 打包镜像
+cd dubbo-samples-mesh-provider
+docker build -f ./Dockerfile -t dubbo-samples-mesh-provider .
+
+cd dubbo-samples-mesh-consumer
+docker build -f ./Dockerfile -t dubbo-samples-mesh-consumer .
+
+# 重命名镜像
+docker tag dubbo-samples-mesh-provider:latest 15841721425/dubbo-samples-mesh-provider
+
+docker tag dubbo-samples-mesh-consumer:latest 15841721425/dubbo-samples-mesh-consumer
+
+# 推到镜像仓库
+docker push 15841721425/dubbo-samples-mesh-provider
+
+docker push 15841721425/dubbo-samples-mesh-consumer
+```
+
+### 3.4 部署到 Kubernetes
+
+#### 3.4.1 部署 Provider
+
+```shell
+# 部署 Service
+kubectl apply -f https://raw.githubusercontent.com/apache/dubbo-samples/master/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/src/main/resources/k8s/Service.yml
+
+# 部署 Deployment
+kubectl apply -f https://raw.githubusercontent.com/apache/dubbo-samples/master/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/src/main/resources/k8s/Deployment.yml
+```
+
+以上命令创建了一个名为 `dubbo-samples-mesh-provider` 的 Service,注意这里的 service name 与项目中的 dubbo 应用名是一样的。
+
+接着 Deployment 部署了一个 2 副本的 pod 实例,至此 Provider 启动完成。  
+可以通过如下命令检查启动日志。
+
+```shell
+# 查看 pod 列表
+kubectl get pods -l app=dubbo-samples-mesh-provider
+
+# 查看 pod 部署日志
+kubectl logs your-pod-id
+```
+
+#### 3.4.2 部署 Consumer
+
+```shell
+# 部署 Service
+kubectl apply -f https://raw.githubusercontent.com/apache/dubbo-samples/master/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/src/main/resources/k8s/Service.yml
+
+# 部署 Deployment
+kubectl apply -f https://raw.githubusercontent.com/apache/dubbo-samples/master/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/src/main/resources/k8s/Deployment.yml
+```
+
+部署 consumer 与 provider 是一样的,这里也保持了 K8S Service 与 Dubbo consumer 名字一致: dubbo-samples-mesh-consumer。
+
+检查启动日志,consumer 完成对 provider 服务的消费。
+
+```shell
+# 查看 pod 列表
+kubectl get pods -l app=dubbo-samples-mesh-consumer
+
+# 查看 pod 部署日志
+kubectl logs your-pod-id
+
+# 查看 pod isitio-proxy 日志
+kubectl logs your-pod-id -c istio-proxy
+```
+
+### 3.5 检查 Provider 和 Consumer 正常通信
+
+可以看到 consumer pod 日志输出如下:
+
+```bash
+[22/04/22 01:10:24:024UTC]main INFO deploy.DefaultApplicationDeployer:[DUBBO]Dubbo Application[1.1](dubbo-samples-apiserver-consumer)is ready.,dubbo version:3.0.7,current host:172.17.0.6
+        result:hello,Kubernetes Api Server
+```
+
+consumer istio-proxy 日志输出如下:
+
+```shell
+[2022-06-22T08:25:02.647Z] "POST /org.apache.dubbo.samples.Greeter/greet HTTP/2" 200 
+- via_upstream - "-" 28 34 25 3 "-" "-" "11830c24-e1d0-997f-9f40-0237f272cd37" "dubbo-samples-mesh-provider:50052" "172.17.0.9:50052" 
+outbound|50052||dubbo-samples-mesh-provider.dubbo-demo.svc.cluster.local 172.17.0.5:34826 172.17.0.9:50052 172.17.0.5:34824 - default
+```
+
+可以看到 provider pod 日志输出如下:
+
+```shell
+[22/06/22 08:22:36:036 UTC] Dubbo-protocol-50052-thread-8  INFO impl.GreeterImpl: 
+Server test dubbo tri k8s received greet request name: "Kubernetes Api Server"
+```
+
+provider istio-proxy 日志输出如下:
+
+```shell
+[2022-06-22T08:23:28.440Z] "POST /org.apache.dubbo.samples.Greeter/greet HTTP/2" 200 
+- via_upstream - "-" 28 34 15 2 "-" "-" "4ed30165-83d5-9c58-8458-9d558ba3bc5a" "dubbo-samples-mesh-provider:50052" "172.17.0.9:50052" 
+inbound|50052|| 127.0.0.6:40761 172.17.0.9:50052 172.17.0.5:33626 outbound_.50052_._.dubbo-samples-mesh-provider.dubbo-demo.svc.cluster.local default
+```
+
+## 5 下一步计划
+
+TODO
+
+* 基于Triple的health服务,envoy探测应用的健康状况,然后模拟应用不健康后envoy的行为是否正常。
+* 探索envoy和istio支持的服务治理的内容,比如开发者需要实现重试,API处要传什么值。
+
+## 6 附录 k8s manifests
+
+ServiceAccount.yml
+
+```yaml
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: dubbo-demo
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  namespace: dubbo-demo
+  name: dubbo-role
+rules:
+  - apiGroups: [ "" ]
+    resources: [ "pods" ]
+    verbs: [ "get", "watch", "list", "update", "patch" ]
+  - apiGroups: [ "", "service.dubbo.apache.org" ]
+    resources: [ "services", "endpoints", "virtualservices", "destinationrules" ]
+    verbs: [ "get", "watch", "list" ]
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: dubbo-sa
+  namespace: dubbo-demo
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: dubbo-sa-bind
+  namespace: dubbo-demo
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: dubbo-role
+subjects:
+  - kind: ServiceAccount
+    name: dubbo-sa
+```
+
+Service.yml
+
+```yaml
+apiVersion: v1
+kind: Service
+metadata:
+  name: dubbo-samples-mesh-provider
+  namespace: dubbo-demo
+spec:
+  clusterIP: None
+  selector:
+    app: dubbo-samples-mesh-provider
+  ports:
+    - name: tri
+      protocol: TCP
+      port: 50052
+      targetPort: 50052
+    - name: dubbo
+      protocol: TCP
+      port: 20880
+      targetPort: 20880
+    - name: qos
+      protocol: TCP
+      port: 22222
+      targetPort: 22222
+    - name: test
+      protocol: TCP
+      port: 31000
+      targetPort: 31000
+```
+
+Deployment.yml
+
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: dubbo-samples-mesh-provider
+  namespace: dubbo-demo
+spec:
+  replicas: 2
+  selector:
+    matchLabels:
+      app: dubbo-samples-mesh-provider
+  template:
+    metadata:
+      labels:
+        app: dubbo-samples-mesh-provider
+    spec:
+      serviceAccountName: dubbo-sa
+      containers:
+        - name: server
+          image: 15841721425/dubbo-samples-mesh-provider
+          imagePullPolicy: IfNotPresent
+          ports:
+            - containerPort: 50052
+              protocol: TCP
+            - containerPort: 22222
+              protocol: TCP
+            - containerPort: 31000
+              protocol: TCP
+          livenessProbe:
+            httpGet:
+              path: /live
+              port: 22222
+            initialDelaySeconds: 5
+            periodSeconds: 5
+          readinessProbe:
+            httpGet:
+              path: /ready
+              port: 22222
+            initialDelaySeconds: 5
+            periodSeconds: 5
+          startupProbe:
+            httpGet:
+              path: /startup
+              port: 22222
+            failureThreshold: 30
+            periodSeconds: 10
+```
+
+Service.yml
+
+```yaml
+apiVersion: v1
+kind: Service
+metadata:
+  name: dubbo-samples-mesh-consumer
+  namespace: dubbo-demo
+spec:
+  clusterIP: None
+  selector:
+    app: dubbo-samples-mesh-consumer
+  ports:
+    - name: dubbo
+      protocol: TCP
+      port: 20880
+      targetPort: 20880
+    - name: qos
+      protocol: TCP
+      port: 22222
+      targetPort: 22222
+    - name: test
+      protocol: TCP
+      port: 31000
+      targetPort: 31000
+```
+
+Deployment.yml
+
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: dubbo-samples-mesh-consumer
+  namespace: dubbo-demo
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: dubbo-samples-mesh-consumer
+  template:
+    metadata:
+      labels:
+        app: dubbo-samples-mesh-consumer
+    spec:
+      serviceAccountName: dubbo-sa
+      containers:
+        - name: server
+          image: 15841721425/dubbo-samples-mesh-consumer
+          imagePullPolicy: IfNotPresent
+          ports:
+            - containerPort: 20880
+              protocol: TCP
+            - containerPort: 22222
+              protocol: TCP
+            - containerPort: 31000
+              protocol: TCP
+          livenessProbe:
+            httpGet:
+              path: /live
+              port: 22222
+            initialDelaySeconds: 5
+            periodSeconds: 5
+          readinessProbe:
+            httpGet:
+              path: /ready
+              port: 22222
+            initialDelaySeconds: 5
+            periodSeconds: 5
+          startupProbe:
+            httpGet:
+              path: /startup
+              port: 22222
+            failureThreshold: 30
+            periodSeconds: 10
+```
diff --git a/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/Dockerfile b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/Dockerfile
new file mode 100644
index 00000000..52a72f48
--- /dev/null
+++ b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/Dockerfile
@@ -0,0 +1,6 @@
+FROM openjdk:8-jre
+ADD ./target/dubbo-samples-mesh-consumer-1.0-SNAPSHOT.jar /dubbo-samples-mesh-consumer-1.0-SNAPSHOT.jar
+EXPOSE 20880
+EXPOSE 31000
+EXPOSE 22222
+CMD java -jar -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=31000 /dubbo-samples-mesh-consumer-1.0-SNAPSHOT.jar
\ No newline at end of file
diff --git a/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/pom.xml b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/pom.xml
new file mode 100644
index 00000000..d4f90bac
--- /dev/null
+++ b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/pom.xml
@@ -0,0 +1,217 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+  ~
+  ~   Licensed to the Apache Software Foundation (ASF) under one or more
+  ~   contributor license agreements.  See the NOTICE file distributed with
+  ~   this work for additional information regarding copyright ownership.
+  ~   The ASF licenses this file to You under the Apache License, Version 2.0
+  ~   (the "License"); you may not use this file except in compliance with
+  ~   the License.  You may obtain a copy of the License at
+  ~
+  ~       http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~   Unless required by applicable law or agreed to in writing, software
+  ~   distributed under the License is distributed on an "AS IS" BASIS,
+  ~   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  ~   See the License for the specific language governing permissions and
+  ~   limitations under the License.
+  ~
+  -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <modelVersion>4.0.0</modelVersion>
+
+    <groupId>org.apache.dubbo</groupId>
+    <artifactId>dubbo-samples-mesh-consumer</artifactId>
+    <version>1.0-SNAPSHOT</version>
+
+    <properties>
+        <source.level>1.8</source.level>
+        <target.level>1.8</target.level>
+        <dubbo.version>3.0.7</dubbo.version>
+        <spring.version>4.3.16.RELEASE</spring.version>
+        <junit.version>4.12</junit.version>
+        <maven-compiler-plugin.version>3.7.0</maven-compiler-plugin.version>
+        <grpc.version>1.31.1</grpc.version>
+        <protoc.version>3.19.4</protoc.version>
+        <dubbo.compiler.version>0.0.4.1-SNAPSHOT</dubbo.compiler.version>
+    </properties>
+
+    <dependencyManagement>
+        <dependencies>
+            <dependency>
+                <groupId>org.springframework</groupId>
+                <artifactId>spring-framework-bom</artifactId>
+                <version>${spring.version}</version>
+                <type>pom</type>
+                <scope>import</scope>
+            </dependency>
+            <dependency>
+                <groupId>org.apache.dubbo</groupId>
+                <artifactId>dubbo-bom</artifactId>
+                <version>${dubbo.version}</version>
+                <type>pom</type>
+                <scope>import</scope>
+            </dependency>
+        </dependencies>
+    </dependencyManagement>
+
+    <dependencies>
+        <dependency>
+            <groupId>org.apache.dubbo</groupId>
+            <artifactId>dubbo</artifactId>
+            <version>${dubbo.version}</version>
+<!--            <version>3.1.0-SNAPSHOT</version>-->
+        </dependency>
+
+        <dependency>
+            <groupId>com.google.protobuf</groupId>
+            <artifactId>protobuf-java</artifactId>
+            <version>3.20.1</version>
+        </dependency>
+
+        <dependency>
+            <groupId>org.apache.dubbo.extensions</groupId>
+            <artifactId>dubbo-registry-kubernetes</artifactId>
+            <version>1.0.2-SNAPSHOT</version>
+        </dependency>
+
+        <dependency>
+            <groupId>org.apache.dubbo</groupId>
+            <artifactId>dubbo-dependencies-zookeeper-curator5</artifactId>
+            <version>${dubbo.version}</version>
+            <type>pom</type>
+        </dependency>
+
+        <dependency>
+            <groupId>io.grpc</groupId>
+            <artifactId>grpc-all</artifactId>
+            <version>${grpc.version}</version>
+        </dependency>
+
+        <dependency>
+            <groupId>junit</groupId>
+            <artifactId>junit</artifactId>
+            <version>${junit.version}</version>
+            <scope>test</scope>
+        </dependency>
+
+        <dependency>
+            <groupId>org.springframework</groupId>
+            <artifactId>spring-test</artifactId>
+            <scope>test</scope>
+        </dependency>
+
+        <dependency>
+            <groupId>org.springframework</groupId>
+            <artifactId>spring-context</artifactId>
+        </dependency>
+
+        <dependency>
+            <groupId>org.testcontainers</groupId>
+            <artifactId>testcontainers</artifactId>
+            <version>1.12.3</version>
+            <scope>test</scope>
+        </dependency>
+    </dependencies>
+
+    <profiles>
+        <!-- For jdk 11 above JavaEE annotation -->
+        <profile>
+            <id>javax.annotation</id>
+            <activation>
+                <jdk>[1.11,)</jdk>
+            </activation>
+            <dependencies>
+                <dependency>
+                    <groupId>javax.annotation</groupId>
+                    <artifactId>javax.annotation-api</artifactId>
+                    <version>1.3.2</version>
+                </dependency>
+            </dependencies>
+        </profile>
+    </profiles>
+
+    <build>
+        <extensions>
+            <extension>
+                <groupId>kr.motd.maven</groupId>
+                <artifactId>os-maven-plugin</artifactId>
+                <version>1.6.1</version>
+            </extension>
+        </extensions>
+        <plugins>
+            <plugin>
+                <groupId>org.apache.maven.plugins</groupId>
+                <artifactId>maven-compiler-plugin</artifactId>
+                <version>${maven-compiler-plugin.version}</version>
+                <configuration>
+                    <source>${source.level}</source>
+                    <target>${target.level}</target>
+                </configuration>
+            </plugin>
+            <plugin>
+                <groupId>org.springframework.boot</groupId>
+                <artifactId>spring-boot-maven-plugin</artifactId>
+                <version>2.6.6</version>
+                <executions>
+                    <execution>
+                        <goals>
+                            <goal>repackage</goal>
+                        </goals>
+                    </execution>
+                </executions>
+            </plugin>
+            <plugin>
+                <groupId>org.xolstice.maven.plugins</groupId>
+                <artifactId>protobuf-maven-plugin</artifactId>
+                <version>0.6.1</version>
+                <configuration>
+                    <protocArtifact>com.google.protobuf:protoc:${protoc.version}:exe:${os.detected.classifier}
+                    </protocArtifact>
+                    <pluginId>grpc-java</pluginId>
+                    <pluginArtifact>io.grpc:protoc-gen-grpc-java:${grpc.version}:exe:${os.detected.classifier}
+                    </pluginArtifact>
+                    <protocPlugins>
+                        <protocPlugin>
+                            <id>dubbo</id>
+                            <groupId>org.apache.dubbo</groupId>
+                            <artifactId>dubbo-compiler</artifactId>
+                            <version>${dubbo.compiler.version}</version>
+                            <mainClass>org.apache.dubbo.gen.tri.Dubbo3TripleGenerator</mainClass>
+                        </protocPlugin>
+                    </protocPlugins>
+                </configuration>
+                <executions>
+                    <execution>
+                        <goals>
+                            <goal>compile</goal>
+                            <goal>test-compile</goal>
+                            <goal>compile-custom</goal>
+                            <goal>test-compile-custom</goal>
+                        </goals>
+                    </execution>
+                </executions>
+            </plugin>
+        </plugins>
+    </build>
+
+    <repositories>
+        <repository>
+            <id>apache.snapshots.https</id>
+            <name>Apache Development Snapshot Repository</name>
+            <url>https://repository.apache.org/content/repositories/snapshots</url>
+            <layout>default</layout>
+            <releases>
+                <enabled>false</enabled>
+            </releases>
+            <snapshots>
+                <enabled>true</enabled>
+                <updatePolicy>daily</updatePolicy>
+            </snapshots>
+        </repository>
+    </repositories>
+
+</project>
\ No newline at end of file
diff --git a/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/src/main/java/org/apache/dubbo/samples/ConsumerBootstrap.java b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/src/main/java/org/apache/dubbo/samples/ConsumerBootstrap.java
new file mode 100644
index 00000000..52ee0ae4
--- /dev/null
+++ b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/src/main/java/org/apache/dubbo/samples/ConsumerBootstrap.java
@@ -0,0 +1,48 @@
+/*
+ *
+ *   Licensed to the Apache Software Foundation (ASF) under one or more
+ *   contributor license agreements.  See the NOTICE file distributed with
+ *   this work for additional information regarding copyright ownership.
+ *   The ASF licenses this file to You under the Apache License, Version 2.0
+ *   (the "License"); you may not use this file except in compliance with
+ *   the License.  You may obtain a copy of the License at
+ *
+ *       http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *   Unless required by applicable law or agreed to in writing, software
+ *   distributed under the License is distributed on an "AS IS" BASIS,
+ *   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *   See the License for the specific language governing permissions and
+ *   limitations under the License.
+ *
+ */
+
+package org.apache.dubbo.samples;
+
+import org.apache.dubbo.config.spring.context.annotation.EnableDubbo;
+import org.apache.dubbo.samples.action.GreetingServiceConsumer;
+
+import org.springframework.context.annotation.AnnotationConfigApplicationContext;
+import org.springframework.context.annotation.ComponentScan;
+import org.springframework.context.annotation.Configuration;
+import org.springframework.context.annotation.PropertySource;
+
+public class ConsumerBootstrap {
+
+    public static void main(String[] args) {
+        AnnotationConfigApplicationContext context = new AnnotationConfigApplicationContext(ConsumerConfiguration.class);
+        context.start();
+        System.out.println("==================== dubbo invoke started ====================");
+        GreetingServiceConsumer greetingServiceConsumer = context.getBean(GreetingServiceConsumer.class);
+        greetingServiceConsumer.doSayHello("Kubernetes Api Server");
+        System.out.println("==================== dubbo invoke end ====================");
+    }
+
+    @Configuration
+    @EnableDubbo(scanBasePackages = "org.apache.dubbo.samples.action")
+    @PropertySource("classpath:/spring/dubbo-consumer.properties")
+    @ComponentScan(value = {"org.apache.dubbo.samples.action"})
+    static class ConsumerConfiguration {
+
+    }
+}
\ No newline at end of file
diff --git a/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/src/main/java/org/apache/dubbo/samples/action/GreetingServiceConsumer.java b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/src/main/java/org/apache/dubbo/samples/action/GreetingServiceConsumer.java
new file mode 100644
index 00000000..ac04d876
--- /dev/null
+++ b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/src/main/java/org/apache/dubbo/samples/action/GreetingServiceConsumer.java
@@ -0,0 +1,46 @@
+/*
+ *
+ *   Licensed to the Apache Software Foundation (ASF) under one or more
+ *   contributor license agreements.  See the NOTICE file distributed with
+ *   this work for additional information regarding copyright ownership.
+ *   The ASF licenses this file to You under the Apache License, Version 2.0
+ *   (the "License"); you may not use this file except in compliance with
+ *   the License.  You may obtain a copy of the License at
+ *
+ *       http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *   Unless required by applicable law or agreed to in writing, software
+ *   distributed under the License is distributed on an "AS IS" BASIS,
+ *   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *   See the License for the specific language governing permissions and
+ *   limitations under the License.
+ *
+ */
+
+package org.apache.dubbo.samples.action;
+
+import org.apache.dubbo.config.annotation.DubboReference;
+import org.apache.dubbo.samples.Greeter;
+import org.apache.dubbo.samples.GreeterReply;
+import org.apache.dubbo.samples.GreeterRequest;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.stereotype.Component;
+
+@Component("annotatedConsumer")
+public class GreetingServiceConsumer {
+
+    private static final Logger LOGGER = LoggerFactory.getLogger(GreetingServiceConsumer.class);
+
+    @DubboReference(version = "1.0.0", providedBy = "dubbo-samples-mesh-provider", url = "tri://dubbo-samples-mesh-provider:50052")
+    private Greeter greeter;
+
+    public void doSayHello(String name) {
+        final GreeterReply reply = greeter.greet(GreeterRequest.newBuilder()
+                .setName(name)
+                .build());
+        LOGGER.info("consumer Unary reply <-{}", reply);
+    }
+
+}
diff --git a/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/src/main/proto/greeter.proto b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/src/main/proto/greeter.proto
new file mode 100644
index 00000000..fee77610
--- /dev/null
+++ b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/src/main/proto/greeter.proto
@@ -0,0 +1,54 @@
+syntax = "proto3";
+
+option java_multiple_files = true;
+
+package org.apache.dubbo.samples;
+
+
+// The request message containing the user's name.
+message GreeterRequest {
+  string name = 1;
+}
+
+// The response message containing the greetings
+message GreeterReply {
+  string message = 1;
+}
+
+service Greeter{
+
+  // unary
+  rpc greet(GreeterRequest) returns (GreeterReply);
+
+  // bi streaming
+  rpc greetStream(stream GreeterRequest) returns (stream GreeterReply);
+
+  // serverStream
+  rpc greetServerStream(GreeterRequest) returns (stream GreeterReply);
+
+  //  upper case method
+  rpc UpperCaseGreet(GreeterRequest) returns (GreeterReply);
+
+  rpc greetWithAttachment (GreeterRequest) returns (GreeterReply);
+
+  rpc greetReturnBigAttachment (GreeterRequest) returns (GreeterReply);
+
+  rpc greetException(GreeterRequest) returns (GreeterReply);
+
+
+  rpc cancelBiStream(stream GreeterRequest) returns (stream GreeterReply);
+
+  rpc cancelBiStream2(stream GreeterRequest) returns (stream GreeterReply);
+
+  rpc compressorBiStream(stream GreeterRequest) returns (stream GreeterReply);
+
+  rpc clientCompressorBiStream(stream GreeterRequest) returns (stream GreeterReply);
+
+  rpc serverCompressorBiStream(stream GreeterRequest) returns (stream GreeterReply);
+
+
+  rpc cancelServerStream(GreeterRequest) returns (stream GreeterReply);
+
+  rpc queryCancelResult(GreeterRequest) returns (GreeterReply);
+
+}
diff --git a/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/src/main/resources/k8s/Deployment.yml b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/src/main/resources/k8s/Deployment.yml
new file mode 100644
index 00000000..32cdb620
--- /dev/null
+++ b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/src/main/resources/k8s/Deployment.yml
@@ -0,0 +1,45 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: dubbo-samples-apiserver-consumer
+  namespace: dubbo-demo
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: dubbo-samples-mesh-consumer
+  template:
+    metadata:
+      labels:
+        app: dubbo-samples-mesh-consumer
+    spec:
+      serviceAccountName: dubbo-sa
+      containers:
+        - name: server
+          image: 15841721425/dubbo-samples-mesh-consumer
+          imagePullPolicy: IfNotPresent
+          ports:
+            - containerPort: 20880
+              protocol: TCP
+            - containerPort: 22222
+              protocol: TCP
+            - containerPort: 31000
+              protocol: TCP
+          livenessProbe:
+            httpGet:
+              path: /live
+              port: 22222
+            initialDelaySeconds: 5
+            periodSeconds: 5
+          readinessProbe:
+            httpGet:
+              path: /ready
+              port: 22222
+            initialDelaySeconds: 5
+            periodSeconds: 5
+          startupProbe:
+            httpGet:
+              path: /startup
+              port: 22222
+            failureThreshold: 30
+            periodSeconds: 10
\ No newline at end of file
diff --git a/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/src/main/resources/k8s/Service.yml b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/src/main/resources/k8s/Service.yml
new file mode 100644
index 00000000..a1eb11b6
--- /dev/null
+++ b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/src/main/resources/k8s/Service.yml
@@ -0,0 +1,22 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: dubbo-samples-mesh-consumer
+  namespace: dubbo-demo
+spec:
+  clusterIP: None
+  selector:
+    app: dubbo-samples-mesh-consumer
+  ports:
+    - name: dubbo
+      protocol: TCP
+      port: 20880
+      targetPort: 20880
+    - name: qos
+      protocol: TCP
+      port: 22222
+      targetPort: 22222
+    - name: test
+      protocol: TCP
+      port: 31000
+      targetPort: 31000
\ No newline at end of file
diff --git a/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/src/main/resources/log4j.properties b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/src/main/resources/log4j.properties
new file mode 100644
index 00000000..9e04984b
--- /dev/null
+++ b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/src/main/resources/log4j.properties
@@ -0,0 +1,23 @@
+#
+#
+#   Licensed to the Apache Software Foundation (ASF) under one or more
+#   contributor license agreements.  See the NOTICE file distributed with
+#   this work for additional information regarding copyright ownership.
+#   The ASF licenses this file to You under the Apache License, Version 2.0
+#   (the "License"); you may not use this file except in compliance with
+#   the License.  You may obtain a copy of the License at
+#  
+#       http://www.apache.org/licenses/LICENSE-2.0
+#  
+#   Unless required by applicable law or agreed to in writing, software
+#   distributed under the License is distributed on an "AS IS" BASIS,
+#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#   See the License for the specific language governing permissions and
+#   limitations under the License.
+#  
+#
+log4j.rootLogger=info, stdout
+log4j.appender.stdout=org.apache.log4j.ConsoleAppender
+log4j.appender.stdout.Target=System.out
+log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
+log4j.appender.stdout.layout.ConversionPattern=[%d{dd/MM/yy hh:mm:ss:sss z}] %t %5p %c{2}: %m%n
\ No newline at end of file
diff --git a/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/src/main/resources/spring/dubbo-consumer.properties b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/src/main/resources/spring/dubbo-consumer.properties
new file mode 100644
index 00000000..aef4b9cb
--- /dev/null
+++ b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-consumer/src/main/resources/spring/dubbo-consumer.properties
@@ -0,0 +1,28 @@
+#
+#
+#   Licensed to the Apache Software Foundation (ASF) under one or more
+#   contributor license agreements.  See the NOTICE file distributed with
+#   this work for additional information regarding copyright ownership.
+#   The ASF licenses this file to You under the Apache License, Version 2.0
+#   (the "License"); you may not use this file except in compliance with
+#   the License.  You may obtain a copy of the License at
+#
+#       http://www.apache.org/licenses/LICENSE-2.0
+#
+#   Unless required by applicable law or agreed to in writing, software
+#   distributed under the License is distributed on an "AS IS" BASIS,
+#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#   See the License for the specific language governing permissions and
+#   limitations under the License.
+#
+#
+dubbo.application.name=dubbo-samples-mesh-consumer
+dubbo.application.metadataServicePort=20885
+#dubbo.registry.address=N/A
+dubbo.registry.address=kubernetes://DEFAULT_MASTER_HOST?registry-type=service&duplicate=false&namespace=dubbo-demo&trustCerts=true&subscribed-services=dubbo-samples-mesh-provider
+#dubbo.registry.address=zookeeper://101.42.243.67:2181?timeout=60000
+#dubbo.protocol.name=tri
+#dubbo.protocol.port=-1
+dubbo.consumer.timeout=30000
+dubbo.application.qosEnable=true
+dubbo.application.qosAcceptForeignIp=true
\ No newline at end of file
diff --git a/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/Dockerfile b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/Dockerfile
new file mode 100644
index 00000000..fe8d00b5
--- /dev/null
+++ b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/Dockerfile
@@ -0,0 +1,6 @@
+FROM openjdk:8-jre
+ADD ./target/dubbo-samples-mesh-provider-1.0-SNAPSHOT.jar /dubbo-samples-mesh-provider-1.0-SNAPSHOT.jar
+EXPOSE 50052
+EXPOSE 31000
+EXPOSE 22222
+CMD java -jar -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=31000 /dubbo-samples-mesh-provider-1.0-SNAPSHOT.jar
diff --git a/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/pom.xml b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/pom.xml
new file mode 100644
index 00000000..fd941afa
--- /dev/null
+++ b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/pom.xml
@@ -0,0 +1,198 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <modelVersion>4.0.0</modelVersion>
+
+    <groupId>org.apache.dubbo</groupId>
+    <artifactId>dubbo-samples-mesh-provider</artifactId>
+    <version>1.0-SNAPSHOT</version>
+
+    <properties>
+        <source.level>1.8</source.level>
+        <target.level>1.8</target.level>
+        <dubbo.version>3.1.0-SNAPSHOT</dubbo.version>
+        <grpc.version>1.31.1</grpc.version>
+        <spring.version>4.3.16.RELEASE</spring.version>
+        <junit.version>4.12</junit.version>
+        <maven-compiler-plugin.version>3.7.0</maven-compiler-plugin.version>
+        <protoc.version>3.19.4</protoc.version>
+        <dubbo.compiler.version>0.0.4.1-SNAPSHOT</dubbo.compiler.version>
+    </properties>
+
+    <dependencyManagement>
+        <dependencies>
+            <dependency>
+                <groupId>org.springframework</groupId>
+                <artifactId>spring-framework-bom</artifactId>
+                <version>${spring.version}</version>
+                <type>pom</type>
+                <scope>import</scope>
+            </dependency>
+            <dependency>
+                <groupId>org.apache.dubbo</groupId>
+                <artifactId>dubbo-bom</artifactId>
+                <version>${dubbo.version}</version>
+                <type>pom</type>
+                <scope>import</scope>
+            </dependency>
+        </dependencies>
+    </dependencyManagement>
+
+    <dependencies>
+        <dependency>
+            <groupId>org.apache.dubbo</groupId>
+            <artifactId>dubbo</artifactId>
+            <version>${dubbo.version}</version>
+            <!--            <version>3.1.0-SNAPSHOT</version>-->
+        </dependency>
+
+        <dependency>
+            <groupId>com.google.protobuf</groupId>
+            <artifactId>protobuf-java</artifactId>
+            <version>3.20.1</version>
+        </dependency>
+
+        <dependency>
+            <groupId>org.apache.dubbo.extensions</groupId>
+            <artifactId>dubbo-registry-kubernetes</artifactId>
+            <version>1.0.2-SNAPSHOT</version>
+        </dependency>
+
+        <dependency>
+            <groupId>org.apache.dubbo</groupId>
+            <artifactId>dubbo-dependencies-zookeeper-curator5</artifactId>
+            <version>${dubbo.version}</version>
+            <type>pom</type>
+        </dependency>
+
+        <dependency>
+            <groupId>io.grpc</groupId>
+            <artifactId>grpc-all</artifactId>
+            <version>${grpc.version}</version>
+        </dependency>
+
+        <dependency>
+            <groupId>junit</groupId>
+            <artifactId>junit</artifactId>
+            <version>${junit.version}</version>
+            <scope>test</scope>
+        </dependency>
+
+        <dependency>
+            <groupId>org.springframework</groupId>
+            <artifactId>spring-test</artifactId>
+            <scope>test</scope>
+        </dependency>
+
+        <dependency>
+            <groupId>org.springframework</groupId>
+            <artifactId>spring-context</artifactId>
+        </dependency>
+
+        <dependency>
+            <groupId>org.testcontainers</groupId>
+            <artifactId>testcontainers</artifactId>
+            <version>1.12.3</version>
+            <scope>test</scope>
+        </dependency>
+    </dependencies>
+
+    <profiles>
+        <!-- For jdk 11 above JavaEE annotation -->
+        <profile>
+            <id>javax.annotation</id>
+            <activation>
+                <jdk>[1.11,)</jdk>
+            </activation>
+            <dependencies>
+                <dependency>
+                    <groupId>javax.annotation</groupId>
+                    <artifactId>javax.annotation-api</artifactId>
+                    <version>1.3.2</version>
+                </dependency>
+            </dependencies>
+        </profile>
+    </profiles>
+
+    <build>
+        <extensions>
+            <extension>
+                <groupId>kr.motd.maven</groupId>
+                <artifactId>os-maven-plugin</artifactId>
+                <version>1.6.1</version>
+            </extension>
+        </extensions>
+        <plugins>
+            <plugin>
+                <groupId>org.apache.maven.plugins</groupId>
+                <artifactId>maven-compiler-plugin</artifactId>
+                <version>${maven-compiler-plugin.version}</version>
+                <configuration>
+                    <source>${source.level}</source>
+                    <target>${target.level}</target>
+                </configuration>
+            </plugin>
+            <plugin>
+                <groupId>org.springframework.boot</groupId>
+                <artifactId>spring-boot-maven-plugin</artifactId>
+                <version>2.6.6</version>
+                <executions>
+                    <execution>
+                        <goals>
+                            <goal>repackage</goal>
+                        </goals>
+                    </execution>
+                </executions>
+            </plugin>
+            <plugin>
+                <groupId>org.xolstice.maven.plugins</groupId>
+                <artifactId>protobuf-maven-plugin</artifactId>
+                <version>0.6.1</version>
+                <configuration>
+                    <protocArtifact>com.google.protobuf:protoc:${protoc.version}:exe:${os.detected.classifier}
+                    </protocArtifact>
+                    <pluginId>grpc-java</pluginId>
+                    <pluginArtifact>io.grpc:protoc-gen-grpc-java:${grpc.version}:exe:${os.detected.classifier}
+                    </pluginArtifact>
+                    <protocPlugins>
+                        <protocPlugin>
+                            <id>dubbo</id>
+                            <groupId>org.apache.dubbo</groupId>
+                            <artifactId>dubbo-compiler</artifactId>
+                            <version>${dubbo.compiler.version}</version>
+                            <mainClass>org.apache.dubbo.gen.tri.Dubbo3TripleGenerator</mainClass>
+                        </protocPlugin>
+                    </protocPlugins>
+                </configuration>
+                <executions>
+                    <execution>
+                        <goals>
+                            <goal>compile</goal>
+                            <goal>test-compile</goal>
+                            <goal>compile-custom</goal>
+                            <goal>test-compile-custom</goal>
+                        </goals>
+                    </execution>
+                </executions>
+            </plugin>
+        </plugins>
+    </build>
+
+    <repositories>
+        <repository>
+            <id>apache.snapshots.https</id>
+            <name>Apache Development Snapshot Repository</name>
+            <url>https://repository.apache.org/content/repositories/snapshots</url>
+            <layout>default</layout>
+            <releases>
+                <enabled>false</enabled>
+            </releases>
+            <snapshots>
+                <enabled>true</enabled>
+                <updatePolicy>daily</updatePolicy>
+            </snapshots>
+        </repository>
+    </repositories>
+
+</project>
\ No newline at end of file
diff --git a/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/src/main/java/org/apache/dubbo/samples/ProviderBootstrap.java b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/src/main/java/org/apache/dubbo/samples/ProviderBootstrap.java
new file mode 100644
index 00000000..0619972b
--- /dev/null
+++ b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/src/main/java/org/apache/dubbo/samples/ProviderBootstrap.java
@@ -0,0 +1,44 @@
+/*
+ *
+ *   Licensed to the Apache Software Foundation (ASF) under one or more
+ *   contributor license agreements.  See the NOTICE file distributed with
+ *   this work for additional information regarding copyright ownership.
+ *   The ASF licenses this file to You under the Apache License, Version 2.0
+ *   (the "License"); you may not use this file except in compliance with
+ *   the License.  You may obtain a copy of the License at
+ *
+ *       http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *   Unless required by applicable law or agreed to in writing, software
+ *   distributed under the License is distributed on an "AS IS" BASIS,
+ *   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *   See the License for the specific language governing permissions and
+ *   limitations under the License.
+ *
+ */
+
+package org.apache.dubbo.samples;
+
+import org.apache.dubbo.config.spring.context.annotation.EnableDubbo;
+
+import org.springframework.context.annotation.AnnotationConfigApplicationContext;
+import org.springframework.context.annotation.Configuration;
+import org.springframework.context.annotation.PropertySource;
+
+import java.util.concurrent.CountDownLatch;
+
+public class ProviderBootstrap {
+
+    public static void main(String[] args) throws Exception {
+        AnnotationConfigApplicationContext context = new AnnotationConfigApplicationContext(ProviderConfiguration.class);
+        context.start();
+        System.out.println("=============== dubbo service started ===============");
+        new CountDownLatch(1).await();
+    }
+
+    @Configuration
+    @EnableDubbo(scanBasePackages = "org.apache.dubbo.samples.impl")
+    @PropertySource("classpath:/spring/dubbo-provider.properties")
+    static class ProviderConfiguration {
+    }
+}
diff --git a/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/src/main/java/org/apache/dubbo/samples/impl/GreeterImpl.java b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/src/main/java/org/apache/dubbo/samples/impl/GreeterImpl.java
new file mode 100644
index 00000000..162d0130
--- /dev/null
+++ b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/src/main/java/org/apache/dubbo/samples/impl/GreeterImpl.java
@@ -0,0 +1,256 @@
+/*
+ *
+ *   Licensed to the Apache Software Foundation (ASF) under one or more
+ *   contributor license agreements.  See the NOTICE file distributed with
+ *   this work for additional information regarding copyright ownership.
+ *   The ASF licenses this file to You under the Apache License, Version 2.0
+ *   (the "License"); you may not use this file except in compliance with
+ *   the License.  You may obtain a copy of the License at
+ *
+ *       http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *   Unless required by applicable law or agreed to in writing, software
+ *   distributed under the License is distributed on an "AS IS" BASIS,
+ *   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *   See the License for the specific language governing permissions and
+ *   limitations under the License.
+ *
+ */
+
+package org.apache.dubbo.samples.impl;
+
+import org.apache.dubbo.common.stream.StreamObserver;
+import org.apache.dubbo.config.annotation.DubboService;
+import org.apache.dubbo.rpc.RpcContext;
+import org.apache.dubbo.rpc.protocol.tri.ServerStreamObserver;
+import org.apache.dubbo.samples.DubboGreeterTriple;
+import org.apache.dubbo.samples.GreeterReply;
+import org.apache.dubbo.samples.GreeterRequest;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * @author conghuhu
+ */
+@DubboService(version = "1.0.0")
+public class GreeterImpl extends DubboGreeterTriple.GreeterImplBase {
+    private static final Logger LOGGER = LoggerFactory.getLogger(GreeterImpl.class);
+
+    private final String serverName;
+    public static final Map<String, Boolean> cancelResultMap = new HashMap<>();
+
+    public GreeterImpl() {
+        this.serverName = "test dubbo tri k8s";
+    }
+
+    public GreeterImpl(String serverName) {
+        this.serverName = serverName;
+    }
+
+    @Override
+    public GreeterReply greet(GreeterRequest request) {
+        LOGGER.info("Server {} received greet request {}", serverName, request);
+        return GreeterReply.newBuilder()
+                .setMessage("hello," + request.getName())
+                .build();
+    }
+
+    @Override
+    public GreeterReply greetWithAttachment(GreeterRequest request) {
+        final String key = "user-attachment";
+        final String value = "hello," + RpcContext.getServerAttachment().getAttachment(key);
+        RpcContext.getServerContext().setObjectAttachment(key, value);
+        return GreeterReply.newBuilder().setMessage("hello," + request.getName()).build();
+    }
+
+    @Override
+    public GreeterReply greetReturnBigAttachment(GreeterRequest request) {
+        final String key = "user-attachment";
+        StringBuilder stringBuilder = new StringBuilder();
+        for (int i = 0; i < 8000; i++) {
+            stringBuilder.append(i);
+        }
+        String value = stringBuilder.toString();
+        RpcContext.getServerContext().setObjectAttachment(key, value);
+        return GreeterReply.newBuilder().setMessage("hello," + request.getName()).build();
+    }
+
+    @Override
+    public void cancelServerStream(GreeterRequest request,
+                                   StreamObserver<GreeterReply> replyStream) {
+        RpcContext.getCancellationContext().addListener(context -> {
+            LOGGER.info("cancel--cancelServerStream");
+            cancelResultMap.put("cancelServerStream", true);
+        });
+        for (int i = 0; i < 10; i++) {
+            replyStream.onNext(GreeterReply.newBuilder()
+                    .setMessage(request.getName() + "--" + i)
+                    .build());
+        }
+    }
+
+    @Override
+    public StreamObserver<GreeterRequest> cancelBiStream(StreamObserver<GreeterReply> replyStream) {
+        RpcContext.getCancellationContext()
+                .addListener(context -> {
+                    LOGGER.info("cancel--cancelBiStream");
+                    cancelResultMap.put("cancelBiStream", true);
+                });
+        return new StreamObserver<GreeterRequest>() {
+            @Override
+            public void onNext(GreeterRequest data) {
+                LOGGER.info("Bi-Stream-Request:" + data.getName());
+                replyStream.onNext(GreeterReply.newBuilder()
+                        .setMessage(data.getName())
+                        .build());
+            }
+
+            @Override
+            public void onError(Throwable throwable) {
+                throwable.printStackTrace();
+                replyStream.onError(new IllegalStateException("Stream err"));
+            }
+
+            @Override
+            public void onCompleted() {
+                // replyStream.onCompleted();
+            }
+        };
+    }
+
+    @Override
+    public StreamObserver<GreeterRequest> cancelBiStream2(
+            StreamObserver<GreeterReply> replyStream) {
+        RpcContext.getCancellationContext()
+                .addListener(context -> {
+                    LOGGER.info("cancel--cancelBiStream2");
+                    cancelResultMap.put("cancelBiStream2", true);
+                });
+        return new StreamObserver<GreeterRequest>() {
+            @Override
+            public void onNext(GreeterRequest data) {
+                LOGGER.info("Bi-Stream-Request:" + data.getName());
+                replyStream.onNext(GreeterReply.newBuilder()
+                        .setMessage(data.getName())
+                        .build());
+            }
+
+            @Override
+            public void onError(Throwable throwable) {
+                throwable.printStackTrace();
+                replyStream.onError(new IllegalStateException("Stream err"));
+            }
+
+            @Override
+            public void onCompleted() {
+                // replyStream.onCompleted();
+            }
+        };
+    }
+
+    @Override
+    public StreamObserver<GreeterRequest> compressorBiStream(
+            StreamObserver<GreeterReply> replyStream) {
+        ServerStreamObserver<GreeterReply> replyServerStreamObserver = (ServerStreamObserver<GreeterReply>) replyStream;
+        replyServerStreamObserver.setCompression("gzip");
+        return getGreeterRequestStreamObserver(replyServerStreamObserver);
+    }
+
+    private StreamObserver<GreeterRequest> getGreeterRequestStreamObserver(
+            StreamObserver<GreeterReply> streamObserver) {
+        return new StreamObserver<GreeterRequest>() {
+            @Override
+            public void onNext(GreeterRequest data) {
+                streamObserver.onNext(GreeterReply.newBuilder()
+                        .setMessage(data.getName())
+                        .build());
+            }
+
+            @Override
+            public void onError(Throwable throwable) {
+                throwable.printStackTrace();
+                streamObserver.onError(new IllegalStateException("Stream err"));
+            }
+
+            @Override
+            public void onCompleted() {
+                streamObserver.onCompleted();
+            }
+        };
+    }
+
+    @Override
+    public StreamObserver<GreeterRequest> clientCompressorBiStream(
+            StreamObserver<GreeterReply> replyStream) {
+        ServerStreamObserver<GreeterReply> replyServerStreamObserver = (ServerStreamObserver<GreeterReply>) replyStream;
+        return getGreeterRequestStreamObserver(replyServerStreamObserver);
+    }
+
+    @Override
+    public StreamObserver<GreeterRequest> serverCompressorBiStream(
+            StreamObserver<GreeterReply> replyStream) {
+        ServerStreamObserver<GreeterReply> replyServerStreamObserver = (ServerStreamObserver<GreeterReply>) replyStream;
+        replyServerStreamObserver.setCompression("gzip");
+        return getGreeterRequestStreamObserver(replyServerStreamObserver);
+    }
+
+    @Override
+    public GreeterReply queryCancelResult(GreeterRequest request) {
+        boolean canceled = cancelResultMap.getOrDefault(request.getName(), false);
+        return GreeterReply.newBuilder()
+                .setMessage(String.valueOf(canceled))
+                .build();
+    }
+
+
+    public GreeterReply greetException(GreeterRequest request) {
+        RpcContext.getServerContext().setAttachment("str", "str")
+                .setAttachment("integer", 1)
+                .setAttachment("raw", new byte[]{1, 2, 3, 4});
+        throw new RuntimeException("Biz Exception");
+    }
+
+    @Override
+    public StreamObserver<GreeterRequest> greetStream(StreamObserver<GreeterReply> replyStream) {
+        return new StreamObserver<GreeterRequest>() {
+            int n = 0;
+
+            @Override
+            public void onNext(GreeterRequest data) {
+                n++;
+                LOGGER.info(data.getName() + " " + n);
+                replyStream.onNext(GreeterReply.newBuilder()
+                        .setMessage(data.getName() + " " + n)
+                        .build());
+            }
+
+            @Override
+            public void onError(Throwable throwable) {
+                throwable.printStackTrace();
+                replyStream.onError(new IllegalStateException("Stream err"));
+            }
+
+            @Override
+            public void onCompleted() {
+                LOGGER.info("[greetStream] onCompleted");
+                replyStream.onCompleted();
+            }
+        };
+    }
+
+    @Override
+    public void greetServerStream(GreeterRequest request,
+                                  StreamObserver<GreeterReply> replyStream) {
+        for (int i = 0; i < 10; i++) {
+            replyStream.onNext(GreeterReply.newBuilder()
+                    .setMessage(request.getName() + "--" + i)
+                    .build());
+        }
+        replyStream.onCompleted();
+    }
+}
+
diff --git a/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/src/main/proto/greeter.proto b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/src/main/proto/greeter.proto
new file mode 100644
index 00000000..fee77610
--- /dev/null
+++ b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/src/main/proto/greeter.proto
@@ -0,0 +1,54 @@
+syntax = "proto3";
+
+option java_multiple_files = true;
+
+package org.apache.dubbo.samples;
+
+
+// The request message containing the user's name.
+message GreeterRequest {
+  string name = 1;
+}
+
+// The response message containing the greetings
+message GreeterReply {
+  string message = 1;
+}
+
+service Greeter{
+
+  // unary
+  rpc greet(GreeterRequest) returns (GreeterReply);
+
+  // bi streaming
+  rpc greetStream(stream GreeterRequest) returns (stream GreeterReply);
+
+  // serverStream
+  rpc greetServerStream(GreeterRequest) returns (stream GreeterReply);
+
+  //  upper case method
+  rpc UpperCaseGreet(GreeterRequest) returns (GreeterReply);
+
+  rpc greetWithAttachment (GreeterRequest) returns (GreeterReply);
+
+  rpc greetReturnBigAttachment (GreeterRequest) returns (GreeterReply);
+
+  rpc greetException(GreeterRequest) returns (GreeterReply);
+
+
+  rpc cancelBiStream(stream GreeterRequest) returns (stream GreeterReply);
+
+  rpc cancelBiStream2(stream GreeterRequest) returns (stream GreeterReply);
+
+  rpc compressorBiStream(stream GreeterRequest) returns (stream GreeterReply);
+
+  rpc clientCompressorBiStream(stream GreeterRequest) returns (stream GreeterReply);
+
+  rpc serverCompressorBiStream(stream GreeterRequest) returns (stream GreeterReply);
+
+
+  rpc cancelServerStream(GreeterRequest) returns (stream GreeterReply);
+
+  rpc queryCancelResult(GreeterRequest) returns (GreeterReply);
+
+}
diff --git a/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/src/main/resources/k8s/Deployment.yml b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/src/main/resources/k8s/Deployment.yml
new file mode 100644
index 00000000..dbed26cb
--- /dev/null
+++ b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/src/main/resources/k8s/Deployment.yml
@@ -0,0 +1,45 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: dubbo-samples-mesh-provider
+  namespace: dubbo-demo
+spec:
+  replicas: 2
+  selector:
+    matchLabels:
+      app: dubbo-samples-mesh-provider
+  template:
+    metadata:
+      labels:
+        app: dubbo-samples-mesh-provider
+    spec:
+      serviceAccountName: dubbo-sa
+      containers:
+        - name: server
+          image: 15841721425/dubbo-samples-mesh-provider
+          imagePullPolicy: IfNotPresent
+          ports:
+            - containerPort: 50052
+              protocol: TCP
+            - containerPort: 22222
+              protocol: TCP
+            - containerPort: 31000
+              protocol: TCP
+          livenessProbe:
+            httpGet:
+              path: /live
+              port: 22222
+            initialDelaySeconds: 5
+            periodSeconds: 5
+          readinessProbe:
+            httpGet:
+              path: /ready
+              port: 22222
+            initialDelaySeconds: 5
+            periodSeconds: 5
+          startupProbe:
+            httpGet:
+              path: /startup
+              port: 22222
+            failureThreshold: 30
+            periodSeconds: 10
\ No newline at end of file
diff --git a/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/src/main/resources/k8s/Service.yml b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/src/main/resources/k8s/Service.yml
new file mode 100644
index 00000000..a2c403ba
--- /dev/null
+++ b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/src/main/resources/k8s/Service.yml
@@ -0,0 +1,26 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: dubbo-samples-mesh-provider
+  namespace: dubbo-demo
+spec:
+  clusterIP: None
+  selector:
+    app: dubbo-samples-mesh-provider
+  ports:
+    - name: tri
+      protocol: TCP
+      port: 50052
+      targetPort: 50052
+    - name: dubbo
+      protocol: TCP
+      port: 20880
+      targetPort: 20880
+    - name: qos
+      protocol: TCP
+      port: 22222
+      targetPort: 22222
+    - name: test
+      protocol: TCP
+      port: 31000
+      targetPort: 31000
\ No newline at end of file
diff --git a/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/src/main/resources/k8s/ServiceAccount.yml b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/src/main/resources/k8s/ServiceAccount.yml
new file mode 100644
index 00000000..bd7d37f2
--- /dev/null
+++ b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/src/main/resources/k8s/ServiceAccount.yml
@@ -0,0 +1,36 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: dubbo-demo
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  namespace: dubbo-demo
+  name: dubbo-role
+rules:
+  - apiGroups: [ "" ]
+    resources: [ "pods" ]
+    verbs: [ "get", "watch", "list", "update", "patch" ]
+  - apiGroups: [ "", "service.dubbo.apache.org" ]
+    resources: [ "services", "endpoints", "virtualservices", "destinationrules" ]
+    verbs: [ "get", "watch", "list" ]
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: dubbo-sa
+  namespace: dubbo-demo
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: dubbo-sa-bind
+  namespace: dubbo-demo
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: dubbo-role
+subjects:
+  - kind: ServiceAccount
+    name: dubbo-sa
\ No newline at end of file
diff --git a/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/src/main/resources/log4j.properties b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/src/main/resources/log4j.properties
new file mode 100644
index 00000000..9e04984b
--- /dev/null
+++ b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/src/main/resources/log4j.properties
@@ -0,0 +1,23 @@
+#
+#
+#   Licensed to the Apache Software Foundation (ASF) under one or more
+#   contributor license agreements.  See the NOTICE file distributed with
+#   this work for additional information regarding copyright ownership.
+#   The ASF licenses this file to You under the Apache License, Version 2.0
+#   (the "License"); you may not use this file except in compliance with
+#   the License.  You may obtain a copy of the License at
+#  
+#       http://www.apache.org/licenses/LICENSE-2.0
+#  
+#   Unless required by applicable law or agreed to in writing, software
+#   distributed under the License is distributed on an "AS IS" BASIS,
+#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#   See the License for the specific language governing permissions and
+#   limitations under the License.
+#  
+#
+log4j.rootLogger=info, stdout
+log4j.appender.stdout=org.apache.log4j.ConsoleAppender
+log4j.appender.stdout.Target=System.out
+log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
+log4j.appender.stdout.layout.ConversionPattern=[%d{dd/MM/yy hh:mm:ss:sss z}] %t %5p %c{2}: %m%n
\ No newline at end of file
diff --git a/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/src/main/resources/spring/dubbo-provider.properties b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/src/main/resources/spring/dubbo-provider.properties
new file mode 100644
index 00000000..a4ec19f4
--- /dev/null
+++ b/dubbo-samples-mesh-k8s/dubbo-samples-mesh-provider/src/main/resources/spring/dubbo-provider.properties
@@ -0,0 +1,30 @@
+#
+#
+#   Licensed to the Apache Software Foundation (ASF) under one or more
+#   contributor license agreements.  See the NOTICE file distributed with
+#   this work for additional information regarding copyright ownership.
+#   The ASF licenses this file to You under the Apache License, Version 2.0
+#   (the "License"); you may not use this file except in compliance with
+#   the License.  You may obtain a copy of the License at
+#
+#       http://www.apache.org/licenses/LICENSE-2.0
+#
+#   Unless required by applicable law or agreed to in writing, software
+#   distributed under the License is distributed on an "AS IS" BASIS,
+#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#   See the License for the specific language governing permissions and
+#   limitations under the License.
+#
+#
+dubbo.application.name=dubbo-samples-mesh-provider
+dubbo.application.metadataServicePort=20885
+#dubbo.registry.address=N/A
+dubbo.registry.address=kubernetes://DEFAULT_MASTER_HOST?registry-type=service&duplicate=false&namespace=dubbo-demo&trustCerts=true
+#dubbo.registry.address=zookeeper://101.42.243.67:2181?timeout=60000
+dubbo.protocol.name=tri
+dubbo.protocol.port=50052
+# qos??
+dubbo.application.qosEnable=true
+dubbo.application.qosAcceptForeignIp=true
+# token??
+#dubbo.provider.token=true
\ No newline at end of file
diff --git a/dubbo-samples-mesh-k8s/pom.xml b/dubbo-samples-mesh-k8s/pom.xml
new file mode 100644
index 00000000..2cf98be2
--- /dev/null
+++ b/dubbo-samples-mesh-k8s/pom.xml
@@ -0,0 +1,50 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+  ~
+  ~   Licensed to the Apache Software Foundation (ASF) under one or more
+  ~   contributor license agreements.  See the NOTICE file distributed with
+  ~   this work for additional information regarding copyright ownership.
+  ~   The ASF licenses this file to You under the Apache License, Version 2.0
+  ~   (the "License"); you may not use this file except in compliance with
+  ~   the License.  You may obtain a copy of the License at
+  ~
+  ~       http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~   Unless required by applicable law or agreed to in writing, software
+  ~   distributed under the License is distributed on an "AS IS" BASIS,
+  ~   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  ~   See the License for the specific language governing permissions and
+  ~   limitations under the License.
+  ~
+  -->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <groupId>org.apache.dubbo</groupId>
+    <version>1.0-SNAPSHOT</version>
+    <modules>
+        <module>dubbo-samples-mesh-consumer</module>
+        <module>dubbo-samples-mesh-provider</module>
+    </modules>
+
+    <modelVersion>4.0.0</modelVersion>
+    <packaging>pom</packaging>
+    <artifactId>dubbo-samples-mesh-k8s</artifactId>
+
+    <repositories>
+        <repository>
+            <id>apache.snapshots.https</id>
+            <name>Apache Development Snapshot Repository</name>
+            <url>https://repository.apache.org/content/repositories/snapshots</url>
+            <layout>default</layout>
+            <releases>
+                <enabled>false</enabled>
+            </releases>
+            <snapshots>
+                <enabled>true</enabled>
+                <updatePolicy>daily</updatePolicy>
+            </snapshots>
+        </repository>
+    </repositories>
+
+</project>
\ No newline at end of file
diff --git a/pom.xml b/pom.xml
index c938c093..b1ae9557 100644
--- a/pom.xml
+++ b/pom.xml
@@ -48,6 +48,7 @@
         <module>dubbo-samples-jetty</module>
         <module>dubbo-samples-local</module>
         <module>dubbo-samples-merge</module>
+        <module>dubbo-samples-mesh-k8s</module>
         <module>dubbo-samples-metadata-report</module>
         <module>dubbo-samples-migration</module>
         <module>dubbo-samples-mock</module>


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org