You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by pe...@apache.org on 2018/09/25 12:23:42 UTC

[incubator-skywalking] branch master updated: Alarm module code core ready (#1644)

This is an automated email from the ASF dual-hosted git repository.

pengys pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-skywalking.git


The following commit(s) were added to refs/heads/master by this push:
     new 933ae20  Alarm module code core ready (#1644)
933ae20 is described below

commit 933ae20b1cdd9d2772b7a841717a9acd1c3defdb
Author: 吴晟 Wu Sheng <wu...@foxmail.com>
AuthorDate: Tue Sep 25 20:23:36 2018 +0800

    Alarm module code core ready (#1644)
    
    * Support alarm message.
    
    * Fix document words.
    
    * Support alarm hook.
    
    * Fix javadoc issue.
    
    * Sync submodule
    
    * Sync submodule
---
 .../optional-plugins/trace-ignore-plugin/README.md |  17 ---
 .../trace-ignore-plugin/README_CN.md               |  18 ---
 .../optional-plugins/trace-ignore-plugin/pom.xml   |  29 -----
 docs/en/setup/service-agent/java-agent/README.md   |   2 +-
 .../agent-optional-plugins/trace-ignore-plugin.md  |   6 +-
 .../core/alarm/provider/AlarmMessageFormatter.java |  97 ++++++++++++++++
 .../oap/server/core/alarm/provider/AlarmRule.java  |   1 +
 .../server/core/alarm/provider/RulesReader.java    |   1 +
 .../server/core/alarm/provider/RunningRule.java    |  13 ++-
 .../core/alarm/provider/WebhookCallback.java       |  61 +++++++++++
 .../alarm/provider/AlarmMessageFormatterTest.java  |  83 ++++++++++++++
 .../core/alarm/provider/AlarmRuleInitTest.java     |   3 +
 .../core/alarm/provider/RunningRuleTest.java       |  12 +-
 .../core/alarm/provider/WebhookCallbackTest.java   | 122 +++++++++++++++++++++
 .../src/test/resources/alarm-settings.yml          |   1 +
 .../oap/server/core/alarm/AlarmMessage.java        |   7 ++
 16 files changed, 399 insertions(+), 74 deletions(-)

diff --git a/apm-sniffer/optional-plugins/trace-ignore-plugin/README.md b/apm-sniffer/optional-plugins/trace-ignore-plugin/README.md
deleted file mode 100644
index 89195e4..0000000
--- a/apm-sniffer/optional-plugins/trace-ignore-plugin/README.md
+++ /dev/null
@@ -1,17 +0,0 @@
-## Support custom trace ignore
-Here is an optional plugin `apm-trace-ignore-plugin`
-
-## Introduce
-- The purpose of this plugin is to filter custom services which are expected to ignore from the tracing system.
-- You can set up multiple URL path patterns, The services matches these patterns means won't be traced and analysis by agent and collector.
-- The current matching rule follows `Ant Path` match style , like `/path/*`, `/path/**`, `/path/?`.
-- Copy `apm-trace-ignore-plugin-x.jar` to `agent/plugins`, restarting the `agent` can effect the plugin.                                                                                                         
-
-## How to configure
-There are two ways to configure ignore patterns. Settings through system env has higher priority.
- 1. Set through the system environment variable,you need to add `skywalking.trace.ignore_path` to the system variables, the value is the path that you need to ignore, multiple paths should be separated by `,`
- 2. Copy`/agent/optional-plugins/apm-trace-ignore-plugin/apm-trace-ignore-plugin.config` to `/agent/config/` dir, and add rules to filter traces
-```
-trace.ignore_path=/your/path/1/**,/your/path/2/**
-```
-
diff --git a/apm-sniffer/optional-plugins/trace-ignore-plugin/README_CN.md b/apm-sniffer/optional-plugins/trace-ignore-plugin/README_CN.md
deleted file mode 100644
index 3a21bec..0000000
--- a/apm-sniffer/optional-plugins/trace-ignore-plugin/README_CN.md
+++ /dev/null
@@ -1,18 +0,0 @@
-## 个性化服务过滤
-提供了一个可选插件 `apm-trace-ignore-plugin`
-
-## 介绍
-- 这个插件的作用是对追踪的个性化服务过滤.
-- 你可以设置多个需要忽略的URL路径, 意味着包含这些路径的`追踪信息`不会被`agent`发送到 `collector`.
-- 当前的路径匹配规则是 `Ant Path`匹配风格 , 例如 `/path/*`, `/path/**`, `/path/?`.
-- 将`apm-trace-ignore-plugin-x.jar`拷贝到`agent/plugins`后,重启探针即可生效
-- [Skywalking-使用可选插件 apm-trace-ignore-plugin](https://blog.csdn.net/u013095337/article/details/80452088) 有详细使用介绍
-
-## 如何配置路径 
-有两种配置方式,可使用任意一种,配置生效的优先级从高到低
- 1. 在系统环境变量中配置,你需要在系统变量中添加`skywalking.trace.ignore_path`, 值是你需要忽略的路径,多个以`,`号分隔
- 2. 将`/agent/optional-plugins/apm-trace-ignore-plugin/apm-trace-ignore-plugin.config` 复制或剪切到 `/agent/config/` 目录下,加上配置
-```
-trace.ignore_path=/your/path/1/**,/your/path/2/**
-```
-
diff --git a/apm-sniffer/optional-plugins/trace-ignore-plugin/pom.xml b/apm-sniffer/optional-plugins/trace-ignore-plugin/pom.xml
index 7f2701c..ac0fee7 100644
--- a/apm-sniffer/optional-plugins/trace-ignore-plugin/pom.xml
+++ b/apm-sniffer/optional-plugins/trace-ignore-plugin/pom.xml
@@ -30,33 +30,4 @@
 
     <name>apm-trace-ignore-plugin</name>
     <url>http://maven.apache.org</url>
-
-    <build>
-        <plugins>
-            <plugin>
-                <artifactId>maven-antrun-plugin</artifactId>
-                <executions>
-                    <execution>
-                        <phase>package</phase>
-                        <goals>
-                            <goal>run</goal>
-                        </goals>
-                        <configuration combine.self="override">
-                            <tasks>
-                                <taskdef resource="net/sf/antcontrib/antcontrib.properties" classpathref="maven.runtime.classpath" />
-                                <mkdir dir="${optional.plugin.dest.dir}/${project.artifactId}" />
-                                <!-- copy jar -->
-                                <copy file="${project.build.directory}/${project.artifactId}-${project.version}.jar" tofile="${optional.plugin.dest.dir}/${project.artifactId}/${project.artifactId}-${project.version}.jar" overwrite="true" />
-                                <!-- copy config file -->
-                                <copy file="${project.basedir}/${project.name}.config" tofile="${optional.plugin.dest.dir}/${project.name}/${project.name}.config" overwrite="true" />
-                                <!-- copy introduction -->
-                                <copy file="${project.basedir}/README.md" tofile="${optional.plugin.dest.dir}/${project.name}/README.md" overwrite="true" />
-                                <copy file="${project.basedir}/README_CN.md" tofile="${optional.plugin.dest.dir}/${project.name}/README_CN.md" overwrite="true" />
-                            </tasks>
-                        </configuration>
-                    </execution>
-                </executions>
-            </plugin>
-        </plugins>
-    </build>
 </project>
diff --git a/docs/en/setup/service-agent/java-agent/README.md b/docs/en/setup/service-agent/java-agent/README.md
index 3d8641d..1352277 100644
--- a/docs/en/setup/service-agent/java-agent/README.md
+++ b/docs/en/setup/service-agent/java-agent/README.md
@@ -53,7 +53,7 @@ For using these plugins, you need to compile source codes by yourself, or copy t
 Now, we have the following known plugins.
 * [Trace Spring beans](agent-optional-plugins/Spring-bean-plugins.md)
 * [Trace Oracle and Resin](agent-optional-plugins/Oracle-Resin-plugins.md)
-* [Filter traces through custom services](agent-optional-plugins/trace-ignore-plugin.md)
+* [Filter traces through specified endpoint name patterns](agent-optional-plugins/trace-ignore-plugin.md)
 
 ## Advanced Features
 * Set the settings through system properties for config file override. Read [setting override](Setting-override.md).
diff --git a/docs/en/setup/service-agent/java-agent/agent-optional-plugins/trace-ignore-plugin.md b/docs/en/setup/service-agent/java-agent/agent-optional-plugins/trace-ignore-plugin.md
index 89195e4..6da90cf 100644
--- a/docs/en/setup/service-agent/java-agent/agent-optional-plugins/trace-ignore-plugin.md
+++ b/docs/en/setup/service-agent/java-agent/agent-optional-plugins/trace-ignore-plugin.md
@@ -2,9 +2,9 @@
 Here is an optional plugin `apm-trace-ignore-plugin`
 
 ## Introduce
-- The purpose of this plugin is to filter custom services which are expected to ignore from the tracing system.
-- You can set up multiple URL path patterns, The services matches these patterns means won't be traced and analysis by agent and collector.
-- The current matching rule follows `Ant Path` match style , like `/path/*`, `/path/**`, `/path/?`.
+- The purpose of this plugin is to filter endpoint which are expected to be ignored by the tracing system.
+- You can setup multiple URL path patterns, The endpoints match these patterns wouldn't be traced.
+- The current matching rules follow `Ant Path` match style , like `/path/*`, `/path/**`, `/path/?`.
 - Copy `apm-trace-ignore-plugin-x.jar` to `agent/plugins`, restarting the `agent` can effect the plugin.                                                                                                         
 
 ## How to configure
diff --git a/oap-server/server-alarm-plugin/src/main/java/org/apache/skywalking/oap/server/core/alarm/provider/AlarmMessageFormatter.java b/oap-server/server-alarm-plugin/src/main/java/org/apache/skywalking/oap/server/core/alarm/provider/AlarmMessageFormatter.java
new file mode 100644
index 0000000..a209b0c
--- /dev/null
+++ b/oap-server/server-alarm-plugin/src/main/java/org/apache/skywalking/oap/server/core/alarm/provider/AlarmMessageFormatter.java
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.core.alarm.provider;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.skywalking.oap.server.core.alarm.MetaInAlarm;
+
+/**
+ * This is a formatter especially for alarm message.
+ *
+ * Format string in alarm-settings.yml, such as:
+ *
+ * - Successful rate of endpoint {name} is lower than 75%
+ *
+ * @author wusheng
+ */
+public class AlarmMessageFormatter {
+    private List<String> formatSegments;
+    private List<ValueFrom> valueFroms;
+
+    public AlarmMessageFormatter(String format) {
+        if (format == null) {
+            format = "";
+        }
+        formatSegments = new ArrayList<>();
+        this.valueFroms = new ArrayList<>();
+        boolean match = false;
+        int idx = 0;
+        do {
+            match = false;
+            int start = format.indexOf("{", idx);
+            if (start > -1) {
+                int end = format.indexOf("}", start);
+                if (end > -1) {
+
+                    String name = format.substring(start + 1, end);
+                    switch (name) {
+                        case "id":
+                            valueFroms.add(ValueFrom.ID);
+                            break;
+                        case "name":
+                            valueFroms.add(ValueFrom.NAME);
+                            break;
+                        default:
+                            throw new IllegalArgumentException("Var [" + name + "] in alarm message [" + format + "] is illegal");
+                    }
+                    formatSegments.add(format.substring(idx, start));
+                    idx = end + 1;
+                    match = true;
+                }
+            }
+
+            if (!match) {
+                formatSegments.add(format.substring(idx));
+            }
+        }
+        while (match);
+    }
+
+    public String format(MetaInAlarm meta) {
+        StringBuilder message = new StringBuilder();
+        for (int i = 0; i < formatSegments.size(); i++) {
+            message.append(formatSegments.get(i));
+            if (i != formatSegments.size() - 1) {
+                switch (valueFroms.get(i)) {
+                    case ID:
+                        message.append(meta.getId0());
+                        break;
+                    case NAME:
+                        message.append(meta.getName());
+                }
+            }
+        }
+        return message.toString();
+    }
+
+    private enum ValueFrom {
+        ID, NAME
+    }
+}
diff --git a/oap-server/server-alarm-plugin/src/main/java/org/apache/skywalking/oap/server/core/alarm/provider/AlarmRule.java b/oap-server/server-alarm-plugin/src/main/java/org/apache/skywalking/oap/server/core/alarm/provider/AlarmRule.java
index 7f3e1dc..03f299f 100644
--- a/oap-server/server-alarm-plugin/src/main/java/org/apache/skywalking/oap/server/core/alarm/provider/AlarmRule.java
+++ b/oap-server/server-alarm-plugin/src/main/java/org/apache/skywalking/oap/server/core/alarm/provider/AlarmRule.java
@@ -38,4 +38,5 @@ public class AlarmRule {
     private int period;
     private int count;
     private int silencePeriod;
+    private String message;
 }
diff --git a/oap-server/server-alarm-plugin/src/main/java/org/apache/skywalking/oap/server/core/alarm/provider/RulesReader.java b/oap-server/server-alarm-plugin/src/main/java/org/apache/skywalking/oap/server/core/alarm/provider/RulesReader.java
index 77de2e5..e2a2873 100644
--- a/oap-server/server-alarm-plugin/src/main/java/org/apache/skywalking/oap/server/core/alarm/provider/RulesReader.java
+++ b/oap-server/server-alarm-plugin/src/main/java/org/apache/skywalking/oap/server/core/alarm/provider/RulesReader.java
@@ -66,6 +66,7 @@ public class RulesReader {
                     alarmRule.setPeriod((Integer)settings.getOrDefault("period", 1));
                     alarmRule.setCount((Integer)settings.getOrDefault("count", 1));
                     alarmRule.setSilencePeriod((Integer)settings.getOrDefault("silence-period", alarmRule.getPeriod()));
+                    alarmRule.setMessage((String)settings.getOrDefault("message", "Alarm caused by Rule " + alarmRule.getAlarmRuleName()));
 
                     rules.getRules().add(alarmRule);
                 }
diff --git a/oap-server/server-alarm-plugin/src/main/java/org/apache/skywalking/oap/server/core/alarm/provider/RunningRule.java b/oap-server/server-alarm-plugin/src/main/java/org/apache/skywalking/oap/server/core/alarm/provider/RunningRule.java
index 872d5a2..f2a96fa 100644
--- a/oap-server/server-alarm-plugin/src/main/java/org/apache/skywalking/oap/server/core/alarm/provider/RunningRule.java
+++ b/oap-server/server-alarm-plugin/src/main/java/org/apache/skywalking/oap/server/core/alarm/provider/RunningRule.java
@@ -58,6 +58,7 @@ public class RunningRule {
     private volatile IndicatorValueType valueType;
     private Scope targetScope;
     private List<String> includeNames;
+    private AlarmMessageFormatter formatter;
 
     public RunningRule(AlarmRule alarmRule) {
         indicatorName = alarmRule.getIndicatorName();
@@ -75,6 +76,7 @@ public class RunningRule {
         this.silencePeriod = alarmRule.getSilencePeriod();
 
         this.includeNames = alarmRule.getIncludeNames();
+        this.formatter = new AlarmMessageFormatter(alarmRule.getMessage());
     }
 
     /**
@@ -136,9 +138,16 @@ public class RunningRule {
     public List<AlarmMessage> check() {
         List<AlarmMessage> alarmMessageList = new ArrayList<>(30);
 
-        windows.values().forEach(window -> {
+        windows.entrySet().forEach(entry -> {
+            MetaInAlarm meta = entry.getKey();
+            Window window = entry.getValue();
             AlarmMessage alarmMessage = window.checkAlarm();
             if (alarmMessage != AlarmMessage.NONE) {
+                alarmMessage.setScope(meta.getScope());
+                alarmMessage.setName(meta.getName());
+                alarmMessage.setId0(meta.getId0());
+                alarmMessage.setId1(meta.getId1());
+                alarmMessage.setAlarmMessage(formatter.format(meta));
                 alarmMessageList.add(alarmMessage);
             }
         });
@@ -238,7 +247,7 @@ public class RunningRule {
                 if (counter >= countThreshold && silenceCountdown < 1) {
                     silenceCountdown = silencePeriod;
 
-                    //TODO
+                    // set empty message, but new message
                     AlarmMessage message = new AlarmMessage();
                     return message;
                 } else {
diff --git a/oap-server/server-alarm-plugin/src/main/java/org/apache/skywalking/oap/server/core/alarm/provider/WebhookCallback.java b/oap-server/server-alarm-plugin/src/main/java/org/apache/skywalking/oap/server/core/alarm/provider/WebhookCallback.java
index 7655b42..582bc4f 100644
--- a/oap-server/server-alarm-plugin/src/main/java/org/apache/skywalking/oap/server/core/alarm/provider/WebhookCallback.java
+++ b/oap-server/server-alarm-plugin/src/main/java/org/apache/skywalking/oap/server/core/alarm/provider/WebhookCallback.java
@@ -18,21 +18,82 @@
 
 package org.apache.skywalking.oap.server.core.alarm.provider;
 
+import com.google.gson.Gson;
+import java.io.IOException;
+import java.io.UnsupportedEncodingException;
 import java.util.List;
+import org.apache.http.StatusLine;
+import org.apache.http.client.ClientProtocolException;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
 import org.apache.skywalking.oap.server.core.alarm.AlarmCallback;
 import org.apache.skywalking.oap.server.core.alarm.AlarmMessage;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /**
  * Use SkyWalking alarm webhook API call a remote endpoints.
+ *
+ * @author wusheng
  */
 public class WebhookCallback implements AlarmCallback {
+    private static final Logger logger = LoggerFactory.getLogger(WebhookCallback.class);
+    private static final int HTTP_CONNECT_TIMEOUT = 1000;
+    private static final int HTTP_CONNECTION_REQUEST_TIMEOUT = 1000;
+    private static final int HTTP_SOCKET_TIMEOUT = 10000;
+
     private List<String> remoteEndpoints;
+    private RequestConfig requestConfig;
+    private Gson gson = new Gson();
 
     public WebhookCallback(List<String> remoteEndpoints) {
         this.remoteEndpoints = remoteEndpoints;
+        requestConfig = RequestConfig.custom()
+            .setConnectTimeout(HTTP_CONNECT_TIMEOUT)
+            .setConnectionRequestTimeout(HTTP_CONNECTION_REQUEST_TIMEOUT)
+            .setSocketTimeout(HTTP_SOCKET_TIMEOUT).build();
     }
 
     @Override public void doAlarm(List<AlarmMessage> alarmMessage) {
+        if (remoteEndpoints.size() == 0) {
+            return;
+        }
+
+        CloseableHttpClient httpClient = HttpClients.custom().build();
+        try {
+            remoteEndpoints.forEach(url -> {
+                HttpPost post = new HttpPost(url);
+                post.setConfig(requestConfig);
+                post.setHeader("Accept", "application/json");
+                post.setHeader("Content-type", "application/json");
 
+                StringEntity entity = null;
+                try {
+                    entity = new StringEntity(gson.toJson(alarmMessage));
+                    post.setEntity(entity);
+                    CloseableHttpResponse httpResponse = httpClient.execute(post);
+                    StatusLine statusLine = httpResponse.getStatusLine();
+                    if (statusLine != null && statusLine.getStatusCode() != 200) {
+                        logger.error("send alarm to " + url + " failure. Response code: " + statusLine.getStatusCode());
+                    }
+                } catch (UnsupportedEncodingException e) {
+                    logger.error("Alarm to JSON error, " + e.getMessage(), e);
+                } catch (ClientProtocolException e) {
+                    logger.error("send alarm to " + url + " failure.", e);
+                } catch (IOException e) {
+                    logger.error("send alarm to " + url + " failure.", e);
+                }
+            });
+        } finally {
+            try {
+                httpClient.close();
+            } catch (IOException e) {
+                logger.error(e.getMessage(), e);
+            }
+        }
     }
 }
diff --git a/oap-server/server-alarm-plugin/src/test/java/org/apache/skywalking/oap/server/core/alarm/provider/AlarmMessageFormatterTest.java b/oap-server/server-alarm-plugin/src/test/java/org/apache/skywalking/oap/server/core/alarm/provider/AlarmMessageFormatterTest.java
new file mode 100644
index 0000000..02a3699
--- /dev/null
+++ b/oap-server/server-alarm-plugin/src/test/java/org/apache/skywalking/oap/server/core/alarm/provider/AlarmMessageFormatterTest.java
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.core.alarm.provider;
+
+import org.apache.skywalking.oap.server.core.alarm.MetaInAlarm;
+import org.apache.skywalking.oap.server.core.source.Scope;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class AlarmMessageFormatterTest {
+    @Test
+    public void testStringFormatWithNoArg() {
+        AlarmMessageFormatter formatter = new AlarmMessageFormatter("abc words {sdf");
+        String message = formatter.format(new MetaInAlarm() {
+
+            @Override public Scope getScope() {
+                return null;
+            }
+
+            @Override public String getName() {
+                return null;
+            }
+
+            @Override public String getIndicatorName() {
+                return null;
+            }
+
+            @Override public int getId0() {
+                return 0;
+            }
+
+            @Override public int getId1() {
+                return 0;
+            }
+        });
+
+        Assert.assertEquals("abc words {sdf", message);
+    }
+
+    @Test
+    public void testStringFormatWithArg() {
+        AlarmMessageFormatter formatter = new AlarmMessageFormatter("abc} words {name} - {id} .. {");
+        String message = formatter.format(new MetaInAlarm() {
+
+            @Override public Scope getScope() {
+                return null;
+            }
+
+            @Override public String getName() {
+                return "service";
+            }
+
+            @Override public String getIndicatorName() {
+                return null;
+            }
+
+            @Override public int getId0() {
+                return 1290;
+            }
+
+            @Override public int getId1() {
+                return 0;
+            }
+        });
+        Assert.assertEquals("abc} words service - 1290 .. {", message);
+    }
+}
diff --git a/oap-server/server-alarm-plugin/src/test/java/org/apache/skywalking/oap/server/core/alarm/provider/AlarmRuleInitTest.java b/oap-server/server-alarm-plugin/src/test/java/org/apache/skywalking/oap/server/core/alarm/provider/AlarmRuleInitTest.java
index 09aeae6..debcf5c 100644
--- a/oap-server/server-alarm-plugin/src/test/java/org/apache/skywalking/oap/server/core/alarm/provider/AlarmRuleInitTest.java
+++ b/oap-server/server-alarm-plugin/src/test/java/org/apache/skywalking/oap/server/core/alarm/provider/AlarmRuleInitTest.java
@@ -34,11 +34,14 @@ public class AlarmRuleInitTest {
         Assert.assertEquals("85", ruleList.get(1).getThreshold());
         Assert.assertEquals("endpoint_percent_rule", ruleList.get(0).getAlarmRuleName());
         Assert.assertEquals(0, ruleList.get(0).getIncludeNames().size());
+        Assert.assertEquals("Successful rate of endpoint {name} is lower than 75%", ruleList.get(0).getMessage());
 
         Assert.assertEquals("service_b", ruleList.get(1).getIncludeNames().get(1));
+        Assert.assertEquals("Alarm caused by Rule service_percent_rule", ruleList.get(1).getMessage());
 
         List<String> rulesWebhooks = rules.getWebhooks();
         Assert.assertEquals(2, rulesWebhooks.size());
         Assert.assertEquals("http://127.0.0.1/go-wechat/", rulesWebhooks.get(1));
+
     }
 }
diff --git a/oap-server/server-alarm-plugin/src/test/java/org/apache/skywalking/oap/server/core/alarm/provider/RunningRuleTest.java b/oap-server/server-alarm-plugin/src/test/java/org/apache/skywalking/oap/server/core/alarm/provider/RunningRuleTest.java
index 12121df..336e6df 100644
--- a/oap-server/server-alarm-plugin/src/test/java/org/apache/skywalking/oap/server/core/alarm/provider/RunningRuleTest.java
+++ b/oap-server/server-alarm-plugin/src/test/java/org/apache/skywalking/oap/server/core/alarm/provider/RunningRuleTest.java
@@ -82,6 +82,7 @@ public class RunningRuleTest {
         alarmRule.setThreshold("75");
         alarmRule.setCount(3);
         alarmRule.setPeriod(15);
+        alarmRule.setMessage("Successful rate of endpoint {name} is lower than 75%");
 
         RunningRule runningRule = new RunningRule(alarmRule);
         LocalDateTime startTime = TIME_BUCKET_FORMATTER.parseLocalDateTime("201808301440");
@@ -95,14 +96,17 @@ public class RunningRuleTest {
         runningRule.in(getMetaInAlarm(123), getIndicator(timeInPeriod3, 74));
 
         // check at 201808301440
-        Assert.assertEquals(0, runningRule.check().size());
+        List<AlarmMessage> alarmMessages = runningRule.check();
+        Assert.assertEquals(0, alarmMessages.size());
         runningRule.moveTo(TIME_BUCKET_FORMATTER.parseLocalDateTime("201808301441"));
         // check at 201808301441
-        Assert.assertEquals(0, runningRule.check().size());
+        alarmMessages = runningRule.check();
+        Assert.assertEquals(0, alarmMessages.size());
         runningRule.moveTo(TIME_BUCKET_FORMATTER.parseLocalDateTime("201808301442"));
         // check at 201808301442
-        Assert.assertEquals(1, runningRule.check().size());
-
+        alarmMessages = runningRule.check();
+        Assert.assertEquals(1, alarmMessages.size());
+        Assert.assertEquals("Successful rate of endpoint Service_123 is lower than 75%", alarmMessages.get(0).getAlarmMessage());
     }
 
     @Test
diff --git a/oap-server/server-alarm-plugin/src/test/java/org/apache/skywalking/oap/server/core/alarm/provider/WebhookCallbackTest.java b/oap-server/server-alarm-plugin/src/test/java/org/apache/skywalking/oap/server/core/alarm/provider/WebhookCallbackTest.java
new file mode 100644
index 0000000..961acf7
--- /dev/null
+++ b/oap-server/server-alarm-plugin/src/test/java/org/apache/skywalking/oap/server/core/alarm/provider/WebhookCallbackTest.java
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.core.alarm.provider;
+
+import com.google.gson.Gson;
+import com.google.gson.JsonArray;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.InetSocketAddress;
+import java.util.ArrayList;
+import java.util.List;
+import javax.servlet.Servlet;
+import javax.servlet.ServletConfig;
+import javax.servlet.ServletException;
+import javax.servlet.ServletRequest;
+import javax.servlet.ServletResponse;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import org.apache.skywalking.oap.server.core.alarm.AlarmMessage;
+import org.eclipse.jetty.server.Server;
+import org.eclipse.jetty.servlet.ServletContextHandler;
+import org.eclipse.jetty.servlet.ServletHolder;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+public class WebhookCallbackTest implements Servlet {
+    private Server server;
+    private volatile boolean isSuccess = false;
+
+    @Before
+    public void init() throws Exception {
+        server = new Server(new InetSocketAddress("127.0.0.1", 8778));
+        ServletContextHandler servletContextHandler = new ServletContextHandler(ServletContextHandler.NO_SESSIONS);
+        servletContextHandler.setContextPath("/webhook");
+
+        server.setHandler(servletContextHandler);
+
+        ServletHolder servletHolder = new ServletHolder();
+        servletHolder.setServlet(this);
+        servletContextHandler.addServlet(servletHolder, "/receiveAlarm");
+
+        server.start();
+    }
+
+    @After
+    public void stop() throws Exception {
+        server.stop();
+    }
+
+    @Test
+    public void testWebhook() {
+        List<String> remoteEndpoints = new ArrayList<>();
+        remoteEndpoints.add("http://127.0.0.1:8778/webhook/receiveAlarm");
+        WebhookCallback webhookCallback = new WebhookCallback(remoteEndpoints);
+        List<AlarmMessage> alarmMessages = new ArrayList<>(2);
+        alarmMessages.add(new AlarmMessage());
+        alarmMessages.add(new AlarmMessage());
+        webhookCallback.doAlarm(alarmMessages);
+
+        Assert.assertTrue(isSuccess);
+    }
+
+    @Override public void init(ServletConfig config) throws ServletException {
+
+    }
+
+    @Override public ServletConfig getServletConfig() {
+        return null;
+    }
+
+    @Override
+    public void service(ServletRequest request, ServletResponse response) throws ServletException, IOException {
+        HttpServletRequest httpServletRequest = (HttpServletRequest)request;
+        if (httpServletRequest.getContentType().equals("application/json")) {
+            InputStream inputStream = request.getInputStream();
+            ByteArrayOutputStream out = new ByteArrayOutputStream();
+            byte[] buffer = new byte[2048];
+            int readCntOnce;
+
+            while ((readCntOnce = inputStream.read(buffer)) >= 0) {
+                out.write(buffer, 0, readCntOnce);
+            }
+
+            JsonArray elements = new Gson().fromJson(new String(out.toByteArray()), JsonArray.class);
+            if (elements.size() == 2) {
+                ((HttpServletResponse)response).setStatus(200);
+                isSuccess = true;
+                return;
+            }
+
+            ((HttpServletResponse)response).setStatus(500);
+        }
+    }
+
+    @Override public String getServletInfo() {
+        return null;
+    }
+
+    @Override public void destroy() {
+
+    }
+
+}
diff --git a/oap-server/server-alarm-plugin/src/test/resources/alarm-settings.yml b/oap-server/server-alarm-plugin/src/test/resources/alarm-settings.yml
index 7fce7f4..a392648 100644
--- a/oap-server/server-alarm-plugin/src/test/resources/alarm-settings.yml
+++ b/oap-server/server-alarm-plugin/src/test/resources/alarm-settings.yml
@@ -27,6 +27,7 @@ rules:
     count: 3
     # How many times of checks, the alarm keeps silence after alarm triggered, default as same as period.
     silence-period: 10
+    message: Successful rate of endpoint {name} is lower than 75%
   service_percent_rule:
     indicator-name: service_percent
     # [Optional] Default, match all services in this indicator
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/alarm/AlarmMessage.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/alarm/AlarmMessage.java
index 7a8f28e..c0ab5b0 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/alarm/AlarmMessage.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/alarm/AlarmMessage.java
@@ -21,6 +21,7 @@ package org.apache.skywalking.oap.server.core.alarm;
 import lombok.AccessLevel;
 import lombok.Getter;
 import lombok.Setter;
+import org.apache.skywalking.oap.server.core.source.Scope;
 
 /**
  * Alarm message represents the details of each alarm.
@@ -32,6 +33,12 @@ import lombok.Setter;
 public class AlarmMessage {
     public static AlarmMessage NONE = new NoAlarm();
 
+    private Scope scope;
+    private String name;
+    private int id0;
+    private int id1;
+    private String alarmMessage;
+
     private static class NoAlarm extends AlarmMessage {
 
     }