You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@servicecomb.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2018/03/02 06:26:00 UTC

[jira] [Commented] (SCB-324) Chassis must support network failure simulation, so that I can developers can enhance the robustness of the app

    [ https://issues.apache.org/jira/browse/SCB-324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16383240#comment-16383240 ] 

ASF GitHub Bot commented on SCB-324:
------------------------------------

maheshrajus closed pull request #543: [SCB-324] Fault-Injection handler implementation
URL: https://github.com/apache/incubator-servicecomb-java-chassis/pull/543
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/coverage-reports/pom.xml b/coverage-reports/pom.xml
index 7608697fe..5935c7791 100644
--- a/coverage-reports/pom.xml
+++ b/coverage-reports/pom.xml
@@ -58,6 +58,10 @@
       <groupId>org.apache.servicecomb</groupId>
       <artifactId>handler-bizkeeper</artifactId>
     </dependency>
+    <dependency>
+      <groupId>org.apache.servicecomb</groupId>
+      <artifactId>handler-fault-injection</artifactId>
+    </dependency>
     <dependency>
       <groupId>org.apache.servicecomb</groupId>
       <artifactId>handler-loadbalance</artifactId>
diff --git a/handlers/handler-fault-injection/pom.xml b/handlers/handler-fault-injection/pom.xml
new file mode 100755
index 000000000..1f97348ab
--- /dev/null
+++ b/handlers/handler-fault-injection/pom.xml
@@ -0,0 +1,44 @@
+<!--
+  ~ Licensed to the Apache Software Foundation (ASF) under one or more
+  ~ contributor license agreements.  See the NOTICE file distributed with
+  ~ this work for additional information regarding copyright ownership.
+  ~ The ASF licenses this file to You under the Apache License, Version 2.0
+  ~ (the "License"); you may not use this file except in compliance with
+  ~ the License.  You may obtain a copy of the License at
+  ~
+  ~     http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing, software
+  ~ distributed under the License is distributed on an "AS IS" BASIS,
+  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  ~ See the License for the specific language governing permissions and
+  ~ limitations under the License.
+  -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+	<modelVersion>4.0.0</modelVersion>
+	<parent>
+		<groupId>org.apache.servicecomb</groupId>
+		<artifactId>handlers</artifactId>
+		<version>0.6.0-SNAPSHOT</version>
+	</parent>
+	<artifactId>handler-fault-injection</artifactId>
+	<name>Java Chassis::Handlers::Fault Injection</name>
+	<dependencies>
+		<dependency>
+			<groupId>org.apache.servicecomb</groupId>
+			<artifactId>java-chassis-core</artifactId>
+		</dependency>
+		<dependency>
+			<groupId>org.slf4j</groupId>
+			<artifactId>slf4j-log4j12</artifactId>
+			<scope>test</scope>
+		</dependency>
+		<dependency>
+			<groupId>log4j</groupId>
+			<artifactId>log4j</artifactId>
+			<scope>test</scope>
+		</dependency>
+	</dependencies>
+</project>
diff --git a/handlers/handler-fault-injection/src/main/java/org/apache/servicecomb/faultinjection/FaultInjectionConfig.java b/handlers/handler-fault-injection/src/main/java/org/apache/servicecomb/faultinjection/FaultInjectionConfig.java
new file mode 100755
index 000000000..41a34e9a5
--- /dev/null
+++ b/handlers/handler-fault-injection/src/main/java/org/apache/servicecomb/faultinjection/FaultInjectionConfig.java
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.servicecomb.faultinjection;
+
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.apache.servicecomb.foundation.common.concurrent.ConcurrentHashMapEx;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.netflix.config.DynamicIntProperty;
+import com.netflix.config.DynamicPropertyFactory;
+
+/**
+ * Handles the fault injection configuration read from micro service file/config
+ * center.
+ */
+public final class FaultInjectionConfig {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(FaultInjectionConfig.class);
+
+  // key is configuration parameter.
+  public static Map<String, String> cfgCallback = new ConcurrentHashMapEx<>();
+
+  public int getConfigVal(String config, int defaultValue) {
+    DynamicIntProperty dynamicIntProperty = DynamicPropertyFactory.getInstance().getIntProperty(config,
+        defaultValue);
+
+    if (cfgCallback.get(config) == null) {
+      cfgCallback.put(config, config);
+      dynamicIntProperty.addCallback(() -> {
+        int newValue = dynamicIntProperty.get();
+        String cfgName = dynamicIntProperty.getName();
+
+        //store the value in config center map and check for next requests.
+        FaultInjectionUtil.setConfigCenterValue(cfgName, new AtomicInteger(newValue));
+        LOGGER.info("{} changed to {}", cfgName, newValue);
+      });
+    }
+
+    return dynamicIntProperty.get();
+  }
+}
diff --git a/handlers/handler-fault-injection/src/main/java/org/apache/servicecomb/faultinjection/FaultInjectionConst.java b/handlers/handler-fault-injection/src/main/java/org/apache/servicecomb/faultinjection/FaultInjectionConst.java
new file mode 100644
index 000000000..8bafc5ee7
--- /dev/null
+++ b/handlers/handler-fault-injection/src/main/java/org/apache/servicecomb/faultinjection/FaultInjectionConst.java
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.servicecomb.faultinjection;
+
+/**
+ * Handles the all constant values for fault injection.
+ */
+public class FaultInjectionConst {
+  public static final int FAULT_INJECTION_DELAY_DEFAULT = 5;
+
+  public static final int FAULT_INJECTION_DELAY_PERCENTAGE_DEFAULT = 100;
+
+  public static final int FAULT_INJECTION_ABORT_PERCENTAGE_DEFAULT = 100;
+
+  public static final int FAULT_INJECTION_ABORT_ERROR_MSG_DEFAULT = 421;
+
+  public static final int FAULT_INJECTION_CFG_NULL = -1;
+
+  public static final String CONSUMER_FAULTINJECTION = "cse.governance.Consumer.";
+
+  public static final String CONSUMER_FAULTINJECTION_GLOBAL = "cse.governance.Consumer._global.";
+
+  public static final String CONSUMER_FAULTINJECTION_REST = "policy.fault.protocols.rest.";
+
+  public static final String CONSUMER_FAULTINJECTION_HIGHWAY = "policy.fault.protocols.highway.";
+
+  public static final String FAULTINJECTION_HIGHWAY_TRANSPORT = "highway";
+
+  public static final String FAULTINJECTION_REST_TRANSPORT = "rest";
+
+  private FaultInjectionConst() {
+  }
+}
diff --git a/handlers/handler-fault-injection/src/main/java/org/apache/servicecomb/faultinjection/FaultInjectionHandler.java b/handlers/handler-fault-injection/src/main/java/org/apache/servicecomb/faultinjection/FaultInjectionHandler.java
new file mode 100755
index 000000000..1dec70299
--- /dev/null
+++ b/handlers/handler-fault-injection/src/main/java/org/apache/servicecomb/faultinjection/FaultInjectionHandler.java
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.servicecomb.faultinjection;
+
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.apache.servicecomb.core.Handler;
+import org.apache.servicecomb.core.Invocation;
+import org.apache.servicecomb.swagger.invocation.AsyncResponse;
+import org.apache.servicecomb.swagger.invocation.context.HttpStatus;
+import org.apache.servicecomb.swagger.invocation.exception.CommonExceptionData;
+import org.apache.servicecomb.swagger.invocation.exception.InvocationException;
+import static org.apache.servicecomb.faultinjection.FaultInjectionConst.*;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Fault injection handler which injects the delay/abort for requests based on
+ * the percentage configured in service file.
+ *
+ */
+public class FaultInjectionHandler implements Handler {
+  private static final Logger LOGGER = LoggerFactory.getLogger(FaultInjectionHandler.class);
+
+  @Override
+  public void handle(Invocation invocation, AsyncResponse asyncResp) throws Exception {
+
+    //if no transport then call next handler.
+    if ((invocation.getTransport() != null) && (invocation.getTransport().getName() != null)) {
+
+      // prepare the key and lookup for request count.
+      String key = invocation.getTransport().getName() + invocation.getMicroserviceQualifiedName();
+
+      AtomicLong reqCount = FaultInjectionUtil.getOperMetTotalReq(key);
+      long reqCountCurrent = reqCount.get();
+      // increment the request count here after checking the delay/abort condition.
+      reqCount.incrementAndGet();
+
+      // get the config values related to delay percentage.
+      int delayPercent = getFaultInjectionConfig(invocation,
+          "delay.percent",
+          FAULT_INJECTION_DELAY_PERCENTAGE_DEFAULT);
+
+      // check fault delay condition.
+      boolean isDelay = checkFaultInjectionDelayAndAbort(reqCountCurrent, delayPercent);
+      if (isDelay) {
+        LOGGER.info("delay is added for the request by fault inject handler");
+        long delay = getFaultInjectionConfig(invocation,
+            "delay.fixedDelay",
+            FAULT_INJECTION_DELAY_DEFAULT);
+
+        Thread.sleep(delay);
+      }
+
+      // get the config values related to delay.
+      int abortPercent = getFaultInjectionConfig(invocation,
+          "abort.percent",
+          FAULT_INJECTION_ABORT_PERCENTAGE_DEFAULT);
+
+      // check fault delay condition.
+      boolean isAbort = checkFaultInjectionDelayAndAbort(reqCountCurrent, abortPercent);
+      if (isAbort) {
+        // get the config values related to delay percentage.
+        int errorCode = getFaultInjectionConfig(invocation,
+            "abort.httpStatus",
+            FAULT_INJECTION_ABORT_ERROR_MSG_DEFAULT);
+        // if request need to be abort then return failure with given error code
+        CommonExceptionData errorData = new CommonExceptionData("aborted by fault inject");
+        asyncResp.consumerFail(
+            new InvocationException(new HttpStatus(errorCode, "aborted by fault inject"), errorData));
+        return;
+      }
+    }
+
+    // if no delay and no abort then continue to next handler.
+    invocation.next(asyncResp);
+  }
+
+  /**
+   * It will check the delay/abort condition based on request count and percentage
+   * received.
+   * 
+   * @param reqCount
+   * @param percentage
+   * @param key
+   * @return true/false
+   */
+  private boolean checkFaultInjectionDelayAndAbort(long reqCount, int percentage) {
+    /*
+     * Example: delay/abort percentage configured is 10% and Get the count(suppose
+     * if it is 10th request) from map and calculate resultNew(10th request) and
+     * requestOld(9th request). Like this for every request it will calculate
+     * current request count and previous count. if both not matched need to add
+     * delay/abort otherwise no need to add.
+     */
+
+    // calculate the value with current request count.
+    long resultNew = (reqCount * percentage) / 100;
+
+    // calculate the value with previous count value.
+    long resultOld = ((reqCount - 1) * percentage) / 100;
+
+    // if both are not matching then delay/abort should be added.
+    if (resultNew != resultOld) {
+      return true;
+    }
+    return false;
+  }
+
+  /**
+   * Handles the reading fault injection configuration.
+   * 
+   * @param invocation
+   *            invocation of request
+   * @param key
+   *            configuration key
+   * @return configuration value
+   */
+  private int getFaultInjectionConfig(Invocation invocation, String key, int defaultValue) {
+    int value = 0;
+    String config;
+    FaultInjectionConfig faultCfg = new FaultInjectionConfig();
+
+    //first need to check in config center map which has high priority.
+    Map<String, AtomicInteger> cfgMap = FaultInjectionUtil.getConfigCenterMap();
+
+    // get the config base on priority. operationName-->schema-->service-->global
+    String operationName = invocation.getOperationName();
+    String schema = invocation.getSchemaId();
+    String serviceName = invocation.getMicroserviceName();
+
+    if (operationName != null && schema != null && serviceName != null) {
+      if (invocation.getTransport().getName().equals(FAULTINJECTION_HIGHWAY_TRANSPORT)) {
+        config = CONSUMER_FAULTINJECTION + serviceName + ".schemas." + schema + ".operations." + operationName + "."
+            + CONSUMER_FAULTINJECTION_HIGHWAY + key;
+      } else {
+        config = CONSUMER_FAULTINJECTION + serviceName + ".schemas." + schema + ".operations." + operationName + "."
+            + CONSUMER_FAULTINJECTION_REST + key;
+      }
+
+      if (cfgMap.containsKey(config)) {
+        return cfgMap.get(config).get();
+      }
+
+      value = faultCfg.getConfigVal(config, FAULT_INJECTION_CFG_NULL);
+      if ((value != FAULT_INJECTION_CFG_NULL)) {
+        return value;
+      }
+    }
+
+    if (schema != null && serviceName != null) {
+      if (invocation.getTransport().getName().equals(FAULTINJECTION_HIGHWAY_TRANSPORT)) {
+        config = CONSUMER_FAULTINJECTION + serviceName + ".schemas." + schema + "."
+            + CONSUMER_FAULTINJECTION_HIGHWAY + key;
+      } else {
+        config = CONSUMER_FAULTINJECTION + serviceName + ".schemas." + schema + "."
+            + CONSUMER_FAULTINJECTION_REST + key;
+      }
+
+      if (cfgMap.containsKey(config)) {
+        return cfgMap.get(config).get();
+      }
+
+      value = faultCfg.getConfigVal(config, FAULT_INJECTION_CFG_NULL);
+      if ((value != FAULT_INJECTION_CFG_NULL)) {
+        return value;
+      }
+    }
+
+    if (serviceName != null) {
+      if (invocation.getTransport().getName().equals(FAULTINJECTION_HIGHWAY_TRANSPORT)) {
+        config = CONSUMER_FAULTINJECTION + serviceName + "."
+            + CONSUMER_FAULTINJECTION_HIGHWAY + key;
+      } else {
+        config = CONSUMER_FAULTINJECTION + serviceName + "."
+            + CONSUMER_FAULTINJECTION_REST + key;
+      }
+
+      if (cfgMap.containsKey(config)) {
+        return cfgMap.get(config).get();
+      }
+
+      value = faultCfg.getConfigVal(config, FAULT_INJECTION_CFG_NULL);
+      if ((value != FAULT_INJECTION_CFG_NULL)) {
+        return value;
+      }
+    }
+
+    if (invocation.getTransport().getName().equals(FAULTINJECTION_HIGHWAY_TRANSPORT)) {
+      config = CONSUMER_FAULTINJECTION_GLOBAL
+          + CONSUMER_FAULTINJECTION_HIGHWAY + key;
+    } else {
+      config = CONSUMER_FAULTINJECTION_GLOBAL
+          + CONSUMER_FAULTINJECTION_REST + key;
+    }
+
+    if (cfgMap.containsKey(config)) {
+      return cfgMap.get(config).get();
+    }
+
+    value = faultCfg.getConfigVal(config, defaultValue);
+    return value;
+  }
+}
diff --git a/handlers/handler-fault-injection/src/main/java/org/apache/servicecomb/faultinjection/FaultInjectionUtil.java b/handlers/handler-fault-injection/src/main/java/org/apache/servicecomb/faultinjection/FaultInjectionUtil.java
new file mode 100755
index 000000000..756f34677
--- /dev/null
+++ b/handlers/handler-fault-injection/src/main/java/org/apache/servicecomb/faultinjection/FaultInjectionUtil.java
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.servicecomb.faultinjection;
+
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.apache.servicecomb.foundation.common.concurrent.ConcurrentHashMapEx;
+
+/**
+ * Handles the count for all request based key[transport + microservice qualified name].
+ */
+public class FaultInjectionUtil {
+
+  // key is transport+operQualifiedName
+  public static Map<String, AtomicLong> requestCount = new ConcurrentHashMapEx<>();
+
+  // key is config paramter
+  public static Map<String, AtomicInteger> configCenterValue = new ConcurrentHashMapEx<>();
+
+  /**
+   * Returns total requests per provider for operational level.
+   * 
+   * @param key
+   *            transport+operational name
+   * @return long total requests
+   */
+  public static AtomicLong getOperMetTotalReq(String key) {
+    return requestCount.computeIfAbsent(key, p -> new AtomicLong(1));
+  }
+
+  /**
+   * Returns the map of config parameter key and values.
+   * @return value of config parameter map
+   */
+  public static Map<String, AtomicInteger> getConfigCenterMap() {
+    return configCenterValue;
+  }
+
+  /**
+   * Sets the value for given config parameter.
+   * @param key
+   * @param value
+   */
+  public static void setConfigCenterValue(String key, AtomicInteger value) {
+    configCenterValue.put(key, value);
+  }
+}
diff --git a/handlers/handler-fault-injection/src/main/resources/config/cse.handler.xml b/handlers/handler-fault-injection/src/main/resources/config/cse.handler.xml
new file mode 100755
index 000000000..477963b97
--- /dev/null
+++ b/handlers/handler-fault-injection/src/main/resources/config/cse.handler.xml
@@ -0,0 +1,21 @@
+<!--
+  ~ Licensed to the Apache Software Foundation (ASF) under one or more
+  ~ contributor license agreements.  See the NOTICE file distributed with
+  ~ this work for additional information regarding copyright ownership.
+  ~ The ASF licenses this file to You under the Apache License, Version 2.0
+  ~ (the "License"); you may not use this file except in compliance with
+  ~ the License.  You may obtain a copy of the License at
+  ~
+  ~     http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing, software
+  ~ distributed under the License is distributed on an "AS IS" BASIS,
+  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  ~ See the License for the specific language governing permissions and
+  ~ limitations under the License.
+  -->
+
+<config>
+	<handler id="fault-injection"
+		class="org.apache.servicecomb.faultinjection.FaultInjectionHandler" />
+</config>
diff --git a/handlers/handler-fault-injection/src/test/java/org/apache/servicecomb/faultinjection/TestFaultInjectConfig.java b/handlers/handler-fault-injection/src/test/java/org/apache/servicecomb/faultinjection/TestFaultInjectConfig.java
new file mode 100644
index 000000000..42178663c
--- /dev/null
+++ b/handlers/handler-fault-injection/src/test/java/org/apache/servicecomb/faultinjection/TestFaultInjectConfig.java
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.servicecomb.faultinjection;
+
+import static org.junit.Assert.assertEquals;
+
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+/**
+ * Tests the fault injection configuration.
+ */
+public class TestFaultInjectConfig {
+  FaultInjectionConfig faultCfg;
+
+  @Before
+  public void setUp() throws Exception {
+    faultCfg = new FaultInjectionConfig();
+  }
+
+  @After
+  public void tearDown() throws Exception {
+    faultCfg = null;
+  }
+
+  @Test
+  public void testFaultInjectConfig() throws Exception {
+    int val = faultCfg.getConfigVal("cse.servicecomb.handler.consumer.faultinject.config", 0);
+    Assert.assertEquals(0, val);
+  }
+
+  @Test
+  public void testConstants() {
+    assertEquals("cse.governance.Consumer.", FaultInjectionConst.CONSUMER_FAULTINJECTION);
+    assertEquals("policy.fault.protocols.rest.", FaultInjectionConst.CONSUMER_FAULTINJECTION_REST);
+    assertEquals("policy.fault.protocols.highway.", FaultInjectionConst.CONSUMER_FAULTINJECTION_HIGHWAY);
+
+    assertEquals(5, FaultInjectionConst.FAULT_INJECTION_DELAY_DEFAULT);
+    assertEquals(100, FaultInjectionConst.FAULT_INJECTION_DELAY_PERCENTAGE_DEFAULT);
+    assertEquals(100, FaultInjectionConst.FAULT_INJECTION_ABORT_PERCENTAGE_DEFAULT);
+    assertEquals(421, FaultInjectionConst.FAULT_INJECTION_ABORT_ERROR_MSG_DEFAULT);
+    assertEquals(-1, FaultInjectionConst.FAULT_INJECTION_CFG_NULL);
+  }
+}
diff --git a/handlers/handler-fault-injection/src/test/java/org/apache/servicecomb/faultinjection/TestFaultInjectHandler.java b/handlers/handler-fault-injection/src/test/java/org/apache/servicecomb/faultinjection/TestFaultInjectHandler.java
new file mode 100644
index 000000000..61632321c
--- /dev/null
+++ b/handlers/handler-fault-injection/src/test/java/org/apache/servicecomb/faultinjection/TestFaultInjectHandler.java
@@ -0,0 +1,478 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.servicecomb.faultinjection;
+
+import static org.junit.Assert.assertEquals;
+import java.util.concurrent.atomic.AtomicLong;
+import org.apache.servicecomb.core.Invocation;
+import org.apache.servicecomb.core.Transport;
+import org.apache.servicecomb.core.definition.OperationMeta;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+import org.apache.servicecomb.swagger.invocation.AsyncResponse;
+import org.apache.servicecomb.swagger.invocation.Response;
+
+/**
+ * Tests the fault injection handler functionality.
+ */
+public class TestFaultInjectHandler {
+  FaultInjectionHandler handler;
+
+  Invocation invocation;
+
+  AsyncResponse asyncResp;
+
+  OperationMeta operationMeta;
+
+  private Transport transport;
+
+  Response response;
+
+  AsyncResponse ar = new AsyncResponse() {
+    @Override
+    public void handle(Response resp) {
+      response = resp;
+    }
+  };
+
+  @Before
+  public void setUp() throws Exception {
+    handler = new FaultInjectionHandler();
+
+    invocation = Mockito.mock(Invocation.class);
+
+    asyncResp = Mockito.mock(AsyncResponse.class);
+
+    operationMeta = Mockito.mock(OperationMeta.class);
+
+    transport = Mockito.mock(Transport.class);
+  }
+
+  @After
+  public void tearDown() throws Exception {
+    handler = null;
+
+    invocation = null;
+
+    asyncResp = null;
+
+    operationMeta = null;
+
+    transport = null;
+  }
+
+  /**
+   * Tests the fault injection handler functionality with default values for
+   * highway transport.
+   * 
+   * @throws Exception
+   */
+  @Test
+  public void testFaultInjectHandlerHighwayWithDefaultCfg() throws Exception {
+
+    Mockito.when(invocation.getMicroserviceQualifiedName()).thenReturn("MicroserviceQualifiedName1");
+    Mockito.when(invocation.getTransport()).thenReturn(transport);
+    Mockito.when(transport.getName()).thenReturn("highway");
+    Mockito.when(invocation.getOperationName()).thenReturn("sayHello");
+    Mockito.when(invocation.getSchemaId()).thenReturn("sayHelloSchema");
+    Mockito.when(invocation.getMicroserviceName()).thenReturn("hello");
+
+    handler.handle(invocation, asyncResp);
+
+    AtomicLong count = FaultInjectionUtil.getOperMetTotalReq("highwayMicroserviceQualifiedName1");
+    assertEquals(2, count.get());
+  }
+
+  /**
+   * Tests the fault injection handler functionality with default values for rest
+   * transport.
+   * 
+   * @throws Exception
+   */
+  @Test
+  public void testFaultInjectHandlerRestWithDefaultCfg() throws Exception {
+
+    Mockito.when(invocation.getMicroserviceQualifiedName()).thenReturn("MicroserviceQualifiedName2");
+    Mockito.when(invocation.getTransport()).thenReturn(transport);
+    Mockito.when(transport.getName()).thenReturn("rest");
+    Mockito.when(invocation.getOperationName()).thenReturn("sayHello");
+    Mockito.when(invocation.getSchemaId()).thenReturn("sayHelloSchema");
+    Mockito.when(invocation.getMicroserviceName()).thenReturn("hello");
+
+    handler.handle(invocation, asyncResp);
+
+    AtomicLong count = FaultInjectionUtil.getOperMetTotalReq("restMicroserviceQualifiedName2");
+    assertEquals(2, count.get());
+  }
+
+  /**
+   * Tests the fault injection handler functionality with global configuration
+   * with delay/abort condition.
+   * 
+   * @throws Exception
+   */
+  @Test
+  public void testFaultInjectHandlerConfigChangeGlobal() throws Exception {
+
+    System.setProperty("cse.governance.Consumer._global.policy.fault.protocols.rest.delay.fixedDelay", "5");
+    System.setProperty("cse.governance.Consumer._global.policy.fault.protocols.rest.delay.percent", "10");
+    System.setProperty("cse.governance.Consumer._global.policy.fault.protocols.rest.abort.percent", "10");
+    System.setProperty("cse.governance.Consumer._global.policy.fault.protocols.rest.abort.httpStatus", "421");
+
+    Mockito.when(invocation.getMicroserviceQualifiedName()).thenReturn("MicroserviceQualifiedName3");
+    Mockito.when(invocation.getTransport()).thenReturn(transport);
+    Mockito.when(transport.getName()).thenReturn("rest");
+    Mockito.when(invocation.getOperationName()).thenReturn("sayHello");
+    Mockito.when(invocation.getSchemaId()).thenReturn("sayHelloSchema");
+    Mockito.when(invocation.getMicroserviceName()).thenReturn("hello");
+
+    boolean validAssert;
+    try {
+      validAssert = true;
+      handler.handle(invocation, asyncResp);
+    } catch (Exception e) {
+      validAssert = false;
+    }
+
+    System.getProperties().remove("cse.governance.Consumer._global.policy.fault.protocols.rest.delay.fixedDelay");
+    System.getProperties().remove("cse.governance.Consumer._global.policy.fault.protocols.rest.delay.percent");
+    System.getProperties().remove("cse.governance.Consumer._global.policy.fault.protocols.rest.abort.percent");
+    System.getProperties().remove("cse.governance.Consumer._global.policy.fault.protocols.rest.abort.httpStatus");
+
+    Assert.assertTrue(validAssert);
+    AtomicLong count = FaultInjectionUtil.getOperMetTotalReq("restMicroserviceQualifiedName3");
+    assertEquals(2, count.get());
+  }
+
+  /**
+   * Tests the fault injection handler functionality with service level configuration
+   * with delay/abort condition.
+   * 
+   * @throws Exception
+   */
+  @Test
+  public void testFaultInjectHandlerServiceCfgSuccess() throws Exception {
+
+    System.setProperty("cse.governance.Consumer.carts.policy.fault.protocols.rest.delay.fixedDelay", "1");
+    System.setProperty("cse.governance.Consumer.carts.policy.fault.protocols.rest.delay.percent", "10");
+    System.setProperty("cse.governance.Consumer.carts.policy.fault.protocols.rest.abort.percent", "10");
+    System.setProperty("cse.governance.Consumer.carts.policy.fault.protocols.rest.abort.httpStatus", "421");
+
+    Mockito.when(invocation.getMicroserviceQualifiedName()).thenReturn("MicroserviceQualifiedName4");
+    Mockito.when(invocation.getTransport()).thenReturn(transport);
+    Mockito.when(transport.getName()).thenReturn("rest");
+    Mockito.when(invocation.getOperationName()).thenReturn("sayHello");
+    Mockito.when(invocation.getSchemaId()).thenReturn("sayHelloSchema");
+    Mockito.when(invocation.getMicroserviceName()).thenReturn("carts");
+
+    boolean validAssert;
+    try {
+      validAssert = true;
+      handler.handle(invocation, asyncResp);
+    } catch (Exception e) {
+      validAssert = false;
+    }
+
+    System.getProperties().remove("cse.governance.Consumer.carts.policy.fault.protocols.rest.delay.fixedDelay");
+    System.getProperties().remove("cse.governance.Consumer.carts.policy.fault.protocols.rest.delay.percent");
+    System.getProperties().remove("cse.governance.Consumer.carts.policy.fault.protocols.rest.abort.percent");
+    System.getProperties().remove("cse.governance.Consumer.carts.policy.fault.protocols.rest.abort.httpStatus");
+
+    Assert.assertTrue(validAssert);
+    AtomicLong count = FaultInjectionUtil.getOperMetTotalReq("restMicroserviceQualifiedName4");
+    assertEquals(2, count.get());
+  }
+
+  /**
+   * Tests the fault injection handler functionality with schema level configuration
+   * with delay/abort condition.
+   * 
+   * @throws Exception
+   */
+  @Test
+  public void testFaultInjectHandlerSchemaCfgSuccess() throws Exception {
+
+    System.setProperty("cse.governance.Consumer.carts.schemas.testSchema.policy.fault.protocols.rest.delay.fixedDelay",
+        "1");
+    System.setProperty("cse.governance.Consumer.carts.schemas.testSchema.policy.fault.protocols.rest.delay.percent",
+        "10");
+    System.setProperty("cse.governance.Consumer.carts.schemas.testSchema.policy.fault.protocols.rest.abort.percent",
+        "10");
+    System.setProperty("cse.governance.Consumer.carts.schemas.testSchema.policy.fault.protocols.rest.abort.httpStatus",
+        "421");
+
+    Mockito.when(invocation.getMicroserviceQualifiedName()).thenReturn("MicroserviceQualifiedName5");
+    Mockito.when(invocation.getTransport()).thenReturn(transport);
+    Mockito.when(transport.getName()).thenReturn("rest");
+    Mockito.when(invocation.getOperationName()).thenReturn("sayHello");
+    Mockito.when(invocation.getSchemaId()).thenReturn("testSchema");
+    Mockito.when(invocation.getMicroserviceName()).thenReturn("carts");
+
+    boolean validAssert;
+    try {
+      validAssert = true;
+      handler.handle(invocation, asyncResp);
+    } catch (Exception e) {
+      validAssert = false;
+    }
+
+    System.getProperties()
+        .remove("cse.governance.Consumer.carts.schemas.testSchema.policy.fault.protocols.rest.delay.fixedDelay");
+    System.getProperties()
+        .remove("cse.governance.Consumer.carts.schemas.testSchema.policy.fault.protocols.rest.delay.percent");
+    System.getProperties()
+        .remove("cse.governance.Consumer.carts.schemas.testSchema.policy.fault.protocols.rest.abort.percent");
+    System.getProperties()
+        .remove("cse.governance.Consumer.carts.schemas.testSchema.policy.fault.protocols.rest.abort.httpStatus");
+
+    Assert.assertTrue(validAssert);
+    AtomicLong count = FaultInjectionUtil.getOperMetTotalReq("restMicroserviceQualifiedName5");
+    assertEquals(2, count.get());
+  }
+
+  /**
+   * Tests the fault injection handler functionality with operation level configuration
+   * with delay/abort condition.
+   * 
+   * @throws Exception
+   */
+  @Test
+  public void testFaultInjectHandlerOperationCfgSuccess() throws Exception {
+
+    System.setProperty(
+        "cse.governance.Consumer.carts.schemas.testSchema.operations.sayHi.policy.fault.protocols.rest.delay.fixedDelay",
+        "1");
+    System.setProperty(
+        "cse.governance.Consumer.carts.schemas.testSchema.operations.sayHi.policy.fault.protocols.rest.delay.percent",
+        "10");
+    System.setProperty(
+        "cse.governance.Consumer.carts.schemas.testSchema.operations.sayHi.policy.fault.protocols.rest.abort.percent",
+        "10");
+    System.setProperty(
+        "cse.governance.Consumer.carts.schemas.testSchema.operations.sayHi.policy.fault.protocols.rest.abort.httpStatus",
+        "421");
+
+    Mockito.when(invocation.getMicroserviceQualifiedName()).thenReturn("MicroserviceQualifiedName6");
+    Mockito.when(invocation.getTransport()).thenReturn(transport);
+    Mockito.when(transport.getName()).thenReturn("rest");
+    Mockito.when(invocation.getOperationName()).thenReturn("sayHi");
+    Mockito.when(invocation.getSchemaId()).thenReturn("testSchema");
+    Mockito.when(invocation.getMicroserviceName()).thenReturn("carts");
+
+    boolean validAssert;
+    try {
+      validAssert = true;
+      handler.handle(invocation, asyncResp);
+    } catch (Exception e) {
+      validAssert = false;
+    }
+
+    System.getProperties()
+        .remove(
+            "cse.governance.Consumer.carts.schemas.testSchema.operations.sayHi.policy.fault.protocols.rest.delay.fixedDelay");
+    System.getProperties()
+        .remove(
+            "cse.governance.Consumer.carts.schemas.testSchema.operations.sayHi.policy.fault.protocols.rest.delay.percent");
+    System.getProperties()
+        .remove(
+            "cse.governance.Consumer.carts.schemas.testSchema.operations.sayHi.policy.fault.protocols.rest.abort.percent");
+    System.getProperties()
+        .remove(
+            "cse.governance.Consumer.carts.schemas.testSchema.operations.sayHi.policy.fault.protocols.rest.abort.httpStatus");
+    Assert.assertTrue(validAssert);
+    AtomicLong count = FaultInjectionUtil.getOperMetTotalReq("restMicroserviceQualifiedName6");
+    assertEquals(2, count.get());
+  }
+
+  /**
+   * Tests the fault injection handler functionality with configuration change event for global level config.
+   * 
+   * @throws Exception
+   */
+  @Test
+  public void testFaultInjectHandlerConfigChangeEvent1() throws Exception {
+
+    Mockito.when(invocation.getMicroserviceQualifiedName()).thenReturn("MicroserviceQualifiedName7");
+    Mockito.when(invocation.getTransport()).thenReturn(transport);
+    Mockito.when(transport.getName()).thenReturn("rest");
+    Mockito.when(invocation.getOperationName()).thenReturn("sayBye1");
+    Mockito.when(invocation.getSchemaId()).thenReturn("testSchema1");
+    Mockito.when(invocation.getMicroserviceName()).thenReturn("carts1");
+    boolean validAssert;
+    long timeOld = System.currentTimeMillis();
+    try {
+      validAssert = true;
+      handler.handle(invocation, ar);
+    } catch (Exception e) {
+      validAssert = false;
+    }
+
+    TestFaultInjectUtil
+        .updateProperty("cse.governance.Consumer._global.policy.fault.protocols.rest.delay.fixedDelay", 500);
+
+    handler.handle(invocation, ar);
+    long timeNow = System.currentTimeMillis();
+
+    Assert.assertTrue(validAssert);
+    AtomicLong count = FaultInjectionUtil.getOperMetTotalReq("restMicroserviceQualifiedName7");
+    assertEquals(3, count.get());
+    //check whether error code return, defaut is 421.
+    assertEquals(421, response.getStatusCode());
+    assertEquals(true, response.isFailed());
+    //if really time delay is added it should be greater than 5s.
+    Assert.assertTrue((timeNow - timeOld) >= 500);
+  }
+
+  /**
+   * Tests the fault injection handler functionality with configuration change event for operation level config.
+   * 
+   * @throws Exception
+   */
+  @Test
+  public void testFaultInjectHandlerConfigChangeEvent2() throws Exception {
+    System.setProperty(
+        "cse.governance.Consumer.carts2.schemas.testSchema2.operations.sayBye2.policy.fault.protocols.rest.delay.fixedDelay",
+        "1");
+
+    Mockito.when(invocation.getMicroserviceQualifiedName()).thenReturn("MicroserviceQualifiedName8");
+    Mockito.when(invocation.getTransport()).thenReturn(transport);
+    Mockito.when(transport.getName()).thenReturn("rest");
+    Mockito.when(invocation.getOperationName()).thenReturn("sayBye2");
+    Mockito.when(invocation.getSchemaId()).thenReturn("testSchema2");
+    Mockito.when(invocation.getMicroserviceName()).thenReturn("carts2");
+    boolean validAssert;
+    long timeOld = System.currentTimeMillis();
+    try {
+      validAssert = true;
+      handler.handle(invocation, ar);
+    } catch (Exception e) {
+      validAssert = false;
+    }
+
+    TestFaultInjectUtil
+        .updateProperty(
+            "cse.governance.Consumer.carts2.schemas.testSchema2.operations.sayBye2.policy.fault.protocols.rest.delay.fixedDelay",
+            500);
+
+    handler.handle(invocation, ar);
+    long timeNow = System.currentTimeMillis();
+    System.getProperties()
+        .remove(
+            "cse.governance.Consumer.carts2.schemas.testSchema2.operations.sayBye2.policy.fault.protocols.rest.delay.fixedDelay");
+
+    Assert.assertTrue(validAssert);
+    AtomicLong count = FaultInjectionUtil.getOperMetTotalReq("restMicroserviceQualifiedName8");
+    assertEquals(3, count.get());
+    //check whether error code return, defaut is 421.
+    assertEquals(421, response.getStatusCode());
+    assertEquals(true, response.isFailed());
+    //if really time delay is added it should be greater than 5s.
+    Assert.assertTrue((timeNow - timeOld) >= 500);
+  }
+
+  /**
+   * Tests the fault injection handler functionality with configuration change event for schema level config.
+   * 
+   * @throws Exception
+   */
+  @Test
+  public void testFaultInjectHandlerConfigChangeEvent3() throws Exception {
+    System.setProperty(
+        "cse.governance.Consumer.carts3.schemas.testSchema3.policy.fault.protocols.rest.delay.fixedDelay",
+        "1");
+
+    Mockito.when(invocation.getMicroserviceQualifiedName()).thenReturn("MicroserviceQualifiedName9");
+    Mockito.when(invocation.getTransport()).thenReturn(transport);
+    Mockito.when(transport.getName()).thenReturn("rest");
+    Mockito.when(invocation.getOperationName()).thenReturn("sayBye3");
+    Mockito.when(invocation.getSchemaId()).thenReturn("testSchema3");
+    Mockito.when(invocation.getMicroserviceName()).thenReturn("carts3");
+    boolean validAssert;
+    long timeOld = System.currentTimeMillis();
+    try {
+      validAssert = true;
+      handler.handle(invocation, ar);
+    } catch (Exception e) {
+      validAssert = false;
+    }
+
+    TestFaultInjectUtil
+        .updateProperty(
+            "cse.governance.Consumer.carts3.schemas.testSchema3.policy.fault.protocols.rest.delay.fixedDelay",
+            500);
+
+    handler.handle(invocation, ar);
+    long timeNow = System.currentTimeMillis();
+    System.getProperties()
+        .remove("cse.governance.Consumer.carts3.schemas.testSchema3.policy.fault.protocols.rest.delay.fixedDelay");
+
+    Assert.assertTrue(validAssert);
+    AtomicLong count = FaultInjectionUtil.getOperMetTotalReq("restMicroserviceQualifiedName9");
+    assertEquals(3, count.get());
+    //check whether error code return, defaut is 421.
+    assertEquals(421, response.getStatusCode());
+    assertEquals(true, response.isFailed());
+    //if really time delay is added it should be greater than 5s.
+    Assert.assertTrue((timeNow - timeOld) >= 500);
+  }
+
+  /**
+   * Tests the fault injection handler functionality with configuration change event for service level config.
+   * 
+   * @throws Exception
+   */
+  @Test
+  public void testFaultInjectHandlerConfigChangeEvent4() throws Exception {
+    System.setProperty("cse.governance.Consumer.carts4.policy.fault.protocols.rest.delay.fixedDelay", "1");
+
+    Mockito.when(invocation.getMicroserviceQualifiedName()).thenReturn("MicroserviceQualifiedName10");
+    Mockito.when(invocation.getTransport()).thenReturn(transport);
+    Mockito.when(transport.getName()).thenReturn("rest");
+    Mockito.when(invocation.getOperationName()).thenReturn("sayBye4");
+    Mockito.when(invocation.getSchemaId()).thenReturn("testSchema4");
+    Mockito.when(invocation.getMicroserviceName()).thenReturn("carts4");
+    boolean validAssert;
+    long timeOld = System.currentTimeMillis();
+    try {
+      validAssert = true;
+      handler.handle(invocation, ar);
+    } catch (Exception e) {
+      validAssert = false;
+    }
+
+    TestFaultInjectUtil
+        .updateProperty("cse.governance.Consumer.carts4.policy.fault.protocols.rest.delay.fixedDelay", 500);
+
+    handler.handle(invocation, ar);
+    long timeNow = System.currentTimeMillis();
+    System.getProperties()
+        .remove("cse.governance.Consumer.carts4.policy.fault.protocols.rest.delay.fixedDelay");
+
+    Assert.assertTrue(validAssert);
+    AtomicLong count = FaultInjectionUtil.getOperMetTotalReq("restMicroserviceQualifiedName10");
+    assertEquals(3, count.get());
+    //check whether error code return, defaut is 421.
+    assertEquals(421, response.getStatusCode());
+    assertEquals(true, response.isFailed());
+    //if really time delay is added it should be greater than 5s.
+    Assert.assertTrue((timeNow - timeOld) >= 500);
+  }
+}
diff --git a/handlers/handler-fault-injection/src/test/java/org/apache/servicecomb/faultinjection/TestFaultInjectUtil.java b/handlers/handler-fault-injection/src/test/java/org/apache/servicecomb/faultinjection/TestFaultInjectUtil.java
new file mode 100644
index 000000000..44cc56ae9
--- /dev/null
+++ b/handlers/handler-fault-injection/src/test/java/org/apache/servicecomb/faultinjection/TestFaultInjectUtil.java
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.servicecomb.faultinjection;
+
+import java.lang.reflect.Method;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.junit.Assert;
+import org.junit.Test;
+import org.springframework.util.ReflectionUtils;
+
+import com.netflix.config.DynamicProperty;
+
+/**
+ * Tests the fault inject util functionality.
+ */
+public class TestFaultInjectUtil {
+  private static Method updatePropertyMethod =
+      ReflectionUtils.findMethod(DynamicProperty.class, "updateProperty", String.class, Object.class);
+
+  static {
+    updatePropertyMethod.setAccessible(true);
+  }
+
+  public static void updateProperty(String key, Object value) {
+    ReflectionUtils.invokeMethod(updatePropertyMethod, null, key, value);
+  }
+
+  @Test
+  public void testFaultInjectUtil() throws Exception {
+    AtomicLong count1 = FaultInjectionUtil.getOperMetTotalReq("test");
+    Assert.assertEquals(1, count1.get());
+    count1.incrementAndGet();
+    AtomicLong count2 = FaultInjectionUtil.getOperMetTotalReq("test");
+    Assert.assertEquals(2, count2.get());
+    FaultInjectionUtil.setConfigCenterValue("sayHi", new AtomicInteger(123));
+    int value = FaultInjectionUtil.getConfigCenterMap().get("sayHi").get();
+    Assert.assertEquals(123, value);
+  }
+}
diff --git a/handlers/pom.xml b/handlers/pom.xml
index f0db2696a..9476e29a7 100644
--- a/handlers/pom.xml
+++ b/handlers/pom.xml
@@ -36,6 +36,7 @@
     <module>handler-bizkeeper</module>
     <module>handler-flowcontrol-qps</module>
     <module>handler-loadbalance</module>
+    <module>handler-fault-injection</module>
     <module>handler-publickey-auth</module>
   </modules>
 
diff --git a/java-chassis-dependencies/pom.xml b/java-chassis-dependencies/pom.xml
index 7e8aae035..e3e75ddb0 100644
--- a/java-chassis-dependencies/pom.xml
+++ b/java-chassis-dependencies/pom.xml
@@ -808,6 +808,11 @@
         <artifactId>handler-bizkeeper</artifactId>
         <version>0.6.0-SNAPSHOT</version>
       </dependency>
+      <dependency>
+        <groupId>org.apache.servicecomb</groupId>
+        <artifactId>handler-fault-injection</artifactId>
+        <version>0.6.0-SNAPSHOT</version>
+      </dependency>
       <dependency>
         <groupId>org.apache.servicecomb</groupId>
         <artifactId>handler-loadbalance</artifactId>
diff --git a/java-chassis-distribution/pom.xml b/java-chassis-distribution/pom.xml
index 663dfac39..76e9441e7 100644
--- a/java-chassis-distribution/pom.xml
+++ b/java-chassis-distribution/pom.xml
@@ -124,6 +124,10 @@
             <groupId>org.apache.servicecomb</groupId>
             <artifactId>handler-loadbalance</artifactId>
         </dependency>
+        <dependency>
+            <groupId>org.apache.servicecomb</groupId>
+            <artifactId>handler-fault-injection</artifactId>
+        </dependency>
         <dependency>
             <groupId>org.apache.servicecomb</groupId>
             <artifactId>handler-tracing-zipkin</artifactId>


 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Chassis must support network failure simulation, so that I can developers can enhance the robustness of the app
> ---------------------------------------------------------------------------------------------------------------
>
>                 Key: SCB-324
>                 URL: https://issues.apache.org/jira/browse/SCB-324
>             Project: Apache ServiceComb
>          Issue Type: Task
>          Components: Java-Chassis
>            Reporter: sukesh
>            Assignee: sukesh
>            Priority: Minor
>             Fix For: java-chassis-1.0.0-m1
>
>
> Add new handler for fault injection.
> 1) If handler 'fault-injection' is added in application property file(microservice.yaml) then fault-injection handler should include in handler chain.
> 2) When handler is called read the configuration and listen the configuration event related to fault inject. The sample configuration file is in next slide.
> 3) The configuration mainly consist
>     -delay
>     -abort
>  *delay:*can set the delay for requests based on percentage configured. The unit is ms. The delay percentage default value is 100%
>  *abort*: Abort the requests based on percentage configured. For rest transport protocol error code can be configurable(500, 421) and highway protocol does not support for error return. The abort percentage default value is 100%
> 4) This features currently supports at consumer side.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)