You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@metron.apache.org by rm...@apache.org on 2019/03/05 23:24:30 UTC
[metron] branch master updated: METRON-2019 Improve Metron REST
Logging (merrimanr) closes apache/metron#1347
This is an automated email from the ASF dual-hosted git repository.
rmerriman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new 34a1fd5 METRON-2019 Improve Metron REST Logging (merrimanr) closes apache/metron#1347
34a1fd5 is described below
commit 34a1fd513c8f27b130b6ba8edbb3d4a1ed463993
Author: merrimanr <me...@gmail.com>
AuthorDate: Tue Mar 5 17:24:18 2019 -0600
METRON-2019 Improve Metron REST Logging (merrimanr) closes apache/metron#1347
---
dependencies_with_url.csv | 1 +
metron-interface/metron-rest/README.md | 14 ++++
metron-interface/metron-rest/pom.xml | 16 ++--
.../apache/metron/rest/MetronExceptionHandler.java | 48 ++++++++++++
.../apache/metron/rest/MetronRestApplication.java | 3 -
.../apache/metron/rest/config/LoggingConfig.java | 39 ++++++++++
.../rest/web/filter/ResponseLoggingFilter.java | 87 ++++++++++++++++++++++
.../src/main/resources/log4j.properties | 16 ----
.../src/test/resources/log4j.properties | 19 -----
9 files changed, 197 insertions(+), 46 deletions(-)
diff --git a/dependencies_with_url.csv b/dependencies_with_url.csv
index e16650d..0ae7ab4 100644
--- a/dependencies_with_url.csv
+++ b/dependencies_with_url.csv
@@ -385,6 +385,7 @@ org.springframework.boot:spring-boot-starter-jdbc:jar:2.0.1.RELEASE:compile,ASLv
org.springframework.boot:spring-boot-starter-json:jar:2.0.1.RELEASE:compile,ASLv2,https://github.com/spring-projects/spring-boot
org.springframework.boot:spring-boot-starter-logging:jar:1.4.1.RELEASE:compile,ASLv2,https://github.com/spring-projects/spring-boot
org.springframework.boot:spring-boot-starter-logging:jar:2.0.1.RELEASE:compile,ASLv2,https://github.com/spring-projects/spring-boot
+org.springframework.boot:spring-boot-starter-log4j2:jar:2.0.1.RELEASE:compile,ASLv2,https://github.com/spring-projects/spring-boot
org.springframework.boot:spring-boot-starter-security:jar:1.4.1.RELEASE:compile,ASLv2,https://github.com/spring-projects/spring-boot
org.springframework.boot:spring-boot-starter-security:jar:2.0.1.RELEASE:compile,ASLv2,https://github.com/spring-projects/spring-boot
org.springframework.boot:spring-boot-starter-tomcat:jar:1.4.1.RELEASE:compile,ASLv2,https://github.com/spring-projects/spring-boot
diff --git a/metron-interface/metron-rest/README.md b/metron-interface/metron-rest/README.md
index 11f06bf..8d7d99f 100644
--- a/metron-interface/metron-rest/README.md
+++ b/metron-interface/metron-rest/README.md
@@ -248,6 +248,20 @@ Setting active profiles is done with the METRON_SPRING_PROFILES_ACTIVE variable.
METRON_SPRING_PROFILES_ACTIVE="vagrant,dev"
```
+## Logging
+
+Logging for the REST application can be configured in Ambari. Log levels can be changed at the root, package and class level:
+
+1. Navigate to Services > Metron > Configs > REST and locate the `Metron Spring options` setting.
+
+1. Logging configuration is exposed through Spring properties as explained [here](https://docs.spring.io/spring-boot/docs/current/reference/html/howto-logging.html#howto-logging).
+
+1. The root logging level defaults to ERROR but can be changed to INFO by adding `--logging.level.root=INFO` to the `Metron Spring options` setting.
+
+1. The Metron REST logging level can be changed to INFO by adding `--logging.level.org.apache.metron.rest=INFO`.
+
+1. HTTP request and response logging can be enabled by adding `--logging.level.org.springframework.web.filter.CommonsRequestLoggingFilter=DEBUG --logging.level.org.apache.metron.rest.web.filter.ResponseLoggingFilter=DEBUG`.
+
## Pcap Query
The REST application exposes endpoints for querying Pcap data. For more information about filtering options see [Query Filter Utility](../../metron-platform/metron-pcap-backend#query-filter-utility).
diff --git a/metron-interface/metron-rest/pom.xml b/metron-interface/metron-rest/pom.xml
index 01f1aed..cc58bf8 100644
--- a/metron-interface/metron-rest/pom.xml
+++ b/metron-interface/metron-rest/pom.xml
@@ -78,14 +78,10 @@
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
<exclusions>
- <exclusion>
- <groupId>ch.qos.logback</groupId>
- <artifactId>logback-classic</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.slf4j</groupId>
- <artifactId>log4j-over-slf4j</artifactId>
- </exclusion>
+ <exclusion>
+ <groupId>org.springframework.boot</groupId>
+ <artifactId>spring-boot-starter-logging</artifactId>
+ </exclusion>
<exclusion>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-validator</artifactId>
@@ -94,6 +90,10 @@
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
+ <artifactId>spring-boot-starter-log4j2</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-security</artifactId>
</dependency>
<dependency>
diff --git a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/MetronExceptionHandler.java b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/MetronExceptionHandler.java
new file mode 100644
index 0000000..7723723
--- /dev/null
+++ b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/MetronExceptionHandler.java
@@ -0,0 +1,48 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.rest;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.http.HttpHeaders;
+import org.springframework.http.HttpStatus;
+import org.springframework.http.ResponseEntity;
+import org.springframework.http.converter.HttpMessageNotReadableException;
+import org.springframework.web.bind.annotation.ControllerAdvice;
+import org.springframework.web.context.request.WebRequest;
+import org.springframework.web.servlet.mvc.method.annotation.ResponseEntityExceptionHandler;
+
+import java.lang.invoke.MethodHandles;
+
+/**
+ * Ensures message parsing errors are logged at ERROR level.
+ */
+@ControllerAdvice
+public class MetronExceptionHandler extends ResponseEntityExceptionHandler {
+
+ private static final Logger LOG = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ @Override
+ protected ResponseEntity<Object> handleHttpMessageNotReadable(HttpMessageNotReadableException ex,
+ HttpHeaders headers,
+ HttpStatus status,
+ WebRequest request) {
+ LOG.error(ex.getMessage(), ex);
+ return super.handleHttpMessageNotReadable(ex, headers, status, request);
+ }
+}
diff --git a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/MetronRestApplication.java b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/MetronRestApplication.java
index 52cdf8f..cc065ea 100644
--- a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/MetronRestApplication.java
+++ b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/MetronRestApplication.java
@@ -24,15 +24,12 @@ import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.autoconfigure.gson.GsonAutoConfiguration;
import org.springframework.boot.autoconfigure.kafka.KafkaAutoConfiguration;
-import static org.apache.metron.rest.MetronRestConstants.LOGGING_SYSTEM_PROPERTY;
-
@SpringBootApplication
@EnableAutoConfiguration(exclude = { GsonAutoConfiguration.class, KafkaAutoConfiguration.class })
public class MetronRestApplication {
public static void main(String[] args) {
ParserIndex.reload();
- System.setProperty(LOGGING_SYSTEM_PROPERTY, "none");
SpringApplication.run(MetronRestApplication.class, args);
}
}
diff --git a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/LoggingConfig.java b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/LoggingConfig.java
new file mode 100644
index 0000000..a0d25cd
--- /dev/null
+++ b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/LoggingConfig.java
@@ -0,0 +1,39 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.rest.config;
+
+import org.springframework.context.annotation.Bean;
+import org.springframework.context.annotation.Configuration;
+import org.springframework.web.filter.CommonsRequestLoggingFilter;
+
+@Configuration
+public class LoggingConfig {
+
+ @Bean
+ public CommonsRequestLoggingFilter logFilter() {
+ CommonsRequestLoggingFilter filter
+ = new CommonsRequestLoggingFilter();
+ filter.setIncludeQueryString(true);
+ filter.setIncludePayload(true);
+ filter.setMaxPayloadLength(10000);
+ filter.setIncludeHeaders(true);
+ filter.setAfterMessagePrefix("request: ");
+ filter.setAfterMessageSuffix("");
+ return filter;
+ }
+}
diff --git a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/web/filter/ResponseLoggingFilter.java b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/web/filter/ResponseLoggingFilter.java
new file mode 100644
index 0000000..aa5b6fb
--- /dev/null
+++ b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/web/filter/ResponseLoggingFilter.java
@@ -0,0 +1,87 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.rest.web.filter;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Component;
+import org.springframework.web.util.ContentCachingResponseWrapper;
+import org.springframework.web.util.WebUtils;
+
+import javax.servlet.Filter;
+import javax.servlet.FilterChain;
+import javax.servlet.FilterConfig;
+import javax.servlet.ServletException;
+import javax.servlet.ServletRequest;
+import javax.servlet.ServletResponse;
+import javax.servlet.http.HttpServletResponse;
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.nio.charset.StandardCharsets;
+
+/**
+ * Logs information about Http responses including status code and payload.
+ */
+@Component
+public class ResponseLoggingFilter implements Filter {
+
+ private static final Logger LOG = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ @Autowired
+ private ObjectMapper objectMapper;
+
+ @Override
+ public void init(FilterConfig filterConfig) throws ServletException {
+ }
+
+ @Override
+ public void doFilter(ServletRequest request, ServletResponse response, FilterChain filterChain) throws IOException, ServletException {
+ if (LOG.isDebugEnabled()) {
+ response = new ContentCachingResponseWrapper((HttpServletResponse) response);
+ }
+ try {
+ filterChain.doFilter(request, response);
+ } finally {
+ if (LOG.isDebugEnabled()) {
+ int status = -1;
+ String responseBody = "";
+ ContentCachingResponseWrapper wrapper = WebUtils.getNativeResponse(response, ContentCachingResponseWrapper.class);
+ if (wrapper != null) {
+ status = wrapper.getStatus();
+ responseBody = new String(wrapper.getContentAsByteArray(), StandardCharsets.UTF_8);
+ // try to pretty print
+ try {
+ responseBody = objectMapper.writerWithDefaultPrettyPrinter().writeValueAsString(objectMapper.readValue(responseBody, Object.class));
+ } catch(Exception e) {
+ // if pretty printing fails or response is not json, just move on
+ }
+ // need to copy the response back since we're reading it here
+ wrapper.copyBodyToResponse();
+ }
+ LOG.debug(String.format("response: status=%d;payload=%s", status, responseBody));
+ }
+ }
+ }
+
+ @Override
+ public void destroy() {
+ }
+}
diff --git a/metron-interface/metron-rest/src/main/resources/log4j.properties b/metron-interface/metron-rest/src/main/resources/log4j.properties
deleted file mode 100644
index e5f4095..0000000
--- a/metron-interface/metron-rest/src/main/resources/log4j.properties
+++ /dev/null
@@ -1,16 +0,0 @@
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-log4j.rootLogger=ERROR, stdout
-log4j.appender.stdout=org.apache.log4j.ConsoleAppender
-log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
-log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd'T'HH:mm:ss.SSS} %-5p [%c] - %m%n
diff --git a/metron-interface/metron-rest/src/test/resources/log4j.properties b/metron-interface/metron-rest/src/test/resources/log4j.properties
deleted file mode 100644
index edf866e..0000000
--- a/metron-interface/metron-rest/src/test/resources/log4j.properties
+++ /dev/null
@@ -1,19 +0,0 @@
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-log4j.rootLogger=ERROR, stdout
-log4j.appender.stdout=org.apache.log4j.ConsoleAppender
-log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
-log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd'T'HH:mm:ss.SSS} %-5p [%c] - %m%n
-
-# uncomment the following line to enable debug
-#log4j.logger.org.apache.metron.rest=DEBUG