You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@eagle.apache.org by yo...@apache.org on 2016/05/26 05:11:04 UTC
incubator-eagle git commit: EAGLE-280 Mapr integration EAGLE-280
EAGLE-282 EAGLE-284 EAGLE-307: Mapr integration
Repository: incubator-eagle
Updated Branches:
refs/heads/dev 5578cfd91 -> 253f99bce
EAGLE-280 Mapr integration
EAGLE-280 EAGLE-282 EAGLE-284 EAGLE-307: Mapr integration
Author: Daniel Zhou, DadanielZ
Reviewer: Yong Zhang, yonzhang2012@gmail.com
Closes #201
Project: http://git-wip-us.apache.org/repos/asf/incubator-eagle/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-eagle/commit/253f99bc
Tree: http://git-wip-us.apache.org/repos/asf/incubator-eagle/tree/253f99bc
Diff: http://git-wip-us.apache.org/repos/asf/incubator-eagle/diff/253f99bc
Branch: refs/heads/dev
Commit: 253f99bce6fb72fd18823dca0d496516fda6dde0
Parents: 5578cfd
Author: yonzhang <yo...@gmail.com>
Authored: Wed May 25 22:13:02 2016 -0700
Committer: yonzhang <yo...@gmail.com>
Committed: Wed May 25 22:13:02 2016 -0700
----------------------------------------------------------------------
.../src/main/docs/logstash-kafka-conf.md | 59 ++++-
.../eagle-alert-notification-plugin/pom.xml | 4 -
.../org/apache/eagle/common/DateTimeUtil.java | 12 ++
.../security/hdfs/MAPRFSAuditLogObject.java | 31 +++
.../security/hdfs/MAPRFSAuditLogParser.java | 67 ++++++
.../crawler/audit/TestMAPRFSAuditLogParser.java | 64 ++++++
.../eagle-security-maprfs-auditlog/pom.xml | 74 +++++++
.../MapRFSAuditLogKafkaDeserializer.java | 69 ++++++
.../auditlog/MapRFSAuditLogProcessorMain.java | 115 ++++++++++
.../src/main/resources/application.conf | 69 ++++++
.../src/main/resources/log4j.properties | 40 ++++
.../src/main/resources/maprFSAuditLog-init.sh | 213 +++++++++++++++++++
.../main/resources/security-auditlog-storm.yaml | 18 ++
.../eagle-security-maprfs-web/pom.xml | 70 ++++++
.../hdfs/resolver/MAPRFSCommandResolver.java | 72 +++++++
.../hdfs/resolver/MAPRStatusCodeResolver.java | 81 +++++++
eagle-security/pom.xml | 2 +
eagle-webservice/pom.xml | 5 +
pom.xml | 203 +++++++++++++-----
19 files changed, 1205 insertions(+), 63 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/253f99bc/eagle-assembly/src/main/docs/logstash-kafka-conf.md
----------------------------------------------------------------------
diff --git a/eagle-assembly/src/main/docs/logstash-kafka-conf.md b/eagle-assembly/src/main/docs/logstash-kafka-conf.md
index 84952e0..9003fb4 100644
--- a/eagle-assembly/src/main/docs/logstash-kafka-conf.md
+++ b/eagle-assembly/src/main/docs/logstash-kafka-conf.md
@@ -22,7 +22,7 @@
### Install logstash-kafka plugin
-#### For Logstash 1.5.x
+#### For Logstash 1.5.x, 2.x
logstash-kafka has been intergrated into [logstash-input-kafka][logstash-input-kafka] and [logstash-output-kafka][logstash-output-kafka], you can directly use it.
@@ -32,7 +32,7 @@ logstash-kafka has been intergrated into [logstash-input-kafka][logstash-input-k
#### For Logstash 1.4.x
-In logstash 1.4.x, the online version does not support specifying partition\_key for Kafka producer, and data will be produced into each partitions in turn. For eagle, we need to use the src in hdfs\_audit\_log as the partition key, so some hacking work have been done. If you have the same requirment, you can follow it.
+In logstash 1.4.x, the online version does not support specifying partition\_key for Kafka producer, and data will be produced into each partitions in turn. For eagle, we need to use the src in hdfs\_audit\_log as the partition key, so some hacking work have been done. If you have the same requirment, you can follow it.
1. Install logstash-kafka
@@ -51,13 +51,13 @@ In logstash 1.4.x, the online version does not support specifying partition\_key
2. Hacking the kafka.rb
We have added partition\_key\_format, which is used to specify the partition_key and supported by logstash 1.5.x, into lib/logstash/outputs/kafka.rb. More details are shown [here](https://github.xyz.com/eagle/eagle/blob/master/eagle-assembly/src/main/docs/kafka.rb).
-
+
config :partition_key_format, :validate => :string, :default => nil
-
+
public
def register
LogStash::Logger.setup_log4j(@logger)
-
+
options = {
:broker_list => @broker_list,
:compression_codec => @compression_codec,
@@ -80,9 +80,9 @@ In logstash 1.4.x, the online version does not support specifying partition\_key
}
@producer = Kafka::Producer.new(options)
@producer.connect
-
+
@logger.info('Registering kafka producer', :topic_id => @topic_id, :broker_list => @broker_list)
-
+
@codec.on_event do |data|
begin
@producer.send_msg(@current_topic_id,@partition_key,data)
@@ -94,7 +94,7 @@ In logstash 1.4.x, the online version does not support specifying partition\_key
end
end
end # def register
-
+
def receive(event)
return unless output?(event)
if event == LogStash::SHUTDOWN
@@ -111,6 +111,9 @@ In logstash 1.4.x, the online version does not support specifying partition\_key
### Create logstash configuration file
Go to the logstash root dir, and create a configure file
+The 2.0 release of Logstash includes a new version of the Kafka output plugin with significant configuration changes. For more details, please check the documentation pages for the [Logstash1.5](https://www.elastic.co/guide/en/logstash/1.5/plugins-outputs-kafka.html) and [Logstash2.0](https://www.elastic.co/guide/en/logstash/2.0/plugins-outputs-kafka.html) version of the kafka output plugin.
+
+#### For Logstash 1.4.X, 1.5.X
input {
file {
@@ -153,6 +156,46 @@ Go to the logstash root dir, and create a configure file
}
}
+#### For Logstash 2.X
+
+ input {
+ file {
+ type => "hdp-nn-audit"
+ path => "/path/to/audit.log"
+ start_position => end
+ sincedb_path => "/var/log/logstash/"
+ }
+ }
+
+
+ filter{
+ if [type] == "hdp-nn-audit" {
+ grok {
+ match => ["message", "ugi=(?<user>([\w\d\-]+))@|ugi=(?<user>([\w\d\-]+))/[\w\d\-.]+@|ugi=(?<user>([\w\d.\-_]+))[\s(]+"]
+ }
+ }
+ }
+
+ output {
+ if [type] == "hdp-nn-audit" {
+ kafka {
+ codec => plain {
+ format => "%{message}"
+ }
+ bootstrap_servers => "localhost:9092"
+ topic_id => "hdfs_audit_log"
+ acks => \u201c0\u201d
+ timeout_ms => 10000
+ retries => 3
+ retry_backoff_ms => 100
+ batch_size => 16384
+ send_buffer_bytes => 131072
+ client_id => "hdp-nn-audit"
+ }
+ # stdout { codec => rubydebug }
+ }
+ }
+
#### grok pattern testing
We have 3 typical patterns for ugi field as follows
2015-02-11 15:00:00,000 INFO FSNamesystem.audit: allowed=true ugi=user1@xyz.com (auth:TOKEN) ip=/10.115.44.55 cmd=open src=/apps/hdmi-technology/b_pulsar_coe/schema/avroschema/Session.avsc dst=null perm=null
http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/253f99bc/eagle-core/eagle-alert/eagle-alert-notification-plugin/pom.xml
----------------------------------------------------------------------
diff --git a/eagle-core/eagle-alert/eagle-alert-notification-plugin/pom.xml b/eagle-core/eagle-alert/eagle-alert-notification-plugin/pom.xml
index 005af4e..504abb8 100644
--- a/eagle-core/eagle-alert/eagle-alert-notification-plugin/pom.xml
+++ b/eagle-core/eagle-alert/eagle-alert-notification-plugin/pom.xml
@@ -80,9 +80,5 @@
</exclusion>
</exclusions>
</dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-hdfs</artifactId>
- </dependency>
</dependencies>
</project>
http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/253f99bc/eagle-core/eagle-query/eagle-common/src/main/java/org/apache/eagle/common/DateTimeUtil.java
----------------------------------------------------------------------
diff --git a/eagle-core/eagle-query/eagle-common/src/main/java/org/apache/eagle/common/DateTimeUtil.java b/eagle-core/eagle-query/eagle-common/src/main/java/org/apache/eagle/common/DateTimeUtil.java
index dd3a4c6..0f5b189 100644
--- a/eagle-core/eagle-query/eagle-common/src/main/java/org/apache/eagle/common/DateTimeUtil.java
+++ b/eagle-core/eagle-query/eagle-common/src/main/java/org/apache/eagle/common/DateTimeUtil.java
@@ -100,6 +100,18 @@ public class DateTimeUtil {
return 0L;
}
}
+
+ //For mapr
+ //exp: 2015-06-06T10:44:22.800Z
+ public static long maprhumanDateToMilliseconds(String date) throws ParseException{
+ date = date.replace('T',' ');
+ date = date.replace('Z',' ');
+ date = date.replace('.',',');
+ SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss,SSS ");
+ sdf.setTimeZone(CURRENT_TIME_ZONE);
+ Date d = sdf.parse(date);
+ return d.getTime();
+ }
/**
* this could be accurate only when timezone is UTC
* for the timezones other than UTC, there is possibly issue, for example
http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/253f99bc/eagle-security/eagle-security-common/src/main/java/org/apache/eagle/security/hdfs/MAPRFSAuditLogObject.java
----------------------------------------------------------------------
diff --git a/eagle-security/eagle-security-common/src/main/java/org/apache/eagle/security/hdfs/MAPRFSAuditLogObject.java b/eagle-security/eagle-security-common/src/main/java/org/apache/eagle/security/hdfs/MAPRFSAuditLogObject.java
new file mode 100644
index 0000000..630687f
--- /dev/null
+++ b/eagle-security/eagle-security-common/src/main/java/org/apache/eagle/security/hdfs/MAPRFSAuditLogObject.java
@@ -0,0 +1,31 @@
+/*
+ *
+ * * Licensed to the Apache Software Foundation (ASF) under one or more
+ * * contributor license agreements. See the NOTICE file distributed with
+ * * this work for additional information regarding copyright ownership.
+ * * The ASF licenses this file to You under the Apache License, Version 2.0
+ * * (the "License"); you may not use this file except in compliance with
+ * * the License. You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
+package org.apache.eagle.security.hdfs;
+
+
+public class MAPRFSAuditLogObject {
+ public long timestamp;
+ public String host;
+ public String status;
+ public String user;
+ public String cmd;
+ public String src;
+ public String dst;
+ public String volume;
+}
http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/253f99bc/eagle-security/eagle-security-common/src/main/java/org/apache/eagle/security/hdfs/MAPRFSAuditLogParser.java
----------------------------------------------------------------------
diff --git a/eagle-security/eagle-security-common/src/main/java/org/apache/eagle/security/hdfs/MAPRFSAuditLogParser.java b/eagle-security/eagle-security-common/src/main/java/org/apache/eagle/security/hdfs/MAPRFSAuditLogParser.java
new file mode 100644
index 0000000..458de12
--- /dev/null
+++ b/eagle-security/eagle-security-common/src/main/java/org/apache/eagle/security/hdfs/MAPRFSAuditLogParser.java
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.eagle.security.hdfs;
+
+import org.apache.eagle.common.DateTimeUtil;
+import org.codehaus.jettison.json.JSONException;
+import org.codehaus.jettison.json.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.text.ParseException;
+
+
+public final class MAPRFSAuditLogParser {
+ private final static Logger LOG = LoggerFactory.getLogger(MAPRFSAuditLogParser.class);
+
+ public MAPRFSAuditLogParser(){
+ }
+
+ public MAPRFSAuditLogObject parse(String log) throws JSONException, ParseException {
+ JSONObject jsonObject = new JSONObject(log);
+ String timestamp = jsonObject.getJSONObject("timestamp").getString("$date");
+ String cmd = jsonObject.getString("operation");
+ String user = jsonObject.getString("uid");
+ String ip = jsonObject.getString("ipAddress");
+ String status = jsonObject.getString("status");
+ String volumeID = jsonObject.getString("volumeId");
+ String src;
+ String dst;
+ if(jsonObject.has("srcFid")){
+ src = jsonObject.getString("srcFid");
+ }else{
+ src = "null";
+ }
+
+ if(jsonObject.has("dstFid")){
+ dst = jsonObject.getString("dstFid");
+ }else{
+ dst = "null";
+ }
+
+ MAPRFSAuditLogObject entity = new MAPRFSAuditLogObject();
+ entity.user = user;
+ entity.cmd = cmd;
+ entity.src = src;
+ entity.dst = dst;
+ entity.host = ip;
+ entity.status = status;
+ entity.volume = volumeID;
+ entity.timestamp = DateTimeUtil.maprhumanDateToMilliseconds(timestamp);
+ return entity;
+ }
+}
http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/253f99bc/eagle-security/eagle-security-common/src/test/java/org/apache/eagle/security/crawler/audit/TestMAPRFSAuditLogParser.java
----------------------------------------------------------------------
diff --git a/eagle-security/eagle-security-common/src/test/java/org/apache/eagle/security/crawler/audit/TestMAPRFSAuditLogParser.java b/eagle-security/eagle-security-common/src/test/java/org/apache/eagle/security/crawler/audit/TestMAPRFSAuditLogParser.java
new file mode 100644
index 0000000..f0eaf10
--- /dev/null
+++ b/eagle-security/eagle-security-common/src/test/java/org/apache/eagle/security/crawler/audit/TestMAPRFSAuditLogParser.java
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.eagle.security.crawler.audit;
+
+import junit.framework.Assert;
+import org.apache.eagle.security.hdfs.HDFSAuditLogParser;
+import org.apache.eagle.security.hdfs.MAPRFSAuditLogObject;
+import org.apache.eagle.security.hdfs.MAPRFSAuditLogParser;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class TestMAPRFSAuditLogParser {
+
+ private final static Logger LOG = LoggerFactory.getLogger(TestMAPRFSAuditLogParser.class);
+ @Test
+ public void testMAPRParser() {
+ String log = "{\"timestamp\":{\"$date\":\"2015-06-06T10:44:22.800Z\"},\"operation\":\"MKDIR\",\"uid\":0,\"ipAddress\":\n" +
+ "\"10.10.104.51\",\"parentFid\":\"2049.51.131248\",\"childFid\":\"2049.56.131258\",\"childName\":\n" +
+ "\"ycsbTmp_1433587462796\",\"volumeId\":68048396,\"status\":0}";
+ MAPRFSAuditLogParser parser = new MAPRFSAuditLogParser();
+ MAPRFSAuditLogObject entity = new MAPRFSAuditLogObject();
+ try {
+ entity = parser.parse(log);
+ }catch (Exception e){
+ LOG.info(e.getMessage());
+ }
+
+ Assert.assertEquals("MKDIR",entity.cmd);
+ Assert.assertEquals("0",entity.user);
+ Assert.assertEquals("10.10.104.51",entity.host);
+ Assert.assertEquals("null",entity.src);
+ Assert.assertEquals("null",entity.dst);
+ Assert.assertEquals("0", entity.status);
+
+ log = "{\"timestamp\":{\"$date\":\"2016-02-19T01:50:01.962Z\"},\"operation\":\"LOOKUP\",\"uid\":5000,\"ipAddress\":\"192.168.6.148\",\"srcFid\":\"2049.40.131192\",\"dstFid\":\"2049.1032.133268\",\"srcName\":\"share\",\"volumeId\":186635570,\"status\":0}\n";
+ try {
+ entity = parser.parse(log);
+ }catch (Exception e){
+ LOG.info(e.getMessage());
+ }
+ Assert.assertEquals("LOOKUP",entity.cmd);
+ Assert.assertEquals("5000",entity.user);
+ Assert.assertEquals("192.168.6.148",entity.host);
+ Assert.assertEquals("2049.40.131192",entity.src);
+ Assert.assertEquals("2049.1032.133268",entity.dst);
+ Assert.assertEquals("0", entity.status);
+ }
+}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/253f99bc/eagle-security/eagle-security-maprfs-auditlog/pom.xml
----------------------------------------------------------------------
diff --git a/eagle-security/eagle-security-maprfs-auditlog/pom.xml b/eagle-security/eagle-security-maprfs-auditlog/pom.xml
new file mode 100644
index 0000000..71d9127
--- /dev/null
+++ b/eagle-security/eagle-security-maprfs-auditlog/pom.xml
@@ -0,0 +1,74 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.eagle</groupId>
+ <artifactId>eagle-security-parent</artifactId>
+ <version>0.4.0-incubating-SNAPSHOT</version>
+ </parent>
+ <artifactId>eagle-security-maprfs-auditlog</artifactId>
+ <packaging>jar</packaging>
+ <name>eagle-security-maprfs-auditlog</name>
+ <url>http://maven.apache.org</url>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.eagle</groupId>
+ <artifactId>eagle-security-common</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.eagle</groupId>
+ <artifactId>eagle-stream-application-manager</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.eagle</groupId>
+ <artifactId>eagle-security-hdfs-auditlog</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.eagle</groupId>
+ <artifactId>eagle-embed-server</artifactId>
+ <version>${project.version}</version>
+ <classifier>tests</classifier>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.eagle</groupId>
+ <artifactId>eagle-embed-server</artifactId>
+ <version>${project.version}</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.eagle</groupId>
+ <artifactId>eagle-embed-hbase</artifactId>
+ <version>${project.version}</version>
+ <classifier>tests</classifier>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.eagle</groupId>
+ <artifactId>eagle-embed-hbase</artifactId>
+ <version>${project.version}</version>
+ <scope>test</scope>
+ </dependency>
+ </dependencies>
+</project>
http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/253f99bc/eagle-security/eagle-security-maprfs-auditlog/src/main/java/org/apache/eagle/security/auditlog/MapRFSAuditLogKafkaDeserializer.java
----------------------------------------------------------------------
diff --git a/eagle-security/eagle-security-maprfs-auditlog/src/main/java/org/apache/eagle/security/auditlog/MapRFSAuditLogKafkaDeserializer.java b/eagle-security/eagle-security-maprfs-auditlog/src/main/java/org/apache/eagle/security/auditlog/MapRFSAuditLogKafkaDeserializer.java
new file mode 100644
index 0000000..5eb18cd
--- /dev/null
+++ b/eagle-security/eagle-security-maprfs-auditlog/src/main/java/org/apache/eagle/security/auditlog/MapRFSAuditLogKafkaDeserializer.java
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.eagle.security.auditlog;
+
+import org.apache.eagle.dataproc.impl.storm.kafka.SpoutKafkaMessageDeserializer;
+import org.apache.eagle.security.hdfs.MAPRFSAuditLogObject;
+import org.apache.eagle.security.hdfs.MAPRFSAuditLogParser;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Map;
+import java.util.Properties;
+import java.util.TreeMap;
+
+public class MapRFSAuditLogKafkaDeserializer implements SpoutKafkaMessageDeserializer {
+ private static Logger LOG = LoggerFactory.getLogger(MapRFSAuditLogKafkaDeserializer.class);
+ private Properties props;
+
+ public MapRFSAuditLogKafkaDeserializer(Properties props){
+ this.props = props;
+ }
+
+ /**
+ * the steps for deserializing message from kafka
+ * 1. convert byte array to string
+ * 2. parse string to eagle entity
+ */
+ @Override
+ public Object deserialize(byte[] arg0) {
+ String logLine = new String(arg0);
+
+ MAPRFSAuditLogParser parser = new MAPRFSAuditLogParser();
+ MAPRFSAuditLogObject entity = null;
+ try{
+ entity = parser.parse(logLine);
+ }catch(Exception ex){
+ LOG.error("Failing parse audit log message", ex);
+ }
+ if(entity == null){
+ LOG.warn("Event ignored as it can't be correctly parsed, the log is ", logLine);
+ return null;
+ }
+ Map<String, Object> map = new TreeMap<String, Object>();
+ map.put("src", entity.src);
+ map.put("dst", entity.dst);
+ map.put("host", entity.host);
+ map.put("timestamp", entity.timestamp);
+ map.put("status", entity.status);
+ map.put("user", entity.user);
+ map.put("cmd", entity.cmd);
+ map.put("volume", entity.volume);
+
+ return map;
+ }
+}
http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/253f99bc/eagle-security/eagle-security-maprfs-auditlog/src/main/java/org/apache/eagle/security/auditlog/MapRFSAuditLogProcessorMain.java
----------------------------------------------------------------------
diff --git a/eagle-security/eagle-security-maprfs-auditlog/src/main/java/org/apache/eagle/security/auditlog/MapRFSAuditLogProcessorMain.java b/eagle-security/eagle-security-maprfs-auditlog/src/main/java/org/apache/eagle/security/auditlog/MapRFSAuditLogProcessorMain.java
new file mode 100644
index 0000000..b824e8a
--- /dev/null
+++ b/eagle-security/eagle-security-maprfs-auditlog/src/main/java/org/apache/eagle/security/auditlog/MapRFSAuditLogProcessorMain.java
@@ -0,0 +1,115 @@
+/*
+ *
+ * * Licensed to the Apache Software Foundation (ASF) under one or more
+ * * contributor license agreements. See the NOTICE file distributed with
+ * * this work for additional information regarding copyright ownership.
+ * * The ASF licenses this file to You under the Apache License, Version 2.0
+ * * (the "License"); you may not use this file except in compliance with
+ * * the License. You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
+package org.apache.eagle.security.auditlog;
+
+import backtype.storm.spout.SchemeAsMultiScheme;
+import com.typesafe.config.Config;
+import org.apache.commons.lang3.time.DateUtils;
+import org.apache.eagle.common.config.EagleConfigConstants;
+import org.apache.eagle.dataproc.impl.storm.kafka.KafkaSourcedSpoutProvider;
+import org.apache.eagle.dataproc.impl.storm.kafka.KafkaSourcedSpoutScheme;
+import org.apache.eagle.datastream.ExecutionEnvironments;
+import org.apache.eagle.datastream.core.StreamProducer;
+import org.apache.eagle.datastream.storm.StormExecutionEnvironment;
+import org.apache.eagle.partition.DataDistributionDao;
+import org.apache.eagle.partition.PartitionAlgorithm;
+import org.apache.eagle.partition.PartitionStrategy;
+import org.apache.eagle.partition.PartitionStrategyImpl;
+import org.apache.eagle.security.partition.DataDistributionDaoImpl;
+import org.apache.eagle.security.partition.GreedyPartitionAlgorithm;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+public class MapRFSAuditLogProcessorMain {
+
+ public static PartitionStrategy createStrategy(Config config) {
+ // TODO: Refactor configuration structure to avoid repeated config processing configure ~ hao
+ String host = config.getString(EagleConfigConstants.EAGLE_PROPS + "." + EagleConfigConstants.EAGLE_SERVICE + "." + EagleConfigConstants.HOST);
+ Integer port = config.getInt(EagleConfigConstants.EAGLE_PROPS + "." + EagleConfigConstants.EAGLE_SERVICE + "." + EagleConfigConstants.PORT);
+ String username = config.getString(EagleConfigConstants.EAGLE_PROPS + "." + EagleConfigConstants.EAGLE_SERVICE + "." + EagleConfigConstants.USERNAME);
+ String password = config.getString(EagleConfigConstants.EAGLE_PROPS + "." + EagleConfigConstants.EAGLE_SERVICE + "." + EagleConfigConstants.PASSWORD);
+ String topic = config.getString("dataSourceConfig.topic");
+ DataDistributionDao dao = new DataDistributionDaoImpl(host, port, username, password, topic);
+ PartitionAlgorithm algorithm = new GreedyPartitionAlgorithm();
+ String key1 = EagleConfigConstants.EAGLE_PROPS + ".partitionRefreshIntervalInMin";
+ Integer partitionRefreshIntervalInMin = config.hasPath(key1) ? config.getInt(key1) : 60;
+ String key2 = EagleConfigConstants.EAGLE_PROPS + ".kafkaStatisticRangeInMin";
+ Integer kafkaStatisticRangeInMin = config.hasPath(key2) ? config.getInt(key2) : 60;
+ PartitionStrategy strategy = new PartitionStrategyImpl(dao, algorithm, partitionRefreshIntervalInMin * DateUtils.MILLIS_PER_MINUTE, kafkaStatisticRangeInMin * DateUtils.MILLIS_PER_MINUTE);
+ return strategy;
+ }
+
+ public static KafkaSourcedSpoutProvider createProvider(Config config) {
+ String deserClsName = config.getString("dataSourceConfig.deserializerClass");
+ final KafkaSourcedSpoutScheme scheme = new KafkaSourcedSpoutScheme(deserClsName, config) {
+ @Override
+ public List<Object> deserialize(byte[] ser) {
+ Object tmp = deserializer.deserialize(ser);
+ Map<String, Object> map = (Map<String, Object>)tmp;
+ if(tmp == null) return null;
+ return Arrays.asList(map.get("user"), tmp);
+ }
+ };
+
+ KafkaSourcedSpoutProvider provider = new KafkaSourcedSpoutProvider() {
+ @Override
+ public SchemeAsMultiScheme getStreamScheme(String deserClsName, Config context) {
+ return new SchemeAsMultiScheme(scheme);
+ }
+ };
+ return provider;
+ }
+
+ @SuppressWarnings("unchecked")
+ public static void execWithDefaultPartition(StormExecutionEnvironment env, KafkaSourcedSpoutProvider provider) {
+ StreamProducer source = env.fromSpout(provider).withOutputFields(2).nameAs("kafkaMsgConsumer").groupBy(Arrays.asList(0));
+ //StreamProducer reassembler = source.flatMap(new HdfsUserCommandReassembler()).groupBy(Arrays.asList(0));
+ //source.streamUnion(reassembler)
+ source.flatMap(new FileSensitivityDataJoinExecutor()).groupBy(Arrays.asList(0))
+ .flatMap(new IPZoneDataJoinExecutor())
+ .alertWithConsumer("maprFSAuditLogEventStream", "maprFSAuditLogAlertExecutor");
+ env.execute();
+ }
+
+ @SuppressWarnings("unchecked")
+ public static void execWithBalancedPartition(StormExecutionEnvironment env, KafkaSourcedSpoutProvider provider) {
+ PartitionStrategy strategy = createStrategy(env.getConfig());
+ StreamProducer source = env.fromSpout(provider).withOutputFields(2).nameAs("kafkaMsgConsumer").groupBy(strategy);
+ //StreamProducer reassembler = source.flatMap(new HdfsUserCommandReassembler()).groupBy(Arrays.asList(0));
+ //source.streamUnion(reassembler)
+ source.flatMap(new FileSensitivityDataJoinExecutor()).groupBy(Arrays.asList(0))
+ .flatMap(new IPZoneDataJoinExecutor())
+ .alertWithConsumer("maprFSAuditLogEventStream", "maprFSAuditLogAlertExecutor");
+ env.execute();
+ }
+
+ public static void main(String[] args) throws Exception{
+ StormExecutionEnvironment env = ExecutionEnvironments.getStorm(args);
+ Config config = env.getConfig();
+ KafkaSourcedSpoutProvider provider = createProvider(env.getConfig());
+ Boolean balancePartition = config.hasPath("eagleProps.balancePartitionEnabled") && config.getBoolean("eagleProps.balancePartitionEnabled");
+ if (balancePartition) {
+ execWithBalancedPartition(env, provider);
+ } else {
+ execWithDefaultPartition(env, provider);
+ }
+ }
+}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/253f99bc/eagle-security/eagle-security-maprfs-auditlog/src/main/resources/application.conf
----------------------------------------------------------------------
diff --git a/eagle-security/eagle-security-maprfs-auditlog/src/main/resources/application.conf b/eagle-security/eagle-security-maprfs-auditlog/src/main/resources/application.conf
new file mode 100644
index 0000000..52a31a5
--- /dev/null
+++ b/eagle-security/eagle-security-maprfs-auditlog/src/main/resources/application.conf
@@ -0,0 +1,69 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+{
+ "envContextConfig" : {
+ "env" : "storm",
+ "mode" : "local",
+ "topologyName" : "maprFSAuditLogProcessTopology",
+ "stormConfigFile" : "security-auditlog-storm.yaml",
+ "parallelismConfig" : {
+ "kafkaMsgConsumer" : 1,
+ "maprFSAuditLogAlertExecutor*" : 1
+ }
+ },
+ "dataSourceConfig": {
+ "topic" : "maprtest",
+ "zkConnection" : "192.168.6.148:5181",
+ "zkConnectionTimeoutMS" : 15000,
+ "consumerGroupId" : "EagleConsumer",
+ "fetchSize" : 1048586,
+ "deserializerClass" : "org.apache.eagle.security.auditlog.MapRFSAuditLogKafkaDeserializer",
+ "transactionZKServers" : "192.168.6.148",
+ "transactionZKPort" : 5181,
+ "transactionZKRoot" : "/consumers",
+ "transactionStateUpdateMS" : 2000
+ },
+ "alertExecutorConfigs" : {
+ "maprFSAuditLogAlertExecutor" : {
+ "parallelism" : 1,
+ "partitioner" : "org.apache.eagle.policy.DefaultPolicyPartitioner"
+ "needValidation" : "true"
+ }
+ },
+ "eagleProps" : {
+ "site" : "sandbox",
+ "application": "maprFSAuditLog",
+ "dataJoinPollIntervalSec" : 30,
+ "mailHost" : "mailHost.com",
+ "mailSmtpPort":"25",
+ "mailDebug" : "true",
+ "balancePartitionEnabled" : false,
+ #"partitionRefreshIntervalInMin" : 60,
+ #"kafkaStatisticRangeInMin" : 60,
+ "eagleService": {
+ "host": "localhost",
+ "port": 9099,
+ "username": "admin",
+ "password": "secret"
+ },
+ "readHdfsUserCommandPatternFrom" : "file"
+ },
+ "dynamicConfigSource" : {
+ "enabled" : true,
+ "initDelayMillis" : 0,
+ "delayMillis" : 30000
+ }
+}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/253f99bc/eagle-security/eagle-security-maprfs-auditlog/src/main/resources/log4j.properties
----------------------------------------------------------------------
diff --git a/eagle-security/eagle-security-maprfs-auditlog/src/main/resources/log4j.properties b/eagle-security/eagle-security-maprfs-auditlog/src/main/resources/log4j.properties
new file mode 100644
index 0000000..07f8402
--- /dev/null
+++ b/eagle-security/eagle-security-maprfs-auditlog/src/main/resources/log4j.properties
@@ -0,0 +1,40 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+log4j.rootLogger=INFO, stdout, DRFA
+
+eagle.log.dir=./logs
+eagle.log.file=eagle.log
+
+
+#log4j.logger.org.apache.eagle.security.auditlog.IPZoneDataJoinExecutor=DEBUG
+#log4j.logger.org.apache.eagle.security.auditlog.FileSensitivityDataJoinExecutor=DEBUG
+log4j.logger.org.apache.eagle.security.auditlog.HdfsUserCommandReassembler=DEBUG
+#log4j.logger.org.apache.eagle.executor.AlertExecutor=DEBUG
+# standard output
+log4j.appender.stdout=org.apache.log4j.ConsoleAppender
+log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
+log4j.appender.stdout.layout.ConversionPattern=%d{ISO8601} %p [%t] %c{2}[%L]: %m%n
+
+# Daily Rolling File Appender
+log4j.appender.DRFA=org.apache.log4j.DailyRollingFileAppender
+log4j.appender.DRFA.File=${eagle.log.dir}/${eagle.log.file}
+log4j.appender.DRFA.DatePattern=yyyy-MM-dd
+## 30-day backup
+# log4j.appender.DRFA.MaxBackupIndex=30
+log4j.appender.DRFA.layout=org.apache.log4j.PatternLayout
+
+# Pattern format: Date LogLevel LoggerName LogMessage
+log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %p [%t] %c{2}[%L]: %m%n
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/253f99bc/eagle-security/eagle-security-maprfs-auditlog/src/main/resources/maprFSAuditLog-init.sh
----------------------------------------------------------------------
diff --git a/eagle-security/eagle-security-maprfs-auditlog/src/main/resources/maprFSAuditLog-init.sh b/eagle-security/eagle-security-maprfs-auditlog/src/main/resources/maprFSAuditLog-init.sh
new file mode 100644
index 0000000..5f99190
--- /dev/null
+++ b/eagle-security/eagle-security-maprfs-auditlog/src/main/resources/maprFSAuditLog-init.sh
@@ -0,0 +1,213 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with`
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# EAGLE_SERVICE_HOST, default is `hostname -f`
+export EAGLE_SERVICE_HOST=localhost
+# EAGLE_SERVICE_PORT, default is 9099
+export EAGLE_SERVICE_PORT=9099
+# EAGLE_SERVICE_USER
+export EAGLE_SERVICE_USER=admin
+# EAGLE_SERVICE_PASSWORD
+export EAGLE_SERVICE_PASSWD=secret
+curl -u ${EAGLE_SERVICE_USER}:${EAGLE_SERVICE_PASSWD} -X POST -H 'Content-Type:application/json' \
+ "http://${EAGLE_SERVICE_HOST}:${EAGLE_SERVICE_PORT}/eagle-service/rest/entities?serviceName=SiteApplicationService" \
+ -d '
+ [
+ {
+ "tags":{
+ "site":"sandbox",
+ "application":"maprFSAuditLog"
+ },
+ "enabled": true,
+ "config": "classification.fs.defaultFS=maprfs:///"
+ }
+ ]
+ '
+curl -u ${EAGLE_SERVICE_USER}:${EAGLE_SERVICE_PASSWD} -X POST -H 'Content-Type:application/json' \
+ "http://${EAGLE_SERVICE_HOST}:${EAGLE_SERVICE_PORT}/eagle-service/rest/entities?serviceName=ApplicationDescService" \
+ -d '
+ [
+ {
+ "tags":{
+ "application":"maprFSAuditLog"
+ },
+ "description":"mapr AuditLog Monitoring",
+ "alias":"MapRFSAuditLogMonitor",
+ "groupName":"MapR",
+ "features":["common","metadata", "classification"],
+ "config":"{\n\t\"view\": {\n\t\t\"prefix\": \"fileSensitivity\",\n\t\t\"service\": \"FileSensitivityService\",\n\t\t\"keys\": [\n\t\t\t\"filedir\",\n\t\t\t\"sensitivityType\"\n\t\t],\n\t\t\"type\": \"folder\",\n\t\t\"api\": \"hdfsResource\"\n\t}\n}"
+ }
+ ]
+ '
+
+
+
+## AlertStreamService
+echo ""
+echo "Importing AlertStreamService for MapRFS... "
+curl -u ${EAGLE_SERVICE_USER}:${EAGLE_SERVICE_PASSWD} -X POST -H 'Content-Type:application/json' \
+ "http://${EAGLE_SERVICE_HOST}:${EAGLE_SERVICE_PORT}/eagle-service/rest/entities?serviceName=AlertStreamService" \
+ -d '
+ [
+ {
+ "tags":{
+ "application":"maprFSAuditLog",
+ "streamName":"maprFSAuditLogEventStream"
+ },
+ "description":"mapr fs audit log data source stream"
+ }
+ ]
+ '
+## AlertExecutorService: what alert streams are consumed by alert executor
+echo ""
+echo "Importing AlertExecutorService for MapRFS... "
+curl -u ${EAGLE_SERVICE_USER}:${EAGLE_SERVICE_PASSWD} -X POST -H 'Content-Type:application/json' \
+ "http://${EAGLE_SERVICE_HOST}:${EAGLE_SERVICE_PORT}/eagle-service/rest/entities?serviceName=AlertExecutorService" \
+ -d '
+ [
+ {
+ "tags":{
+ "application":"maprFSAuditLog",
+ "alertExecutorId":"maprFSAuditLogAlertExecutor",
+ "streamName":"maprFSAuditLogEventStream"
+ },
+ "description":"executor for mapr fs audit log stream"
+ }
+ ]
+ '
+## AlertStreamSchemaService: schema for event from alert stream
+echo ""
+echo "Importing AlertStreamSchemaService for MapRFS... "
+curl -u ${EAGLE_SERVICE_USER}:${EAGLE_SERVICE_PASSWD} -X POST -H 'Content-Type:application/json' \
+"http://${EAGLE_SERVICE_HOST}:${EAGLE_SERVICE_PORT}/eagle-service/rest/entities?serviceName=AlertStreamSchemaService" \
+ -d '
+ [
+ {
+ "tags": {
+ "application": "maprFSAuditLog",
+ "streamName": "maprFSAuditLogEventStream",
+ "attrName": "status"
+ },
+ "attrDescription": "the status of action result, numeric codes other than 0 for success can appear, for more info, check: http://doc.mapr.com/display/MapR/Status+Codes+That+Can+Appear+in+Audit+Logs",
+ "attrType": "string",
+ "category": "",
+ "attrValueResolver": "org.apache.eagle.service.security.hdfs.resolver.MAPRStatusCodeResolver"
+ },
+ {
+ "tags": {
+ "application": "maprFSAuditLog",
+ "streamName": "maprFSAuditLogEventStream",
+ "attrName": "cmd"
+ },
+ "attrDescription": "file/directory operation, such as MKDIR, LOOKUP",
+ "attrType": "string",
+ "category": "",
+ "attrValueResolver": "org.apache.eagle.service.security.hdfs.resolver.MAPRFSCommandResolver"
+ },
+ {
+ "tags": {
+ "application": "maprFSAuditLog",
+ "streamName": "maprFSAuditLogEventStream",
+ "attrName": "volume"
+ },
+ "attrDescription": "volume name in mapr",
+ "attrType": "string",
+ "category": "",
+ "attrValueResolver": ""
+ },
+ {
+ "tags": {
+ "application": "maprFSAuditLog",
+ "streamName": "maprFSAuditLogEventStream",
+ "attrName": "dst"
+ },
+ "attrDescription": "destination file or directory, such as /tmp",
+ "attrType": "string",
+ "category": "",
+ "attrValueResolver": "org.apache.eagle.service.security.hdfs.resolver.HDFSResourceResolver"
+ },
+ {
+ "tags": {
+ "application": "maprFSAuditLog",
+ "streamName": "maprFSAuditLogEventStream",
+ "attrName": "src"
+ },
+ "attrDescription": "source file or directory, such as /tmp",
+ "attrType": "string",
+ "category": "",
+ "attrValueResolver": "HDFSResourceResolver"
+ },
+ {
+ "tags": {
+ "application": "maprFSAuditLog",
+ "streamName": "maprFSAuditLogEventStream",
+ "attrName": "host"
+ },
+ "attrDescription": "hostname, such as localhost",
+ "attrType": "string",
+ "category": "",
+ "attrValueResolver": ""
+ },
+ {
+ "tags": {
+ "application": "maprFSAuditLog",
+ "streamName": "maprFSAuditLogEventStream",
+ "attrName": "sensitivityType"
+ },
+ "attrDescription": "sensitivity type of of target file/folder, eg: mark such as AUDITLOG, SECURITYLOG",
+ "attrType": "string",
+ "category": "",
+ "attrValueResolver": "org.apache.eagle.service.security.hdfs.resolver.HDFSSensitivityTypeResolver"
+ },
+ {
+ "tags": {
+ "application": "maprFSAuditLog",
+ "streamName": "maprFSAuditLogEventStream",
+ "attrName": "securityZone"
+ },
+ "attrDescription": "",
+ "attrType": "string",
+ "category": "",
+ "attrValueResolver": ""
+ },
+ {
+ "tags": {
+ "application": "maprFSAuditLog",
+ "streamName": "maprFSAuditLogEventStream",
+ "attrName": "user"
+ },
+ "attrDescription": "process user",
+ "attrType": "string",
+ "category": "",
+ "attrValueResolver": ""
+ },
+ {
+ "tags": {
+ "application": "maprFSAuditLog",
+ "streamName": "maprFSAuditLogEventStream",
+ "attrName": "timestamp"
+ },
+ "attrDescription": "milliseconds of the datetime",
+ "attrType": "long",
+ "category": "",
+ "attrValueResolver": ""
+ }
+
+ ]
+ '
+## Finished
+echo ""
+echo "Finished initialization for eagle topology"
http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/253f99bc/eagle-security/eagle-security-maprfs-auditlog/src/main/resources/security-auditlog-storm.yaml
----------------------------------------------------------------------
diff --git a/eagle-security/eagle-security-maprfs-auditlog/src/main/resources/security-auditlog-storm.yaml b/eagle-security/eagle-security-maprfs-auditlog/src/main/resources/security-auditlog-storm.yaml
new file mode 100644
index 0000000..a68a323
--- /dev/null
+++ b/eagle-security/eagle-security-maprfs-auditlog/src/main/resources/security-auditlog-storm.yaml
@@ -0,0 +1,18 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+topology.workers: 1
+topology.acker.executors: 1
+topology.tasks: 1
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/253f99bc/eagle-security/eagle-security-maprfs-web/pom.xml
----------------------------------------------------------------------
diff --git a/eagle-security/eagle-security-maprfs-web/pom.xml b/eagle-security/eagle-security-maprfs-web/pom.xml
new file mode 100644
index 0000000..d10d21a
--- /dev/null
+++ b/eagle-security/eagle-security-maprfs-web/pom.xml
@@ -0,0 +1,70 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<project xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.eagle</groupId>
+ <artifactId>eagle-security-parent</artifactId>
+ <version>0.4.0-incubating-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>eagle-security-maprfs-web</artifactId>
+ <name>eagle-security-maprfs-web</name>
+ <url>http://maven.apache.org</url>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.eagle</groupId>
+ <artifactId>eagle-security-hdfs-web</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.eagle</groupId>
+ <artifactId>eagle-entity-base</artifactId>
+ <version>${project.version}</version>
+ <exclusions>
+ <exclusion>
+ <groupId>org.slf4j</groupId>
+ <artifactId>log4j-over-slf4j</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>com.sun.jersey</groupId>
+ <artifactId>jersey-server</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.eagle</groupId>
+ <artifactId>eagle-service-base</artifactId>
+ <version>${project.version}</version>
+ <exclusions>
+ <exclusion>
+ <groupId>org.slf4j</groupId>
+ <artifactId>log4j-over-slf4j</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.eagle</groupId>
+ <artifactId>eagle-security-common</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ </dependencies>
+</project>
+
http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/253f99bc/eagle-security/eagle-security-maprfs-web/src/main/java/org/apache/eagle/service/security/hdfs/resolver/MAPRFSCommandResolver.java
----------------------------------------------------------------------
diff --git a/eagle-security/eagle-security-maprfs-web/src/main/java/org/apache/eagle/service/security/hdfs/resolver/MAPRFSCommandResolver.java b/eagle-security/eagle-security-maprfs-web/src/main/java/org/apache/eagle/service/security/hdfs/resolver/MAPRFSCommandResolver.java
new file mode 100644
index 0000000..edb0737
--- /dev/null
+++ b/eagle-security/eagle-security-maprfs-web/src/main/java/org/apache/eagle/service/security/hdfs/resolver/MAPRFSCommandResolver.java
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.eagle.service.security.hdfs.resolver;
+
+import org.apache.eagle.service.alert.resolver.AttributeResolvable;
+import org.apache.eagle.service.alert.resolver.AttributeResolveException;
+import org.apache.eagle.service.alert.resolver.BadAttributeResolveRequestException;
+import org.apache.eagle.service.alert.resolver.GenericAttributeResolveRequest;
+import org.apache.commons.lang3.StringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.regex.Pattern;
+
+
+public class MAPRFSCommandResolver implements AttributeResolvable<GenericAttributeResolveRequest,String> {
+ private final static Logger LOG = LoggerFactory.getLogger(MAPRFSCommandResolver.class);
+
+ private final static String [] cmdStrs = {"CHGRP", "CHOWN", "CHPERM","CREATE", "DELETE",
+ "DISABLEAUDIT", "ENABLEAUDIT", "GETATTR","LOOKUP", "MKDIR", "READ", "READDIR",
+ "RENAME", "RMDIR", "SETATTR", "WRITE"};
+
+ private final static String MAPRFS_CMD_RESOLVE_FORMAT_HINT = String.format("maprfs command must be in {%s}", StringUtils.join(cmdStrs, ","));
+
+ private final static List<String> commands = Arrays.asList(cmdStrs);
+
+ public List<String> resolve(GenericAttributeResolveRequest request) throws AttributeResolveException {
+ String query = request.getQuery().trim();
+ List<String> res = new ArrayList<>();
+ for(String cmd : commands) {
+ Pattern pattern = Pattern.compile("^" + query, Pattern.CASE_INSENSITIVE);
+ if(pattern.matcher(cmd).find()) {
+ res.add(cmd);
+ }
+ }
+ if(res.size() == 0) {
+ return commands;
+ }
+ return res;
+ }
+
+ @Override
+ public void validateRequest(GenericAttributeResolveRequest request) throws BadAttributeResolveRequestException {
+ String query = request.getQuery();
+ boolean matched = Pattern.matches("[a-zA-Z]+", query);
+ if (query == null || !matched) {
+ throw new BadAttributeResolveRequestException(MAPRFS_CMD_RESOLVE_FORMAT_HINT);
+ }
+ }
+
+ @Override
+ public Class<GenericAttributeResolveRequest> getRequestClass() {
+ return GenericAttributeResolveRequest.class;
+ }
+}
http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/253f99bc/eagle-security/eagle-security-maprfs-web/src/main/java/org/apache/eagle/service/security/hdfs/resolver/MAPRStatusCodeResolver.java
----------------------------------------------------------------------
diff --git a/eagle-security/eagle-security-maprfs-web/src/main/java/org/apache/eagle/service/security/hdfs/resolver/MAPRStatusCodeResolver.java b/eagle-security/eagle-security-maprfs-web/src/main/java/org/apache/eagle/service/security/hdfs/resolver/MAPRStatusCodeResolver.java
new file mode 100644
index 0000000..248c297
--- /dev/null
+++ b/eagle-security/eagle-security-maprfs-web/src/main/java/org/apache/eagle/service/security/hdfs/resolver/MAPRStatusCodeResolver.java
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eagle.service.security.hdfs.resolver;
+
+import org.apache.eagle.service.alert.resolver.AttributeResolvable;
+import org.apache.eagle.service.alert.resolver.AttributeResolveException;
+import org.apache.eagle.service.alert.resolver.BadAttributeResolveRequestException;
+import org.apache.eagle.service.alert.resolver.GenericAttributeResolveRequest;
+import org.apache.commons.lang3.StringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.regex.Pattern;
+
+
+public class MAPRStatusCodeResolver implements AttributeResolvable<GenericAttributeResolveRequest,String> {
+ private final static Logger LOG = LoggerFactory.getLogger(MAPRStatusCodeResolver.class);
+
+ private final static String [] statusCodes = {"SUCCESS","EPERM","ENOENT","EINTR","EIO","ENXIO","E2BIG","ENOEXEC","EBADF",
+ "ECHILD","EAGAIN","ENOMEM","EACCES","EFAULT","ENOTBLK","EBUSY","EEXIST","EXDEV","ENODEV","ENOTDIR","EISDIR","EINVAL",
+ "ENFILE","EMFILE","ENOTTY","ETXTBSY","EFBIG","ENOSPC","ESPIPE","EROFS","EMLINK","EPIPE","EDOM","ERANGE","EDEADLK","ENAMETOOLONG",
+ "ENOLCK","ENOSYS","ENOTEMPTY","ELOOP","EWOULDBLOCK","ENOMSG","EIDRM","ECHRNG","EL2NSYNC","EL3HLT","EL3RST","ELNRNG","EUNATCH",
+ "ENOCSI","EL2HLT","EBADE","EBADR","EXFULL","ENOANO","EBADRQC","EBADSLT","EDEADLOCK","EBFONT","ENOSTR","ENODATA","ETIME","ENOSR",
+ "ENONET","ENOPKG","EREMOTE","ENOLINK","EADV","ESRMNT","ECOMM","EPROTO","EMULTIHOP","EDOTDOT","EBADMSG","EOVERFLOW","ENOTUNIQ",
+ "EBADFD","EREMCHG","ELIBACC","ELIBBAD","ELIBSCN","ELIBMAX","ELIBEXEC","EILSEQ","ERESTART","ESTRPIPE","EUSERS","ENOTSOCK",
+ "EDESTADDRREQ","EMSGSIZE","EPROTOTYPE","ENOPROTOOPT","EPROTONOSUPPORT","ESOCKTNOSUPPORT","EOPNOTSUPP","EPFNOSUPPORT","EAFNOSUPPORT",
+ "EADDRINUSE","EADDRNOTAVAIL","ENETDOWN","ENETUNREACH","ENETRESET","ECONNABORTED","ECONNRESET","ENOBUFS","EISCONN","ENOTCONN",
+ "ESHUTDOWN","ETOOMANYREFS","ETIMEDOUT","ECONNREFUSED","EHOSTDOWN","EHOSTUNREACH","EALREADY","EINPROGRESS","ESTALE","EUCLEAN","ENOTNAM",
+ "ENAVAIL","EISNAM","EREMOTEIO","EDQUOT","ENOMEDIUM","EMEDIUMTYPE","ECANCELED","ENOKEY","EKEYEXPIRED","EKEYREVOKED","EKEYREJECTED"};
+
+ private final static String MAPRFS_STATUS_CODE_RESOLVE_FORMAT_HINT = String.format("Status code must be in {%s}", StringUtils.join(statusCodes, ","));
+
+ private final static List<String> statusList = Arrays.asList(statusCodes);
+
+ public List<String> resolve(GenericAttributeResolveRequest request) throws AttributeResolveException {
+ String query = request.getQuery().trim();
+ List<String> res = new ArrayList<>();
+ for(String status : statusList) {
+ Pattern pattern = Pattern.compile("^" + query, Pattern.CASE_INSENSITIVE);
+ if(pattern.matcher(status).find()) {
+ res.add(status);
+ }
+ }
+ if(res.size() == 0) {
+ return statusList;
+ }
+ return res;
+ }
+
+ @Override
+ public void validateRequest(GenericAttributeResolveRequest request) throws BadAttributeResolveRequestException {
+ String query = request.getQuery();
+ boolean matched = Pattern.matches("[a-zA-Z]+", query);
+ if (query == null || !matched) {
+ throw new BadAttributeResolveRequestException(MAPRFS_STATUS_CODE_RESOLVE_FORMAT_HINT);
+ }
+ }
+
+ @Override
+ public Class<GenericAttributeResolveRequest> getRequestClass() {
+ return GenericAttributeResolveRequest.class;
+ }
+}
http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/253f99bc/eagle-security/pom.xml
----------------------------------------------------------------------
diff --git a/eagle-security/pom.xml b/eagle-security/pom.xml
index ce1a178..93c5aa1 100644
--- a/eagle-security/pom.xml
+++ b/eagle-security/pom.xml
@@ -34,10 +34,12 @@
<modules>
<module>eagle-security-common</module>
<module>eagle-security-hdfs-auditlog</module>
+ <module>eagle-security-maprfs-auditlog</module>
<module>eagle-security-userprofile</module>
<module>eagle-security-hive</module>
<module>eagle-security-hive-web</module>
<module>eagle-security-hdfs-web</module>
+ <module>eagle-security-maprfs-web</module>
<module>eagle-security-hdfs-securitylog</module>
<module>eagle-security-hbase-securitylog</module>
<module>eagle-security-hbase-web</module>
http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/253f99bc/eagle-webservice/pom.xml
----------------------------------------------------------------------
diff --git a/eagle-webservice/pom.xml b/eagle-webservice/pom.xml
index 44bd4da..c316db2 100644
--- a/eagle-webservice/pom.xml
+++ b/eagle-webservice/pom.xml
@@ -145,6 +145,11 @@
<artifactId>eagle-security-hdfs-web</artifactId>
<version>${project.version}</version>
</dependency>
+ <dependency>
+ <groupId>org.apache.eagle</groupId>
+ <artifactId>eagle-security-maprfs-web</artifactId>
+ <version>${project.version}</version>
+ </dependency>
<!--
<dependency>
<groupId>org.apache.eagle</groupId>
http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/253f99bc/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index 79d5e9d..f17e082 100755
--- a/pom.xml
+++ b/pom.xml
@@ -164,11 +164,16 @@
<hadoop.version>2.6.0.2.2.5.1-3</hadoop.version>
<hbase.version>0.98.4.2.2.5.1-3-hadoop2</hbase.version>
<hive.version>1.2.1</hive.version>
+
+ <mapr-hive.version>1.2.0-mapr-1510</mapr-hive.version>
+ <mapr-hadoop.version>2.7.0-mapr-1506</mapr-hadoop.version>
+ <maprfs.version>5.0.0-mapr</maprfs.version>
+ <mapr-hbase.version>0.98.9-mapr-1503</mapr-hbase.version>
+
<spark.core.version>1.4.0</spark.core.version>
<!-- Client -->
<kafka-client.version>0.9.0.0</kafka-client.version>
-
<!-- Reflection -->
<reflections.version>0.9.8</reflections.version>
@@ -215,6 +220,12 @@
<jgrapht.version>0.9.0</jgrapht.version>
<storm-kafka.version>0.9.3.2.2.0.0-2041</storm-kafka.version>
<storm.version>0.9.3</storm.version>
+
+ <mapr-kafka.version>0.8.1.1</mapr-kafka.version>
+ <mapr-kafka-clients.version>0.8.3-mapr-1509</mapr-kafka-clients.version>
+ <mapr-storm-kafka.version>0.9.3</mapr-storm-kafka.version>
+ <mapr-storm.version>0.9.3-mapr-1509</mapr-storm.version>
+
<curator.version>2.8.0</curator.version>
<!-- Query -->
@@ -239,7 +250,7 @@
<jaxb-impl.version>2.2.6</jaxb-impl.version>
<stax-api.version>1.0.1</stax-api.version>
<org.mortbay.jetty.version>6.1.26</org.mortbay.jetty.version>
-
+
<!-- servlet -->
<servlet-api.version>2.5</servlet-api.version>
@@ -475,11 +486,6 @@
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
- <artifactId>kafka_${scala.version}</artifactId>
- <version>${kafka.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>${kafka-clients.version}</version>
</dependency>
@@ -503,25 +509,6 @@
</exclusions>
</dependency>
<dependency>
- <groupId>org.apache.storm</groupId>
- <artifactId>storm-kafka</artifactId>
- <version>${storm-kafka.version}</version>
- <exclusions>
- <exclusion>
- <groupId>ch.qos.logback</groupId>
- <artifactId>logback-classic</artifactId>
- </exclusion>
- <exclusion>
- <groupId>log4j</groupId>
- <artifactId>log4j</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.slf4j</groupId>
- <artifactId>log4j-over-slf4j</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
- <dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.version}</artifactId>
<version>${spark.core.version}</version>
@@ -751,6 +738,27 @@
<activation>
<activeByDefault>true</activeByDefault>
</activation>
+ <repositories>
+ <repository>
+ <id>HDP Release Repository</id>
+ <url>http://repo.hortonworks.com/content/repositories/releases/</url>
+ <releases>
+ <enabled>true</enabled>
+ <updatePolicy>never</updatePolicy>
+ </releases>
+ </repository>
+ <repository>
+ <id>HDP Central Repository</id>
+ <url>http://repo.hortonworks.com/content/repositories/central/</url>
+ <releases>
+ <enabled>true</enabled>
+ <updatePolicy>never</updatePolicy>
+ <checksumPolicy>warn</checksumPolicy>
+ </releases>
+ <name>HDP Central</name>
+ <layout>default</layout>
+ </repository>
+ </repositories>
<dependencyManagement>
<dependencies>
<dependency>
@@ -758,7 +766,6 @@
<artifactId>hadoop-common</artifactId>
<version>${hadoop.version}</version>
</dependency>
-
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-client</artifactId>
@@ -799,6 +806,119 @@
<artifactId>hive-jdbc</artifactId>
<version>${hive.version}</version>
</dependency>
+ <dependency>
+ <groupId>org.apache.kafka</groupId>
+ <artifactId>kafka_${scala.version}</artifactId>
+ <version>${kafka.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.storm</groupId>
+ <artifactId>storm-kafka</artifactId>
+ <version>${storm-kafka.version}</version>
+ <exclusions>
+ <exclusion>
+ <groupId>ch.qos.logback</groupId>
+ <artifactId>logback-classic</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>log4j</groupId>
+ <artifactId>log4j</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.slf4j</groupId>
+ <artifactId>log4j-over-slf4j</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ </dependencies>
+ </dependencyManagement>
+ </profile>
+
+ <profile>
+ <id>mapr</id>
+ <!--<activation>-->
+ <!--<activeByDefault>true</activeByDefault>-->
+ <!--</activation>-->
+ <repositories>
+ <repository>
+ <id>mapr-releases</id>
+ <url>http://repository.mapr.com/maven/</url>
+ <snapshots><enabled>false</enabled></snapshots>
+ <releases><enabled>true</enabled></releases>
+ </repository>
+ </repositories>
+ <dependencyManagement>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ <version>${mapr-hadoop.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hbase</groupId>
+ <artifactId>hbase-client</artifactId>
+ <version>${mapr-hbase.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>com.mapr.hadoop</groupId>
+ <artifactId>maprfs</artifactId>
+ <version>${maprfs.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-client</artifactId>
+ <version>${mapr-hadoop.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-auth</artifactId>
+ <version>${mapr-hadoop.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hbase</groupId>
+ <artifactId>hbase-server</artifactId>
+ <version>${mapr-hbase.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hbase</groupId>
+ <artifactId>hbase-testing-util</artifactId>
+ <version>${mapr-hbase.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-exec</artifactId>
+ <version>${mapr-hive.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-jdbc</artifactId>
+ <version>${mapr-hive.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.kafka</groupId>
+ <artifactId>kafka_${scala.version}</artifactId>
+ <version>${mapr-kafka.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.storm</groupId>
+ <artifactId>storm-kafka</artifactId>
+ <version>${mapr-storm-kafka.version}</version>
+ <exclusions>
+ <exclusion>
+ <groupId>ch.qos.logback</groupId>
+ <artifactId>logback-classic</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>log4j</groupId>
+ <artifactId>log4j</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.slf4j</groupId>
+ <artifactId>log4j-over-slf4j</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
</dependencies>
</dependencyManagement>
</profile>
@@ -950,33 +1070,14 @@
<exclude>**/webapp/**</exclude>
</excludes>
- </configuration>
- </execution>
- </executions>
- </plugin>
- </plugins>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
</build>
<repositories>
<repository>
- <id>HDP Release Repository</id>
- <url>http://repo.hortonworks.com/content/repositories/releases/</url>
- <releases>
- <enabled>true</enabled>
- <updatePolicy>never</updatePolicy>
- </releases>
- </repository>
- <repository>
- <id>HDP Central Repository</id>
- <url>http://repo.hortonworks.com/content/repositories/central/</url>
- <releases>
- <enabled>true</enabled>
- <updatePolicy>never</updatePolicy>
- <checksumPolicy>warn</checksumPolicy>
- </releases>
- <name>HDP Central</name>
- <layout>default</layout>
- </repository>
- <repository>
<id>Maven Repo1 Repository</id>
<url>http://repo1.maven.org/maven2</url>
<releases>