You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2021/09/05 08:34:29 UTC

[GitHub] [dolphinscheduler] yc322 opened a new pull request #6103: [Feature-5822][api] Add audit log

yc322 opened a new pull request #6103:
URL: https://github.com/apache/dolphinscheduler/pull/6103


   ## Purpose of the pull request
   
   This PR adds audit log function.
   
   ## Brief change log
   - audit log subscriber and publisher
   AuditMessage.java
   AuditPublishService.java
   AuditSubscriber.java
   AuditSubscriberImpl.java
   - audit log service and controller
   AuditServiceImpl.java
   AuditService.java
   AuditLogController.java
   - configuration
   update application-api.properties
   audit.control.global.switch=false
   AuditConfiguration.java
   - Enums
   dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/enums/
   add AuditModuleType and AuditOperationType
   -Dao
   AuditLogMapper and AuditLogMapper.xml
   - audit log ui page
   /monitor/pages/log
   auditLog.vue
   common.js
   
   ## Verify this pull request
   This change added tests and can be verified as follows:
   - Added api-configuration tests
   - Manually verified the change by testing locally
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [dolphinscheduler] ruanwenjun commented on pull request #6103: [Feature-5822][api] Add audit log

Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on pull request #6103:
URL: https://github.com/apache/dolphinscheduler/pull/6103#issuecomment-915720484


   @yc322 Please fix the conflicts first.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [dolphinscheduler] yc322 closed pull request #6103: [Feature-5822][api] Add audit log

Posted by GitBox <gi...@apache.org>.
yc322 closed pull request #6103:
URL: https://github.com/apache/dolphinscheduler/pull/6103


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [dolphinscheduler] yc322 commented on a change in pull request #6103: [Feature-5822][api] Add audit log

Posted by GitBox <gi...@apache.org>.
yc322 commented on a change in pull request #6103:
URL: https://github.com/apache/dolphinscheduler/pull/6103#discussion_r702618695



##########
File path: sql/dolphinscheduler_mysql.sql
##########
@@ -968,3 +968,16 @@ CREATE TABLE `t_ds_alert_plugin_instance` (
   `instance_name` varchar(200) DEFAULT NULL COMMENT 'alert instance name',
   PRIMARY KEY (`id`)
 ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+
+-- ----------------------------
+-- Table structure for t_ds_alert_plugin_instance
+-- ----------------------------
+DROP TABLE IF EXISTS `t_ds_audit_log`;
+CREATE TABLE `t_ds_audit_log` (
+  `id` int NOT NULL AUTO_INCREMENT,
+  `user_id` int(11) NOT NULL COMMENT 'user id',
+  `module` int DEFAULT NULL COMMENT 'module',
+  `operation` int DEFAULT NULL COMMENT 'operation',
+  `time` datetime DEFAULT NULL COMMENT 'create time',

Review comment:
       @ruanwenjun  Thanks for your review, I have addressed the comments. PTAL




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [dolphinscheduler] yc322 commented on a change in pull request #6103: [Feature-5822][api] Add audit log

Posted by GitBox <gi...@apache.org>.
yc322 commented on a change in pull request #6103:
URL: https://github.com/apache/dolphinscheduler/pull/6103#discussion_r702612985



##########
File path: dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/AuditLog.java
##########
@@ -0,0 +1,75 @@
+package org.apache.dolphinscheduler.dao.entity;
+
+import com.baomidou.mybatisplus.annotation.IdType;
+import com.baomidou.mybatisplus.annotation.TableId;
+import com.baomidou.mybatisplus.annotation.TableName;
+import com.fasterxml.jackson.annotation.JsonFormat;
+import org.apache.dolphinscheduler.common.enums.AuditModuleType;
+import org.apache.dolphinscheduler.common.enums.AuditOperationType;
+
+import java.util.Date;
+
+@TableName("t_ds_audit_log")
+public class AuditLog {
+
+    /**
+     * id
+     */
+    @TableId(value = "id", type = IdType.AUTO)
+    private int id;
+
+    /**
+     * user id
+     */
+    private int userId;
+
+    /**
+     * operation module
+     */
+    private AuditModuleType module;

Review comment:
       > Don't use Enum here, this might make code comples.
   
   If so, user might need to use input boxes instead of drop-down boxes to query audit log. 
   In this way, when inserting new audit logs, we need to input modules and operations in String class again.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [dolphinscheduler] ruanwenjun commented on a change in pull request #6103: [Feature-5822][api] Add audit log

Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on a change in pull request #6103:
URL: https://github.com/apache/dolphinscheduler/pull/6103#discussion_r702414468



##########
File path: dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/audit/AuditPublishService.java
##########
@@ -0,0 +1,87 @@
+package org.apache.dolphinscheduler.api.audit;
+
+import org.apache.dolphinscheduler.api.configuration.AuditConfiguration;
+import org.apache.dolphinscheduler.common.enums.AuditModuleType;
+import org.apache.dolphinscheduler.common.enums.AuditOperationType;
+import org.apache.dolphinscheduler.dao.entity.User;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Component;
+
+import javax.annotation.PostConstruct;
+import java.util.Date;
+import java.util.List;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@Component
+public class AuditPublishService {
+
+    /**
+     * audit message queue
+     */
+    private BlockingQueue<AuditMessage> queue = new LinkedBlockingQueue<>(1000);
+
+    /**
+     * subscribers list
+     */
+    @Autowired
+    private List<AuditSubscriber> subscribers;
+
+    @Autowired
+    AuditConfiguration auditConfiguration;
+
+    private static final Logger logger = LoggerFactory.getLogger(AuditPublishService.class);
+
+    /**
+     * create a daemon thread to process the message queue
+     */
+    @PostConstruct
+    private void init() {
+        Thread thread = new Thread(() -> doPublish());
+        thread.setDaemon(true);

Review comment:
       Set thread name

##########
File path: dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/audit/AuditSubscriber.java
##########
@@ -0,0 +1,11 @@
+package org.apache.dolphinscheduler.api.audit;
+
+public interface AuditSubscriber {
+
+    /**
+     * process the message

Review comment:
       ```suggestion
   process the audit message
   ```

##########
File path: dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/AuditLog.java
##########
@@ -0,0 +1,75 @@
+package org.apache.dolphinscheduler.dao.entity;
+
+import com.baomidou.mybatisplus.annotation.IdType;
+import com.baomidou.mybatisplus.annotation.TableId;
+import com.baomidou.mybatisplus.annotation.TableName;
+import com.fasterxml.jackson.annotation.JsonFormat;
+import org.apache.dolphinscheduler.common.enums.AuditModuleType;
+import org.apache.dolphinscheduler.common.enums.AuditOperationType;
+
+import java.util.Date;
+
+@TableName("t_ds_audit_log")
+public class AuditLog {
+
+    /**
+     * id
+     */
+    @TableId(value = "id", type = IdType.AUTO)
+    private int id;
+
+    /**
+     * user id
+     */
+    private int userId;
+
+    /**
+     * operation module
+     */
+    private AuditModuleType module;

Review comment:
       Don't use Enum here, this might make code comples.

##########
File path: dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/audit/AuditPublishService.java
##########
@@ -0,0 +1,87 @@
+package org.apache.dolphinscheduler.api.audit;
+
+import org.apache.dolphinscheduler.api.configuration.AuditConfiguration;
+import org.apache.dolphinscheduler.common.enums.AuditModuleType;
+import org.apache.dolphinscheduler.common.enums.AuditOperationType;
+import org.apache.dolphinscheduler.dao.entity.User;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Component;
+
+import javax.annotation.PostConstruct;
+import java.util.Date;
+import java.util.List;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@Component
+public class AuditPublishService {
+
+    /**
+     * audit message queue
+     */
+    private BlockingQueue<AuditMessage> queue = new LinkedBlockingQueue<>(1000);
+
+    /**
+     * subscribers list
+     */
+    @Autowired
+    private List<AuditSubscriber> subscribers;
+
+    @Autowired
+    AuditConfiguration auditConfiguration;
+
+    private static final Logger logger = LoggerFactory.getLogger(AuditPublishService.class);
+
+    /**
+     * create a daemon thread to process the message queue
+     */
+    @PostConstruct
+    private void init() {
+        Thread thread = new Thread(() -> doPublish());
+        thread.setDaemon(true);
+        thread.start();
+    }
+
+    /**
+     * publish a new audit message
+     *
+     * @param message audit message
+     */
+    public void publish(AuditMessage message) {
+        queue.offer(message);
+    }
+
+    /**
+     *  subscribers execute the message processor method
+     */
+    private void doPublish() {
+        AuditMessage message;
+        if (auditConfiguration.isAuditGlobalControlSwitch()) {

Review comment:
       There might be a bug, if the switch set to false, the application will still add `AuditMessage` to the queue.
   
   It is better to move this switch to `publish` method and `init` method.

##########
File path: dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/AuditLog.java
##########
@@ -0,0 +1,75 @@
+package org.apache.dolphinscheduler.dao.entity;
+
+import com.baomidou.mybatisplus.annotation.IdType;
+import com.baomidou.mybatisplus.annotation.TableId;
+import com.baomidou.mybatisplus.annotation.TableName;
+import com.fasterxml.jackson.annotation.JsonFormat;
+import org.apache.dolphinscheduler.common.enums.AuditModuleType;
+import org.apache.dolphinscheduler.common.enums.AuditOperationType;
+
+import java.util.Date;
+
+@TableName("t_ds_audit_log")
+public class AuditLog {
+
+    /**
+     * id
+     */
+    @TableId(value = "id", type = IdType.AUTO)
+    private int id;
+
+    /**
+     * user id
+     */
+    private int userId;
+
+    /**
+     * operation module
+     */
+    private AuditModuleType module;
+
+    /**
+     * operation
+     */
+    private AuditOperationType operation;
+
+    /**
+     * operation time
+     */
+    @JsonFormat(pattern = "yyyy-MM-dd HH:mm:ss",timezone="GMT+8")
+    private Date time;
+
+    // TODO : 添加项目

Review comment:
       Remove the Chinese.

##########
File path: sql/dolphinscheduler_mysql.sql
##########
@@ -968,3 +968,16 @@ CREATE TABLE `t_ds_alert_plugin_instance` (
   `instance_name` varchar(200) DEFAULT NULL COMMENT 'alert instance name',
   PRIMARY KEY (`id`)
 ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+
+-- ----------------------------
+-- Table structure for t_ds_alert_plugin_instance
+-- ----------------------------
+DROP TABLE IF EXISTS `t_ds_audit_log`;
+CREATE TABLE `t_ds_audit_log` (
+  `id` int NOT NULL AUTO_INCREMENT,
+  `user_id` int(11) NOT NULL COMMENT 'user id',
+  `module` int DEFAULT NULL COMMENT 'module',
+  `operation` int DEFAULT NULL COMMENT 'operation',
+  `time` datetime DEFAULT NULL COMMENT 'create time',

Review comment:
       ```suggestion
   `time` datetime DEFAULT CURRENT_TIMESTAMP COMMENT 'create time',
   ```
   

##########
File path: dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/audit/AuditMessage.java
##########
@@ -0,0 +1,66 @@
+package org.apache.dolphinscheduler.api.audit;
+
+import org.apache.dolphinscheduler.common.enums.AuditModuleType;
+import org.apache.dolphinscheduler.common.enums.AuditOperationType;
+import org.apache.dolphinscheduler.dao.entity.User;
+
+import java.util.Date;
+
+public class AuditMessage {
+    private String messageType;

Review comment:
       
   ```suggestion
   
   ```
   Remove the unused attribute.

##########
File path: dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/configuration/AuditConfiguration.java
##########
@@ -0,0 +1,20 @@
+package org.apache.dolphinscheduler.api.configuration;
+
+import org.springframework.beans.factory.annotation.Value;
+import org.springframework.context.annotation.Configuration;
+
+@Configuration
+public class AuditConfiguration {
+    @Value("${audit.control.global.switch:true}")
+    private boolean auditGlobalControlSwitch;
+
+
+

Review comment:
       ```suggestion
   ```

##########
File path: dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/audit/AuditPublishService.java
##########
@@ -0,0 +1,87 @@
+package org.apache.dolphinscheduler.api.audit;
+
+import org.apache.dolphinscheduler.api.configuration.AuditConfiguration;
+import org.apache.dolphinscheduler.common.enums.AuditModuleType;
+import org.apache.dolphinscheduler.common.enums.AuditOperationType;
+import org.apache.dolphinscheduler.dao.entity.User;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Component;
+
+import javax.annotation.PostConstruct;
+import java.util.Date;
+import java.util.List;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@Component
+public class AuditPublishService {
+
+    /**
+     * audit message queue
+     */
+    private BlockingQueue<AuditMessage> queue = new LinkedBlockingQueue<>(1000);
+
+    /**
+     * subscribers list
+     */
+    @Autowired
+    private List<AuditSubscriber> subscribers;
+
+    @Autowired
+    AuditConfiguration auditConfiguration;
+
+    private static final Logger logger = LoggerFactory.getLogger(AuditPublishService.class);
+
+    /**
+     * create a daemon thread to process the message queue
+     */
+    @PostConstruct
+    private void init() {
+        Thread thread = new Thread(() -> doPublish());
+        thread.setDaemon(true);
+        thread.start();
+    }
+
+    /**
+     * publish a new audit message
+     *
+     * @param message audit message
+     */
+    public void publish(AuditMessage message) {
+        queue.offer(message);
+    }
+
+    /**
+     *  subscribers execute the message processor method
+     */
+    private void doPublish() {
+        AuditMessage message;
+        if (auditConfiguration.isAuditGlobalControlSwitch()) {
+            while (true) {
+                try {
+                    message = queue.take();
+                    for (AuditSubscriber subscriber : subscribers) {
+                        try {
+                            subscriber.execute(message);
+                        } catch (Exception e) {
+                            logger.error("subscriber fail to process message {}", message.getMessageType());
+                        }
+                    }
+                } catch (InterruptedException e) {
+                    Thread.currentThread().interrupt();
+                }
+            }
+        }
+    }
+
+    /**
+     *  generate new Message
+     *
+     */
+    public AuditMessage generateMessage(User user, Date auditDate, AuditModuleType module, AuditOperationType operation) {
+        return new AuditMessage(user, auditDate, module, operation);
+    }

Review comment:
       ```suggestion
   ```
   Remove the unused method




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org