You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@rocketmq.apache.org by du...@apache.org on 2021/08/03 06:50:28 UTC
[rocketmq-streams] branch main updated (7981c23 -> b3063f0)
This is an automated email from the ASF dual-hosted git repository.
duhengforever pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/rocketmq-streams.git.
from 7981c23 Update README.md
new 4ce3b3b Remove .iml file
new 480b1cd Merge branch 'apache:main' into main
new 3209044 add dbinit module, it will create tables into user's database, the first version can support MySql only
new b3063f0 Merge pull request #8 from xstorm1/dev
The 35 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
pom.xml | 2 +-
.../streams/common/configure/ConfigureFileKey.java | 1 +
.../rocketmq-streams-db-operator.iml | 16 --
.../rocketmq/streams/db/driver/orm/ORMUtil.java | 23 +++
.../pom.xml | 37 ++--
.../streams/dbinit/mysql/delegate/DBDelegate.java | 9 +
.../dbinit/mysql/delegate/DBDelegateFactory.java | 27 +++
.../streams/dbinit/mysql/delegate/DBType.java | 6 +
.../dbinit/mysql/delegate/MysqlDelegate.java | 48 +++++
.../src/main/resources/tables_mysql_innodb.sql | 199 +++++++++++++++++++++
10 files changed, 339 insertions(+), 29 deletions(-)
delete mode 100644 rocketmq-streams-db-operator/rocketmq-streams-db-operator.iml
copy {rocketmq-streams-lease => rocketmq-streams-dbinit}/pom.xml (54%)
mode change 100755 => 100644
create mode 100644 rocketmq-streams-dbinit/src/main/java/org/apache/rocketmq/streams/dbinit/mysql/delegate/DBDelegate.java
create mode 100644 rocketmq-streams-dbinit/src/main/java/org/apache/rocketmq/streams/dbinit/mysql/delegate/DBDelegateFactory.java
create mode 100644 rocketmq-streams-dbinit/src/main/java/org/apache/rocketmq/streams/dbinit/mysql/delegate/DBType.java
create mode 100644 rocketmq-streams-dbinit/src/main/java/org/apache/rocketmq/streams/dbinit/mysql/delegate/MysqlDelegate.java
create mode 100644 rocketmq-streams-dbinit/src/main/resources/tables_mysql_innodb.sql
[rocketmq-streams] 23/35: modify README.md
Posted by du...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
duhengforever pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/rocketmq-streams.git
commit aa81fe8256e908dc12c2757cbfd3d71f237f9a14
Author: xstorm1 <xs...@live.cn>
AuthorDate: Mon Aug 2 15:48:21 2021 +0800
modify README.md
---
README.md | 57 +--------------------------------------------------------
1 file changed, 1 insertion(+), 56 deletions(-)
diff --git a/README.md b/README.md
index d2ff23e..92a9b2f 100644
--- a/README.md
+++ b/README.md
@@ -8,19 +8,6 @@
## DataStream Example
```java
-<<<<<<< HEAD
- import org.apache.rocketmq.streams.client.transform.DataStream;
-
-
-
- DataStreamSource source=StreamBuilder.dataStream("namespace","pipeline");
-
- source
- .fromFile("/Users/junjie.cheng/text.txt",false)
- .map(message->message)
- .toPrint(1)
- .start();
-=======
import org.apache.rocketmq.streams.client.transform.DataStream;
DataStreamSource source=StreamBuilder.dataStream("namespace","pipeline");
@@ -30,7 +17,6 @@ DataStreamSource source=StreamBuilder.dataStream("namespace","pipeline");
.map(message->message)
.toPrint(1)
.start();
->>>>>>> main
```
## Maven Repository
@@ -58,10 +44,7 @@ StreamBuilder 用于构建流任务的源； 内部包含```dataStream()```和``
## DataStream API
### Source
-<<<<<<< HEAD
-=======
->>>>>>> main
DataStreamSource 是分段式编程的源头类，用于对接各种数据源， 从各大消息队列中获取数据；
+ ```fromFile``` 从文件中读取数据， 该方法包含俩个参数
@@ -75,15 +58,6 @@ DataStreamSource 是分段式编程的源头类，用于对接各种数据源，
+ ```isJson``` 是否json格式，非必填参数
+ ```tags``` rocketmq消费的tags值，用于过滤消息，非必填参数
-<<<<<<< HEAD
-
-+ ```from``` 自定义的数据源， 通过实现ISource接口实现自己的数据源
-
-### transform
-transform 允许在流计算过程中对输入源的数据进行修改，进行下一步的操作；DataStream API中包括```DataStream```,```JoinStream```, ```SplitStream```,```WindowStream```等多个transform类；
-
-#### DataStream
-=======
+ ```from``` 自定义的数据源， 通过实现ISource接口实现自己的数据源
### transform
@@ -92,7 +66,6 @@ transform 允许在流计算过程中对输入源的数据进行修改，进行
#### DataStream
->>>>>>> main
DataStream实现了一系列常见的流计算算子
+ ```map``` 通过将源的每个记录传递给函数func来返回一个新的DataStream
@@ -106,17 +79,6 @@ DataStream实现了一系列常见的流计算算子
+ ```toFile``` 将结果保存为文件，生成一个新的DataStreamAction实例
+ ```toDB``` 将结果保存到数据库
+ ```toRocketmq``` 将结果输出到rocketmq
-<<<<<<< HEAD
-+ ```toSls``` 将结果输出到sls
-+ ```to``` 将结果经过自定义的ISink接口输出到指定的存储
-+ ```window``` 在窗口内进行相关的统计分析，一般会与```groupBy```连用， ```window()```用来定义窗口的大小， ```groupBy()```用来定义统计分析的主key，可以指定多个
- + ```count``` 在窗口内计数
- + ```min``` 获取窗口内统计值的最小值
- + ```max``` 获取窗口内统计值得最大值
- + ```avg``` 获取窗口内统计值的平均值
- + ```sum``` 获取窗口内统计值的加和值
- + ```reduce``` 在窗口内进行自定义的汇总运算
-=======
+ ```to``` 将结果经过自定义的ISink接口输出到指定的存储
+ ```window``` 在窗口内进行相关的统计分析，一般会与```groupBy```连用， ```window()```用来定义窗口的大小， ```groupBy()```用来定义统计分析的主key，可以指定多个
+ ```count``` 在窗口内计数
@@ -125,27 +87,11 @@ DataStream实现了一系列常见的流计算算子
+ ```avg``` 获取窗口内统计值的平均值
+ ```sum``` 获取窗口内统计值的加和值
+ ```reduce``` 在窗口内进行自定义的汇总运算
->>>>>>> main
+ ```join``` 根据条件将将俩个流进行关联， 合并为一个大流进行相关的运算
+ ```union``` 将俩个流进行合并
+ ```split``` 将一个数据流按照标签进行拆分，分为不同的数据流供下游进行分析计算
+ ```with``` with算子用来指定计算过程中的相关策略，包括checkpoint的存储策略，state的存储策略等
-<<<<<<< HEAD
-
-# Strategy
-策略机制主要用来控制计算引擎运行过程中的底层逻辑，如checkpoint，state的存储方式等，后续还会增加对窗口、双流join等的控制；所有的控制策略通过```with```算子传入，可以同时传入多个策略类型；
-
-```java
- //指定checkpoint的存储策略
- source
- .fromRocketmq("TSG_META_INFO", "")
- .map(message -> message + "--")
- .toPrint(1)
- .with(CheckpointStrategy.db("jdbc:mysql://XXXXX:3306/XXXXX", "", "", 0L))
- .start();
-```
-=======
# Strategy
策略机制主要用来控制计算引擎运行过程中的底层逻辑，如checkpoint，state的存储方式等，后续还会增加对窗口、双流join等的控制；所有的控制策略通过```with```算子传入，可以同时传入多个策略类型；
@@ -158,5 +104,4 @@ source
.toPrint(1)
.with(CheckpointStrategy.db("jdbc:mysql://XXXXX:3306/XXXXX","","",0L))
.start();
-```
->>>>>>> main
+```
\ No newline at end of file
[rocketmq-streams] 24/35: delete target directory
Posted by du...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
duhengforever pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/rocketmq-streams.git
commit 513deafd5159264fd5af9b8aa5f0c8637c4ef0eb
Author: xstorm1 <xs...@live.cn>
AuthorDate: Mon Aug 2 16:26:32 2021 +0800
delete target directory
---
.../rocketmq-streams-channel-rocketmq.iml | 74 -
....streams.common.channel.builder.IChannelBuilder | 1 -
.../rocketmq/streams/RocketMQChannelBuilder.class | Bin 3971 -> 0 bytes
.../apache/rocketmq/streams/RocketMQOffset.class | Bin 3628 -> 0 bytes
.../streams/queue/RocketMQMessageQueue.class | Bin 3849 -> 0 bytes
.../rocketmq/streams/sink/RocketMQSink$1.class | Bin 2040 -> 0 bytes
.../rocketmq/streams/sink/RocketMQSink.class | Bin 10455 -> 0 bytes
.../rocketmq/streams/source/RocketMQSource$1.class | Bin 3170 -> 0 bytes
.../rocketmq/streams/source/RocketMQSource.class | Bin 12146 -> 0 bytes
.../target/maven-archiver/pom.properties | 5 -
...ams-channel-rocketmq-2.0.0-SNAPSHOT-sources.jar | Bin 13382 -> 0 bytes
...tmq-streams-channel-rocketmq-2.0.0-SNAPSHOT.jar | Bin 20715 -> 0 bytes
.../rocketmq-streams-serviceloader.iml | 61 -
...ams.serviceloader.namefinder.IServiceNameGetter | 1 -
.../component/ServiceLoaderComponent.properties | 1 -
.../serviceloader/IServiceLoaderService.class | Bin 413 -> 0 bytes
.../serviceloader/ServiceLoaderComponent.class | Bin 6038 -> 0 bytes
.../namefinder/IServiceNameGetter.class | Bin 235 -> 0 bytes
.../impl/AnnotationServiceNameGetter.class | Bin 1138 -> 0 bytes
.../streams/serviceloader/utils/FileUtil.class | Bin 3352 -> 0 bytes
.../target/maven-archiver/pom.properties | 5 -
...treams-serviceloader-2.0.0-SNAPSHOT-sources.jar | Bin 8077 -> 0 bytes
...cketmq-streams-serviceloader-2.0.0-SNAPSHOT.jar | Bin 9883 -> 0 bytes
.../target/test-classes/log4j.xml | 20 -
.../serviceloader/ServiceLoaderComponentTest.class | Bin 1558 -> 0 bytes
.../rocketmq-streams-window.iml | 82 -
rocketmq-streams-window/target/classes/dipper.cs | 2892 --------------------
.../target/classes/dipper.properties | 21 -
.../streams/window/builder/WindowBuilder.class | Bin 3057 -> 0 bytes
.../streams/window/model/FunctionExecutor.class | Bin 1591 -> 0 bytes
.../streams/window/model/WindowCache.class | Bin 6328 -> 0 bytes
.../streams/window/model/WindowInstance.class | Bin 14044 -> 0 bytes
.../window/offset/IWindowMaxValueManager.class | Bin 1137 -> 0 bytes
.../streams/window/offset/WindowMaxValue.class | Bin 2113 -> 0 bytes
.../window/offset/WindowMaxValueManager.class | Bin 9412 -> 0 bytes
.../window/operator/AbstractShuffleWindow.class | Bin 4175 -> 0 bytes
.../streams/window/operator/AbstractWindow$1.class | Bin 1063 -> 0 bytes
.../streams/window/operator/AbstractWindow.class | Bin 29704 -> 0 bytes
.../streams/window/operator/impl/OverWindow.class | Bin 5387 -> 0 bytes
.../window/operator/impl/SessionWindow.class | Bin 13002 -> 0 bytes
.../window/operator/impl/WindowOperator$1.class | Bin 1944 -> 0 bytes
.../impl/WindowOperator$WindowRowOperator.class | Bin 3973 -> 0 bytes
.../window/operator/impl/WindowOperator.class | Bin 17416 -> 0 bytes
.../streams/window/operator/join/DBOperator.class | Bin 16903 -> 0 bytes
.../window/operator/join/JoinWindow$1.class | Bin 2931 -> 0 bytes
.../window/operator/join/JoinWindow$2.class | Bin 2870 -> 0 bytes
.../streams/window/operator/join/JoinWindow.class | Bin 21377 -> 0 bytes
.../streams/window/operator/join/Operator.class | Bin 2912 -> 0 bytes
.../window/shuffle/AbstractSystemChannel.class | Bin 11888 -> 0 bytes
.../ShuffleChannel$ShuffleOutputDataSource.class | Bin 5054 -> 0 bytes
.../streams/window/shuffle/ShuffleChannel.class | Bin 22485 -> 0 bytes
.../window/source/WindowRireSource$1$1.class | Bin 1524 -> 0 bytes
.../streams/window/source/WindowRireSource$1.class | Bin 2346 -> 0 bytes
.../WindowRireSource$WindowInstanceCache$1$1.class | Bin 1833 -> 0 bytes
.../WindowRireSource$WindowInstanceCache$1.class | Bin 2199 -> 0 bytes
.../WindowRireSource$WindowInstanceCache.class | Bin 1592 -> 0 bytes
.../streams/window/source/WindowRireSource.class | Bin 8081 -> 0 bytes
.../streams/window/state/WindowBaseValue.class | Bin 4083 -> 0 bytes
.../streams/window/state/impl/JoinLeftState.class | Bin 395 -> 0 bytes
.../streams/window/state/impl/JoinRightState.class | Bin 398 -> 0 bytes
.../streams/window/state/impl/JoinState.class | Bin 1943 -> 0 bytes
.../streams/window/state/impl/WindowValue$1.class | Bin 930 -> 0 bytes
.../streams/window/state/impl/WindowValue.class | Bin 23728 -> 0 bytes
.../window/storage/AbstractWindowStorage$1.class | Bin 2596 -> 0 bytes
.../window/storage/AbstractWindowStorage.class | Bin 6724 -> 0 bytes
.../streams/window/storage/ICommonStorage.class | Bin 968 -> 0 bytes
.../streams/window/storage/IKeyGenerator.class | Bin 193 -> 0 bytes
.../window/storage/IShufflePartitionManager.class | Bin 642 -> 0 bytes
.../rocketmq/streams/window/storage/IStorage.class | Bin 2875 -> 0 bytes
.../streams/window/storage/IWindowStorage.class | Bin 2282 -> 0 bytes
.../window/storage/ShufflePartitionManager.class | Bin 2851 -> 0 bytes
.../streams/window/storage/StorageManager.class | Bin 353 -> 0 bytes
.../streams/window/storage/WindowStorage$1.class | Bin 1402 -> 0 bytes
.../WindowStorage$WindowBaseValueIterator.class | Bin 980 -> 0 bytes
.../streams/window/storage/WindowStorage.class | Bin 7364 -> 0 bytes
.../streams/window/storage/db/DBStorage$1.class | Bin 1523 -> 0 bytes
.../window/storage/db/DBStorage$DBIterator.class | Bin 4299 -> 0 bytes
.../streams/window/storage/db/DBStorage.class | Bin 7161 -> 0 bytes
.../window/storage/file/FileStorage$1.class | Bin 2620 -> 0 bytes
.../streams/window/storage/file/FileStorage.class | Bin 9198 -> 0 bytes
.../rocksdb/RocksdbStorage$LocalIterator.class | Bin 3346 -> 0 bytes
.../window/storage/rocksdb/RocksdbStorage.class | Bin 11737 -> 0 bytes
.../target/maven-archiver/pom.properties | 5 -
...ketmq-streams-window-2.0.0-SNAPSHOT-sources.jar | Bin 357527 -> 0 bytes
.../rocketmq-streams-window-2.0.0-SNAPSHOT.jar | Bin 403392 -> 0 bytes
.../target/test-classes/log4j.xml | 20 -
86 files changed, 3188 deletions(-)
diff --git a/rocketmq-streams-channel-rocketmq/rocketmq-streams-channel-rocketmq.iml b/rocketmq-streams-channel-rocketmq/rocketmq-streams-channel-rocketmq.iml
deleted file mode 100644
index d0b5aa3..0000000
--- a/rocketmq-streams-channel-rocketmq/rocketmq-streams-channel-rocketmq.iml
+++ /dev/null
@@ -1,74 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<module org.jetbrains.idea.maven.project.MavenProjectsManager.isMavenModule="true" type="JAVA_MODULE" version="4">
- <component name="NewModuleRootManager" LANGUAGE_LEVEL="JDK_1_8">
- <output url="file://$MODULE_DIR$/target/classes" />
- <output-test url="file://$MODULE_DIR$/target/test-classes" />
- <content url="file://$MODULE_DIR$">
- <sourceFolder url="file://$MODULE_DIR$/src/main/java" isTestSource="false" />
- <sourceFolder url="file://$MODULE_DIR$/src/test/java" isTestSource="true" />
- <excludeFolder url="file://$MODULE_DIR$/target" />
- </content>
- <orderEntry type="inheritedJdk" />
- <orderEntry type="sourceFolder" forTests="false" />
- <orderEntry type="module" module-name="rocketmq-streams-commons" />
- <orderEntry type="library" name="Maven: org.apache.commons:commons-lang3:3.11" level="project" />
- <orderEntry type="library" name="Maven: com.alibaba:fastjson:1.2.27" level="project" />
- <orderEntry type="library" name="Maven: commons-logging:commons-logging:1.1" level="project" />
- <orderEntry type="library" name="Maven: logkit:logkit:1.0.1" level="project" />
- <orderEntry type="library" name="Maven: avalon-framework:avalon-framework:4.1.3" level="project" />
- <orderEntry type="library" name="Maven: javax.servlet:servlet-api:2.3" level="project" />
- <orderEntry type="library" name="Maven: commons-io:commons-io:2.5" level="project" />
- <orderEntry type="library" name="Maven: com.google.code.gson:gson:2.8.5" level="project" />
- <orderEntry type="library" name="Maven: com.google.auto.service:auto-service:1.0-rc5" level="project" />
- <orderEntry type="library" name="Maven: com.google.auto.service:auto-service-annotations:1.0-rc5" level="project" />
- <orderEntry type="library" name="Maven: com.google.auto:auto-common:0.10" level="project" />
- <orderEntry type="library" name="Maven: com.google.guava:guava:25.1-jre" level="project" />
- <orderEntry type="library" name="Maven: com.google.code.findbugs:jsr305:3.0.2" level="project" />
- <orderEntry type="library" name="Maven: org.checkerframework:checker-qual:2.0.0" level="project" />
- <orderEntry type="library" name="Maven: com.google.errorprone:error_prone_annotations:2.1.3" level="project" />
- <orderEntry type="library" name="Maven: com.google.j2objc:j2objc-annotations:1.1" level="project" />
- <orderEntry type="library" name="Maven: org.codehaus.mojo:animal-sniffer-annotations:1.14" level="project" />
- <orderEntry type="library" name="Maven: com.lmax:disruptor:3.2.0" level="project" />
- <orderEntry type="library" name="Maven: com.gliwka.hyperscan:hyperscan:5.4.0-2.0.0" level="project" />
- <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:5.4.0-1.0.0" level="project" />
- <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:linux-x86_64:5.4.0-1.0.0" level="project" />
- <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:windows-x86_64:5.4.0-1.0.0" level="project" />
- <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:macosx-x86_64:5.4.0-1.0.0" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp-platform:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-arm:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-arm64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-x86:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-x86_64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:ios-arm64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:ios-x86_64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-armhf:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-arm64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-ppc64le:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-x86:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-x86_64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:macosx-x86_64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:windows-x86:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:windows-x86_64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: net.java.dev.jna:platform:3.5.2" level="project" />
- <orderEntry type="library" name="Maven: net.java.dev.jna:jna:3.5.2" level="project" />
- <orderEntry type="module" module-name="rocketmq-streams-serviceloader" />
- <orderEntry type="library" name="Maven: org.apache.rocketmq:rocketmq-client:4.5.2" level="project" />
- <orderEntry type="library" name="Maven: org.apache.rocketmq:rocketmq-common:4.5.2" level="project" />
- <orderEntry type="library" name="Maven: org.apache.rocketmq:rocketmq-acl:4.5.2" level="project" />
- <orderEntry type="library" name="Maven: org.apache.rocketmq:rocketmq-remoting:4.5.2" level="project" />
- <orderEntry type="library" name="Maven: io.netty:netty-all:4.0.42.Final" level="project" />
- <orderEntry type="library" name="Maven: io.netty:netty-tcnative-boringssl-static:1.1.33.Fork26" level="project" />
- <orderEntry type="library" name="Maven: org.apache.rocketmq:rocketmq-logging:4.5.2" level="project" />
- <orderEntry type="library" name="Maven: org.apache.rocketmq:rocketmq-srvutil:4.5.2" level="project" />
- <orderEntry type="library" name="Maven: commons-cli:commons-cli:1.2" level="project" />
- <orderEntry type="library" name="Maven: org.yaml:snakeyaml:1.19" level="project" />
- <orderEntry type="library" name="Maven: commons-codec:commons-codec:1.9" level="project" />
- <orderEntry type="library" scope="TEST" name="Maven: junit:junit:4.12" level="project" />
- <orderEntry type="library" scope="TEST" name="Maven: org.hamcrest:hamcrest-core:1.3" level="project" />
- <orderEntry type="library" name="Maven: org.slf4j:slf4j-api:1.7.7" level="project" />
- <orderEntry type="library" name="Maven: org.slf4j:jcl-over-slf4j:1.7.7" level="project" />
- <orderEntry type="library" name="Maven: org.slf4j:slf4j-log4j12:1.7.7" level="project" />
- <orderEntry type="library" name="Maven: log4j:log4j:1.2.17" level="project" />
- </component>
-</module>
\ No newline at end of file
diff --git a/rocketmq-streams-channel-rocketmq/target/classes/META-INF/services/org.apache.rocketmq.streams.common.channel.builder.IChannelBuilder b/rocketmq-streams-channel-rocketmq/target/classes/META-INF/services/org.apache.rocketmq.streams.common.channel.builder.IChannelBuilder
deleted file mode 100644
index fb9e053..0000000
--- a/rocketmq-streams-channel-rocketmq/target/classes/META-INF/services/org.apache.rocketmq.streams.common.channel.builder.IChannelBuilder
+++ /dev/null
@@ -1 +0,0 @@
-org.apache.rocketmq.streams.RocketMQChannelBuilder
diff --git a/rocketmq-streams-channel-rocketmq/target/classes/org/apache/rocketmq/streams/RocketMQChannelBuilder.class b/rocketmq-streams-channel-rocketmq/target/classes/org/apache/rocketmq/streams/RocketMQChannelBuilder.class
deleted file mode 100644
index bcd8f08..0000000
Binary files a/rocketmq-streams-channel-rocketmq/target/classes/org/apache/rocketmq/streams/RocketMQChannelBuilder.class and /dev/null differ
diff --git a/rocketmq-streams-channel-rocketmq/target/classes/org/apache/rocketmq/streams/RocketMQOffset.class b/rocketmq-streams-channel-rocketmq/target/classes/org/apache/rocketmq/streams/RocketMQOffset.class
deleted file mode 100644
index b1f45cf..0000000
Binary files a/rocketmq-streams-channel-rocketmq/target/classes/org/apache/rocketmq/streams/RocketMQOffset.class and /dev/null differ
diff --git a/rocketmq-streams-channel-rocketmq/target/classes/org/apache/rocketmq/streams/queue/RocketMQMessageQueue.class b/rocketmq-streams-channel-rocketmq/target/classes/org/apache/rocketmq/streams/queue/RocketMQMessageQueue.class
deleted file mode 100644
index a9c8a39..0000000
Binary files a/rocketmq-streams-channel-rocketmq/target/classes/org/apache/rocketmq/streams/queue/RocketMQMessageQueue.class and /dev/null differ
diff --git a/rocketmq-streams-channel-rocketmq/target/classes/org/apache/rocketmq/streams/sink/RocketMQSink$1.class b/rocketmq-streams-channel-rocketmq/target/classes/org/apache/rocketmq/streams/sink/RocketMQSink$1.class
deleted file mode 100644
index b2f09df..0000000
Binary files a/rocketmq-streams-channel-rocketmq/target/classes/org/apache/rocketmq/streams/sink/RocketMQSink$1.class and /dev/null differ
diff --git a/rocketmq-streams-channel-rocketmq/target/classes/org/apache/rocketmq/streams/sink/RocketMQSink.class b/rocketmq-streams-channel-rocketmq/target/classes/org/apache/rocketmq/streams/sink/RocketMQSink.class
deleted file mode 100644
index 12fbfb1..0000000
Binary files a/rocketmq-streams-channel-rocketmq/target/classes/org/apache/rocketmq/streams/sink/RocketMQSink.class and /dev/null differ
diff --git a/rocketmq-streams-channel-rocketmq/target/classes/org/apache/rocketmq/streams/source/RocketMQSource$1.class b/rocketmq-streams-channel-rocketmq/target/classes/org/apache/rocketmq/streams/source/RocketMQSource$1.class
deleted file mode 100644
index e9fe35c..0000000
Binary files a/rocketmq-streams-channel-rocketmq/target/classes/org/apache/rocketmq/streams/source/RocketMQSource$1.class and /dev/null differ
diff --git a/rocketmq-streams-channel-rocketmq/target/classes/org/apache/rocketmq/streams/source/RocketMQSource.class b/rocketmq-streams-channel-rocketmq/target/classes/org/apache/rocketmq/streams/source/RocketMQSource.class
deleted file mode 100644
index e186384..0000000
Binary files a/rocketmq-streams-channel-rocketmq/target/classes/org/apache/rocketmq/streams/source/RocketMQSource.class and /dev/null differ
diff --git a/rocketmq-streams-channel-rocketmq/target/maven-archiver/pom.properties b/rocketmq-streams-channel-rocketmq/target/maven-archiver/pom.properties
deleted file mode 100644
index a605b53..0000000
--- a/rocketmq-streams-channel-rocketmq/target/maven-archiver/pom.properties
+++ /dev/null
@@ -1,5 +0,0 @@
-#Generated by Maven
-#Fri Jul 30 11:10:37 CST 2021
-version=2.0.0-SNAPSHOT
-groupId=org.apache.rocketmq
-artifactId=rocketmq-streams-channel-rocketmq
diff --git a/rocketmq-streams-channel-rocketmq/target/rocketmq-streams-channel-rocketmq-2.0.0-SNAPSHOT-sources.jar b/rocketmq-streams-channel-rocketmq/target/rocketmq-streams-channel-rocketmq-2.0.0-SNAPSHOT-sources.jar
deleted file mode 100644
index 6a2ed7f..0000000
Binary files a/rocketmq-streams-channel-rocketmq/target/rocketmq-streams-channel-rocketmq-2.0.0-SNAPSHOT-sources.jar and /dev/null differ
diff --git a/rocketmq-streams-channel-rocketmq/target/rocketmq-streams-channel-rocketmq-2.0.0-SNAPSHOT.jar b/rocketmq-streams-channel-rocketmq/target/rocketmq-streams-channel-rocketmq-2.0.0-SNAPSHOT.jar
deleted file mode 100644
index 333a58d..0000000
Binary files a/rocketmq-streams-channel-rocketmq/target/rocketmq-streams-channel-rocketmq-2.0.0-SNAPSHOT.jar and /dev/null differ
diff --git a/rocketmq-streams-serviceloader/rocketmq-streams-serviceloader.iml b/rocketmq-streams-serviceloader/rocketmq-streams-serviceloader.iml
deleted file mode 100644
index 5a29414..0000000
--- a/rocketmq-streams-serviceloader/rocketmq-streams-serviceloader.iml
+++ /dev/null
@@ -1,61 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<module org.jetbrains.idea.maven.project.MavenProjectsManager.isMavenModule="true" type="JAVA_MODULE" version="4">
- <component name="NewModuleRootManager" LANGUAGE_LEVEL="JDK_1_8">
- <output url="file://$MODULE_DIR$/target/classes" />
- <output-test url="file://$MODULE_DIR$/target/test-classes" />
- <content url="file://$MODULE_DIR$">
- <sourceFolder url="file://$MODULE_DIR$/src/main/java" isTestSource="false" />
- <sourceFolder url="file://$MODULE_DIR$/src/main/resources" type="java-resource" />
- <sourceFolder url="file://$MODULE_DIR$/src/test/java" isTestSource="true" />
- <sourceFolder url="file://$MODULE_DIR$/src/test/resources" type="java-test-resource" />
- <excludeFolder url="file://$MODULE_DIR$/target" />
- </content>
- <orderEntry type="inheritedJdk" />
- <orderEntry type="sourceFolder" forTests="false" />
- <orderEntry type="module" module-name="rocketmq-streams-commons" />
- <orderEntry type="library" name="Maven: org.apache.commons:commons-lang3:3.11" level="project" />
- <orderEntry type="library" name="Maven: junit:junit:4.12" level="project" />
- <orderEntry type="library" name="Maven: org.hamcrest:hamcrest-core:1.3" level="project" />
- <orderEntry type="library" name="Maven: com.alibaba:fastjson:1.2.27" level="project" />
- <orderEntry type="library" name="Maven: commons-logging:commons-logging:1.1" level="project" />
- <orderEntry type="library" name="Maven: logkit:logkit:1.0.1" level="project" />
- <orderEntry type="library" name="Maven: avalon-framework:avalon-framework:4.1.3" level="project" />
- <orderEntry type="library" name="Maven: javax.servlet:servlet-api:2.3" level="project" />
- <orderEntry type="library" name="Maven: commons-io:commons-io:2.5" level="project" />
- <orderEntry type="library" name="Maven: log4j:log4j:1.2.17" level="project" />
- <orderEntry type="library" name="Maven: com.google.code.gson:gson:2.8.5" level="project" />
- <orderEntry type="library" name="Maven: com.google.auto.service:auto-service:1.0-rc5" level="project" />
- <orderEntry type="library" name="Maven: com.google.auto.service:auto-service-annotations:1.0-rc5" level="project" />
- <orderEntry type="library" name="Maven: com.google.auto:auto-common:0.10" level="project" />
- <orderEntry type="library" name="Maven: com.google.guava:guava:25.1-jre" level="project" />
- <orderEntry type="library" name="Maven: com.google.code.findbugs:jsr305:3.0.2" level="project" />
- <orderEntry type="library" name="Maven: org.checkerframework:checker-qual:2.0.0" level="project" />
- <orderEntry type="library" name="Maven: com.google.errorprone:error_prone_annotations:2.1.3" level="project" />
- <orderEntry type="library" name="Maven: com.google.j2objc:j2objc-annotations:1.1" level="project" />
- <orderEntry type="library" name="Maven: org.codehaus.mojo:animal-sniffer-annotations:1.14" level="project" />
- <orderEntry type="library" name="Maven: com.lmax:disruptor:3.2.0" level="project" />
- <orderEntry type="library" name="Maven: com.gliwka.hyperscan:hyperscan:5.4.0-2.0.0" level="project" />
- <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:5.4.0-1.0.0" level="project" />
- <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:linux-x86_64:5.4.0-1.0.0" level="project" />
- <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:windows-x86_64:5.4.0-1.0.0" level="project" />
- <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:macosx-x86_64:5.4.0-1.0.0" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp-platform:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-arm:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-arm64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-x86:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-x86_64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:ios-arm64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:ios-x86_64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-armhf:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-arm64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-ppc64le:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-x86:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-x86_64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:macosx-x86_64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:windows-x86:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:windows-x86_64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: net.java.dev.jna:platform:3.5.2" level="project" />
- <orderEntry type="library" name="Maven: net.java.dev.jna:jna:3.5.2" level="project" />
- </component>
-</module>
\ No newline at end of file
diff --git a/rocketmq-streams-serviceloader/target/classes/META-INF/services/org.apache.rocketmq.streams.serviceloader.namefinder.IServiceNameGetter b/rocketmq-streams-serviceloader/target/classes/META-INF/services/org.apache.rocketmq.streams.serviceloader.namefinder.IServiceNameGetter
deleted file mode 100644
index 30fdc46..0000000
--- a/rocketmq-streams-serviceloader/target/classes/META-INF/services/org.apache.rocketmq.streams.serviceloader.namefinder.IServiceNameGetter
+++ /dev/null
@@ -1 +0,0 @@
-org.apache.rocketmq.streams.serviceloader.namefinder.impl.AnnotationServiceNameGetter
diff --git a/rocketmq-streams-serviceloader/target/classes/component/ServiceLoaderComponent.properties b/rocketmq-streams-serviceloader/target/classes/component/ServiceLoaderComponent.properties
deleted file mode 100644
index 0f1b2bb..0000000
--- a/rocketmq-streams-serviceloader/target/classes/component/ServiceLoaderComponent.properties
+++ /dev/null
@@ -1 +0,0 @@
-#serviceName=class.getName()
\ No newline at end of file
diff --git a/rocketmq-streams-serviceloader/target/classes/org/apache/rocketmq/streams/serviceloader/IServiceLoaderService.class b/rocketmq-streams-serviceloader/target/classes/org/apache/rocketmq/streams/serviceloader/IServiceLoaderService.class
deleted file mode 100644
index bb8c29e..0000000
Binary files a/rocketmq-streams-serviceloader/target/classes/org/apache/rocketmq/streams/serviceloader/IServiceLoaderService.class and /dev/null differ
diff --git a/rocketmq-streams-serviceloader/target/classes/org/apache/rocketmq/streams/serviceloader/ServiceLoaderComponent.class b/rocketmq-streams-serviceloader/target/classes/org/apache/rocketmq/streams/serviceloader/ServiceLoaderComponent.class
deleted file mode 100644
index 3831e75..0000000
Binary files a/rocketmq-streams-serviceloader/target/classes/org/apache/rocketmq/streams/serviceloader/ServiceLoaderComponent.class and /dev/null differ
diff --git a/rocketmq-streams-serviceloader/target/classes/org/apache/rocketmq/streams/serviceloader/namefinder/IServiceNameGetter.class b/rocketmq-streams-serviceloader/target/classes/org/apache/rocketmq/streams/serviceloader/namefinder/IServiceNameGetter.class
deleted file mode 100644
index 84de58a..0000000
Binary files a/rocketmq-streams-serviceloader/target/classes/org/apache/rocketmq/streams/serviceloader/namefinder/IServiceNameGetter.class and /dev/null differ
diff --git a/rocketmq-streams-serviceloader/target/classes/org/apache/rocketmq/streams/serviceloader/namefinder/impl/AnnotationServiceNameGetter.class b/rocketmq-streams-serviceloader/target/classes/org/apache/rocketmq/streams/serviceloader/namefinder/impl/AnnotationServiceNameGetter.class
deleted file mode 100644
index 193c479..0000000
Binary files a/rocketmq-streams-serviceloader/target/classes/org/apache/rocketmq/streams/serviceloader/namefinder/impl/AnnotationServiceNameGetter.class and /dev/null differ
diff --git a/rocketmq-streams-serviceloader/target/classes/org/apache/rocketmq/streams/serviceloader/utils/FileUtil.class b/rocketmq-streams-serviceloader/target/classes/org/apache/rocketmq/streams/serviceloader/utils/FileUtil.class
deleted file mode 100644
index e3b051c..0000000
Binary files a/rocketmq-streams-serviceloader/target/classes/org/apache/rocketmq/streams/serviceloader/utils/FileUtil.class and /dev/null differ
diff --git a/rocketmq-streams-serviceloader/target/maven-archiver/pom.properties b/rocketmq-streams-serviceloader/target/maven-archiver/pom.properties
deleted file mode 100644
index 037c58d..0000000
--- a/rocketmq-streams-serviceloader/target/maven-archiver/pom.properties
+++ /dev/null
@@ -1,5 +0,0 @@
-#Generated by Maven
-#Fri Jul 30 11:10:32 CST 2021
-version=2.0.0-SNAPSHOT
-groupId=org.apache.rocketmq
-artifactId=rocketmq-streams-serviceloader
diff --git a/rocketmq-streams-serviceloader/target/rocketmq-streams-serviceloader-2.0.0-SNAPSHOT-sources.jar b/rocketmq-streams-serviceloader/target/rocketmq-streams-serviceloader-2.0.0-SNAPSHOT-sources.jar
deleted file mode 100644
index 1695e65..0000000
Binary files a/rocketmq-streams-serviceloader/target/rocketmq-streams-serviceloader-2.0.0-SNAPSHOT-sources.jar and /dev/null differ
diff --git a/rocketmq-streams-serviceloader/target/rocketmq-streams-serviceloader-2.0.0-SNAPSHOT.jar b/rocketmq-streams-serviceloader/target/rocketmq-streams-serviceloader-2.0.0-SNAPSHOT.jar
deleted file mode 100644
index 9dc634b..0000000
Binary files a/rocketmq-streams-serviceloader/target/rocketmq-streams-serviceloader-2.0.0-SNAPSHOT.jar and /dev/null differ
diff --git a/rocketmq-streams-serviceloader/target/test-classes/log4j.xml b/rocketmq-streams-serviceloader/target/test-classes/log4j.xml
deleted file mode 100755
index 7812fe7..0000000
--- a/rocketmq-streams-serviceloader/target/test-classes/log4j.xml
+++ /dev/null
@@ -1,20 +0,0 @@
-<?xml version="1.0" encoding="UTF-8" ?>
-<!DOCTYPE log4j:configuration SYSTEM "http://toolkit.alibaba-inc.com/dtd/log4j/log4j.dtd">
-<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
-
- <appender name="Console" class="org.apache.log4j.ConsoleAppender">
- <layout class="org.apache.log4j.PatternLayout">
- <param name="ConversionPattern" value="%d{ISO8601} %l [%t] %-5p - %m%n%n"/>
- </layout>
- <filter class="org.apache.log4j.varia.LevelRangeFilter">
- <param name="LevelMin" value="INFO"/>
- <param name="LevelMax" value="ERROR"/>
- </filter>
- </appender>
-
- <root>
- <priority value="INFO"/>
- <appender-ref ref="Console"/>
- </root>
-
-</log4j:configuration>
\ No newline at end of file
diff --git a/rocketmq-streams-serviceloader/target/test-classes/org/apache/rocketmq/streams/serviceloader/ServiceLoaderComponentTest.class b/rocketmq-streams-serviceloader/target/test-classes/org/apache/rocketmq/streams/serviceloader/ServiceLoaderComponentTest.class
deleted file mode 100644
index 100837f..0000000
Binary files a/rocketmq-streams-serviceloader/target/test-classes/org/apache/rocketmq/streams/serviceloader/ServiceLoaderComponentTest.class and /dev/null differ
diff --git a/rocketmq-streams-window/rocketmq-streams-window.iml b/rocketmq-streams-window/rocketmq-streams-window.iml
deleted file mode 100644
index 793fdfa..0000000
--- a/rocketmq-streams-window/rocketmq-streams-window.iml
+++ /dev/null
@@ -1,82 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<module org.jetbrains.idea.maven.project.MavenProjectsManager.isMavenModule="true" type="JAVA_MODULE" version="4">
- <component name="NewModuleRootManager" LANGUAGE_LEVEL="JDK_1_8">
- <output url="file://$MODULE_DIR$/target/classes" />
- <output-test url="file://$MODULE_DIR$/target/test-classes" />
- <content url="file://$MODULE_DIR$">
- <sourceFolder url="file://$MODULE_DIR$/src/main/java" isTestSource="false" />
- <sourceFolder url="file://$MODULE_DIR$/src/main/resources" type="java-resource" />
- <sourceFolder url="file://$MODULE_DIR$/src/test/java" isTestSource="true" />
- <sourceFolder url="file://$MODULE_DIR$/src/test/resources" type="java-test-resource" />
- <excludeFolder url="file://$MODULE_DIR$/target" />
- </content>
- <orderEntry type="inheritedJdk" />
- <orderEntry type="sourceFolder" forTests="false" />
- <orderEntry type="module" module-name="rocketmq-streams-db-operator" />
- <orderEntry type="module" module-name="rocketmq-streams-configurable" />
- <orderEntry type="module" module-name="rocketmq-streams-serviceloader" />
- <orderEntry type="library" name="Maven: org.springframework:spring-jdbc:3.2.13.RELEASE" level="project" />
- <orderEntry type="library" name="Maven: org.springframework:spring-beans:3.2.13.RELEASE" level="project" />
- <orderEntry type="library" name="Maven: org.springframework:spring-core:3.2.13.RELEASE" level="project" />
- <orderEntry type="library" name="Maven: commons-logging:commons-logging:1.1" level="project" />
- <orderEntry type="library" name="Maven: logkit:logkit:1.0.1" level="project" />
- <orderEntry type="library" name="Maven: avalon-framework:avalon-framework:4.1.3" level="project" />
- <orderEntry type="library" name="Maven: javax.servlet:servlet-api:2.3" level="project" />
- <orderEntry type="library" name="Maven: org.springframework:spring-tx:3.2.13.RELEASE" level="project" />
- <orderEntry type="library" name="Maven: mysql:mysql-connector-java:5.1.40" level="project" />
- <orderEntry type="module" module-name="rocketmq-streams-lease" />
- <orderEntry type="module" module-name="rocketmq-streams-channel-db" />
- <orderEntry type="module" module-name="rocketmq-streams-dim" />
- <orderEntry type="module" module-name="rocketmq-streams-filter" />
- <orderEntry type="module" module-name="rocketmq-streams-script" />
- <orderEntry type="library" name="Maven: io.krakens:java-grok:0.1.9" level="project" />
- <orderEntry type="library" name="Maven: org.codehaus.groovy:groovy-all:2.1.8" level="project" />
- <orderEntry type="library" name="Maven: org.python:jython-standalone:2.7.0" level="project" />
- <orderEntry type="module" module-name="rocketmq-streams-channel-http" />
- <orderEntry type="module" module-name="rocketmq-streams-commons" />
- <orderEntry type="library" name="Maven: org.apache.commons:commons-lang3:3.11" level="project" />
- <orderEntry type="library" name="Maven: junit:junit:4.12" level="project" />
- <orderEntry type="library" name="Maven: org.hamcrest:hamcrest-core:1.3" level="project" />
- <orderEntry type="library" name="Maven: com.alibaba:fastjson:1.2.27" level="project" />
- <orderEntry type="library" name="Maven: log4j:log4j:1.2.17" level="project" />
- <orderEntry type="library" name="Maven: com.google.code.gson:gson:2.8.5" level="project" />
- <orderEntry type="library" name="Maven: com.google.auto.service:auto-service:1.0-rc5" level="project" />
- <orderEntry type="library" name="Maven: com.google.auto.service:auto-service-annotations:1.0-rc5" level="project" />
- <orderEntry type="library" name="Maven: com.google.auto:auto-common:0.10" level="project" />
- <orderEntry type="library" name="Maven: com.google.guava:guava:25.1-jre" level="project" />
- <orderEntry type="library" name="Maven: com.google.code.findbugs:jsr305:3.0.2" level="project" />
- <orderEntry type="library" name="Maven: org.checkerframework:checker-qual:2.0.0" level="project" />
- <orderEntry type="library" name="Maven: com.google.errorprone:error_prone_annotations:2.1.3" level="project" />
- <orderEntry type="library" name="Maven: com.google.j2objc:j2objc-annotations:1.1" level="project" />
- <orderEntry type="library" name="Maven: org.codehaus.mojo:animal-sniffer-annotations:1.14" level="project" />
- <orderEntry type="library" name="Maven: com.lmax:disruptor:3.2.0" level="project" />
- <orderEntry type="library" name="Maven: com.gliwka.hyperscan:hyperscan:5.4.0-2.0.0" level="project" />
- <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:5.4.0-1.0.0" level="project" />
- <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:linux-x86_64:5.4.0-1.0.0" level="project" />
- <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:windows-x86_64:5.4.0-1.0.0" level="project" />
- <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:macosx-x86_64:5.4.0-1.0.0" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp-platform:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-arm:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-arm64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-x86:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-x86_64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:ios-arm64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:ios-x86_64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-armhf:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-arm64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-ppc64le:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-x86:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-x86_64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:macosx-x86_64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:windows-x86:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:windows-x86_64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: net.java.dev.jna:platform:3.5.2" level="project" />
- <orderEntry type="library" name="Maven: net.java.dev.jna:jna:3.5.2" level="project" />
- <orderEntry type="library" name="Maven: commons-io:commons-io:2.5" level="project" />
- <orderEntry type="library" name="Maven: org.apache.httpcomponents:httpclient:4.5.2" level="project" />
- <orderEntry type="library" name="Maven: org.apache.httpcomponents:httpcore:4.4.4" level="project" />
- <orderEntry type="library" name="Maven: commons-codec:commons-codec:1.9" level="project" />
- <orderEntry type="library" name="Maven: org.rocksdb:rocksdbjni:6.6.4" level="project" />
- </component>
-</module>
\ No newline at end of file
diff --git a/rocketmq-streams-window/target/classes/dipper.cs b/rocketmq-streams-window/target/classes/dipper.cs
deleted file mode 100644
index 45919b0..0000000
--- a/rocketmq-streams-window/target/classes/dipper.cs
+++ /dev/null
@@ -1,2892 +0,0 @@
-dipper.private.blink.rules&&&&pipline&&&&apsara_stack_network_accessibility_sca&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.UnionChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"nextStageLables\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"apsara_stack_network_accessibility_sca_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"cancelAfterConfigurableRefreshListerner\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"ownerSqlNodeTableName\\\\\\\":\\\\\\\"ne [...]
-dipper.private.blink.rules&&&&channel&&&&apsara_stack_network_accessibility_sca&&&&{"offset":"0","isJsonData":"true","maxThread":"12","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.MetaqChannel","configureName":"apsara_stack_network_accessibility_sca","type":"channel","version":"1.0","timeout":"30000","activtyTimeOut":"3000","syncTimeout":"5000","groupName":"apsara_stack_network_accessibility_sca","pullIntervalMs":"100","isBatchMessage":"true","isAutoFlush":" [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_apsara_stack_network_accessibility_sca_union_10001&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.UnionChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"piplineName2MsgSourceName\\\\\\\":\\\\\\\"[{\\\\\\\\\\\\\\\"value\\\\\\\\\\\\\\\":\\\\\\\\\\\\\\\"aegis_property_sca\\\\\\\\\\\\\\\",\\\\\\\\\\\\\\\"key\\\\\\\\\\\\\\\":\\\\\\\\\\\\\\\"subpipline_subpipline_apsara_stack_network [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_subpipline_apsara_stack_network_accessibility_sca_union_10001_union_10001&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"entityName\\\\\\\":\\\\\\\"script\\\\\\\",\\\\\\\"scriptName\\\\\\\":\\\\\\\"subpipline_subpipline_ [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_subpipline_apsara_stack_network_accessibility_sca_union_10001_union_10001_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_subpipline_apsara_stack_network_accessibility_sca_union_10001_union_10001_script_10001","type":"script","value":"___!_10001=!((listenIp,regex,'^(127\\.\\d+\\.\\d+\\.\\d+)|(169\\.254\\.\\d+\\.\\d+)$'));\n", [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_subpipline_apsara_stack_network_accessibility_sca_union_10001_union_10001_rule_10001&&&&{"expressionStr":"(port,<>,'')&___!_10001","scriptNames":"[]","expressionName":"10002","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_subpipline_apsara_stack_network_accessibility_sca_union_10001_union_10001_rule_10001","ruleStatus":"0","type":"rule"," [...]
-dipper.private.blink.rules&&&&express&&&&10002&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10001\",\"___!_10001\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10001&&&&{"aesFlag":1,"varName":"port","functionName":"<>","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"wc8fi0p7qwBHnY1x2c6Qsw=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_subpipline_apsara_stack_network_accessibility_sca_union_10001_union_10001_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_subpipline_apsara_stack_network_accessibility_sca_union_10001_union_10001_script_10002","type":"script","value":"retainField(ext,bizType,imageName,processUser,processStarted,bizTypeExt,pid,configPath,snaps [...]
-dipper.private.blink.rules&&&&nameList&&&&30.240.98.174;1616994930825;1&&&&{"indexs":"[\"uuid\",\"instance_id\"]","pollingTime":"60","className":"com.aliyun.filter.namelist.DBNameList","configureName":"30.240.98.174;1616994930825;1","userName":"aegis.sca.esc.userName","type":"nameList","version":"1.0","url":"aegis.sca.esc.url","sql":"select ali_uid,uuid,instance_id,instance_network_type,security_group_id,public_ip_address,allocation_id,eip_ip_address,private_ip_address,vpc_id,instance_st [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_subpipline_apsara_stack_network_accessibility_sca_union_10001_union_10001_script_10003&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_subpipline_apsara_stack_network_accessibility_sca_union_10001_union_10001_script_10003","type":"script","value":"___dim_10001=left_join('dipper.private.blink.rules','30.240.98.174;1616994930825;1','(uuid,= [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_subpipline_apsara_stack_network_accessibility_sca_union_10001_union_10001_rule_10002&&&&{"expressionStr":"ecs_info.instance_status,==,'running'","scriptNames":"[]","expressionName":"10003","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_subpipline_apsara_stack_network_accessibility_sca_union_10001_union_10001_rule_10002","ruleStatus":"0"," [...]
-dipper.private.blink.rules&&&&express&&&&10003&&&&{"aesFlag":1,"varName":"ecs_info.instance_status","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"rfJUsZyCSw5+VvHkYYf4dA=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_subpipline_apsara_stack_network_accessibility_sca_union_10001_union_10001_script_10004&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_subpipline_apsara_stack_network_accessibility_sca_union_10001_union_10001_script_10004","type":"script","value":"vpc_id=ecs_info.vpc_id;\ninstance_id=ecs_info.instance_id;\nprivate_ip_address=ecs_info.priv [...]
-dipper.private.blink.rules&&&&nameList&&&&30.240.98.174;1616994930829;2&&&&{"indexs":"[\"uuid\",\"instance_id\"]","pollingTime":"60","className":"com.aliyun.filter.namelist.DBNameList","configureName":"30.240.98.174;1616994930829;2","userName":"aegis.sca.esc.userName","type":"nameList","version":"1.0","url":"aegis.sca.esc.url","sql":"select ali_uid,uuid,instance_id,instance_network_type,security_group_id,public_ip_address,allocation_id,eip_ip_address,private_ip_address,vpc_id,instance_st [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_subpipline_apsara_stack_network_accessibility_sca_union_10001_union_10001_script_10005&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_subpipline_apsara_stack_network_accessibility_sca_union_10001_union_10001_script_10005","type":"script","value":"___dim_10002=left_join('dipper.private.blink.rules','30.240.98.174;1616994930829;2','(instan [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_subpipline_apsara_stack_network_accessibility_sca_union_10001_union_10001_script_10006&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_subpipline_apsara_stack_network_accessibility_sca_union_10001_union_10001_script_10006","type":"script","value":"___coalesce_10001=coalesce(ecs_info.public_ip_address,'');\n___compare_10001=!equals(___coal [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_subpipline_apsara_stack_network_accessibility_sca_union_10001_union_10001_script_10007&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_subpipline_apsara_stack_network_accessibility_sca_union_10001_union_10001_script_10007","type":"script","value":"retainField(bizType,processUser,processStarted,security_group_id,pid,type,uuid,internet_ip,p [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_subpipline_apsara_stack_network_accessibility_sca_union_10001_union_10002&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"entityName\\\\\\\":\\\\\\\"script\\\\\\\",\\\\\\\"scriptName\\\\\\\":\\\\\\\"subpipline_subpipline_ [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_subpipline_apsara_stack_network_accessibility_sca_union_10001_union_10002_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_subpipline_apsara_stack_network_accessibility_sca_union_10001_union_10002_script_10001","type":"script","value":"___!_10002=!((listenIp,regex,'^(127\\.\\d+\\.\\d+\\.\\d+)|(169\\.254\\.\\d+\\.\\d+)$'));\n", [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_subpipline_apsara_stack_network_accessibility_sca_union_10001_union_10002_rule_10001&&&&{"expressionStr":"(port,<>,'')&___!_10002","scriptNames":"[]","expressionName":"10005","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_subpipline_apsara_stack_network_accessibility_sca_union_10001_union_10002_rule_10001","ruleStatus":"0","type":"rule"," [...]
-dipper.private.blink.rules&&&&express&&&&10004&&&&{"aesFlag":1,"varName":"port","functionName":"<>","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"wc8fi0p7qwBHnY1x2c6Qsw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10005&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10004\",\"___!_10002\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_subpipline_apsara_stack_network_accessibility_sca_union_10001_union_10002_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_subpipline_apsara_stack_network_accessibility_sca_union_10001_union_10002_script_10002","type":"script","value":"retainField(ext,bizType,imageName,processUser,processStarted,bizTypeExt,pid,configPath,snaps [...]
-dipper.private.blink.rules&&&&nameList&&&&30.240.98.174;1616994930830;3&&&&{"indexs":"[\"uuid\",\"instance_id\"]","pollingTime":"60","className":"com.aliyun.filter.namelist.DBNameList","configureName":"30.240.98.174;1616994930830;3","userName":"aegis.sca.esc.userName","type":"nameList","version":"1.0","url":"aegis.sca.esc.url","sql":"select ali_uid,uuid,instance_id,instance_network_type,security_group_id,public_ip_address,allocation_id,eip_ip_address,private_ip_address,vpc_id,instance_st [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_subpipline_apsara_stack_network_accessibility_sca_union_10001_union_10002_script_10003&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_subpipline_apsara_stack_network_accessibility_sca_union_10001_union_10002_script_10003","type":"script","value":"___dim_10003=left_join('dipper.private.blink.rules','30.240.98.174;1616994930830;3','(uuid,= [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_subpipline_apsara_stack_network_accessibility_sca_union_10001_union_10002_rule_10002&&&&{"expressionStr":"ecs_info.instance_status,==,'running'","scriptNames":"[]","expressionName":"10006","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_subpipline_apsara_stack_network_accessibility_sca_union_10001_union_10002_rule_10002","ruleStatus":"0"," [...]
-dipper.private.blink.rules&&&&express&&&&10006&&&&{"aesFlag":1,"varName":"ecs_info.instance_status","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"rfJUsZyCSw5+VvHkYYf4dA=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_subpipline_apsara_stack_network_accessibility_sca_union_10001_union_10002_script_10004&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_subpipline_apsara_stack_network_accessibility_sca_union_10001_union_10002_script_10004","type":"script","value":"vpc_id=ecs_info.vpc_id;\ninstance_id=ecs_info.instance_id;\nprivate_ip_address=ecs_info.priv [...]
-dipper.private.blink.rules&&&&nameList&&&&30.240.98.174;1616994930831;4&&&&{"indexs":"[\"vpc_id;internal_ip\"]","pollingTime":"60","className":"com.aliyun.filter.namelist.DBNameList","configureName":"30.240.98.174;1616994930831;4","userName":"aegis.sca.dnat.userName","type":"nameList","version":"1.0","url":"aegis.sca.dnat.url","sql":"select ali_uid,nat_gateway_id,vpc_id,external_ip,external_port,internal_ip,internal_port,ip_protocol,forward_entry_status from asset_dnat_info where forward [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_subpipline_apsara_stack_network_accessibility_sca_union_10001_union_10002_script_10005&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_subpipline_apsara_stack_network_accessibility_sca_union_10001_union_10002_script_10005","type":"script","value":"constants_10000='any';\nconstants_10001='any';\n___dim_10004=left_join('dipper.private.blink [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_subpipline_apsara_stack_network_accessibility_sca_union_10001_union_10002_rule_10003&&&&{"expressionStr":"dnat_info.forward_entry_status,==,'available'","scriptNames":"[]","expressionName":"10007","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_subpipline_apsara_stack_network_accessibility_sca_union_10001_union_10002_rule_10003","ruleStatu [...]
-dipper.private.blink.rules&&&&express&&&&10007&&&&{"aesFlag":1,"varName":"dnat_info.forward_entry_status","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"3JNgRPWXGIXtB5KaBIDvNA=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_subpipline_apsara_stack_network_accessibility_sca_union_10001_union_10002_script_10006&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_subpipline_apsara_stack_network_accessibility_sca_union_10001_union_10002_script_10006","type":"script","value":"internet_ip=dnat_info.external_ip;\n___compare_10005=equals(dnat_info.external_port,'any');\ [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_subpipline_apsara_stack_network_accessibility_sca_union_10001_union_10002_script_10007&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_subpipline_apsara_stack_network_accessibility_sca_union_10001_union_10002_script_10007","type":"script","value":"retainField(bizType,processUser,processStarted,security_group_id,pid,type,uuid,internet_ip,p [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_apsara_stack_network_accessibility_sca_union_10001_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_apsara_stack_network_accessibility_sca_union_10001_script_10001","type":"script","value":"___splitindex_10001=splitindex(security_group_info.port_range,'/',0);;___cast_10001=cast(___splitindex_10001,'int');;___cast_10002=cast(po [...]
-dipper.private.blink.rules&&&&nameList&&&&30.240.98.174;1616994930835;5&&&&{"indexs":"[\"security_group_id\"]","pollingTime":"60","className":"com.aliyun.filter.namelist.DBNameList","configureName":"30.240.98.174;1616994930835;5","userName":"aegis.sca.security.group.userName","type":"nameList","version":"1.0","url":"aegis.sca.security.group.url","sql":"select ali_uid,security_group_id,dest_cidr_ip,dest_cidr_ip_bit_mask,nic_type,direction,ip_protocol,policy,port_range,priority,source_cidr [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_apsara_stack_network_accessibility_sca_union_10001_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_apsara_stack_network_accessibility_sca_union_10001_script_10002","type":"script","value":"___cast_10007=cast(port,'int');\n___cast_10008=cast(port,'int');\nconstants_10000='all';\nconstants_10001='-1/-1';\n___dim_10005=inner_joi [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_apsara_stack_network_accessibility_sca_union_10001_script_10003&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_apsara_stack_network_accessibility_sca_union_10001_script_10003","type":"script","value":"___cast_10005=cast(security_group_info.source_cidr_ip_bit_mask,'int');\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&subpipline_apsara_stack_network_accessibility_sca_union_10001_rule_10001&&&&{"expressionStr":"((security_group_info.direction,==,'ingress')&(security_group_info.policy,==,'accept'))&(___cast_10005,<,double,8)","scriptNames":"[]","expressionName":"10012","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_apsara_stack_network_accessibility_sca_union_10001 [...]
-dipper.private.blink.rules&&&&express&&&&10008&&&&{"aesFlag":1,"varName":"security_group_info.direction","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"uDVUiOQMghCLKJyqOEQ27A=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10009&&&&{"aesFlag":1,"varName":"security_group_info.policy","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"JvOH9gogkRcXYYaTXLrXLQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10011&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10008\",\"10009\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10010&&&&{"varName":"___cast_10005","functionName":"<","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"8.0"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10012&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10011\",\"10010\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_apsara_stack_network_accessibility_sca_union_10001_script_10004&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_apsara_stack_network_accessibility_sca_union_10001_script_10004","type":"script","value":"security_group_id=security_group_info.security_group_id;\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_apsara_stack_network_accessibility_sca_union_10001_script_10005&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_apsara_stack_network_accessibility_sca_union_10001_script_10005","type":"script","value":"retainField(bizType,processUser,processStarted,security_group_id,pid,type,uuid,internet_ip,path,protocol,cmdline,containerName,proof,image [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_apsara_stack_network_accessibility_sca_union_10001_script_10006&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_apsara_stack_network_accessibility_sca_union_10001_script_10006","type":"script","value":"retainField(bizType,processUser,processStarted,security_group_id,pid,type,uuid,internet_ip,path,protocol,cmdline,containerName,proof,image [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_apsara_stack_network_accessibility_sca_union_10002&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"entityName\\\\\\\":\\\\\\\"script\\\\\\\",\\\\\\\"scriptName\\\\\\\":\\\\\\\"subpipline_apsara_stack_network_accessibility [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_apsara_stack_network_accessibility_sca_union_10002_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_apsara_stack_network_accessibility_sca_union_10002_script_10001","type":"script","value":"___!_10003=!((listenIp,regex,'^(127\\.\\d+\\.\\d+\\.\\d+)|(169\\.254\\.\\d+\\.\\d+)$'));\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&subpipline_apsara_stack_network_accessibility_sca_union_10002_rule_10001&&&&{"expressionStr":"(port,<>,'')&___!_10003","scriptNames":"[]","expressionName":"10014","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_apsara_stack_network_accessibility_sca_union_10002_rule_10001","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames [...]
-dipper.private.blink.rules&&&&express&&&&10013&&&&{"aesFlag":1,"varName":"port","functionName":"<>","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"wc8fi0p7qwBHnY1x2c6Qsw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10014&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10013\",\"___!_10003\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_apsara_stack_network_accessibility_sca_union_10002_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_apsara_stack_network_accessibility_sca_union_10002_script_10002","type":"script","value":"retainField(ext,bizType,imageName,processUser,processStarted,bizTypeExt,pid,configPath,snapshotIdx,type,uuid,version,ppid,listenStatus,web [...]
-dipper.private.blink.rules&&&&nameList&&&&30.240.98.174;1616994930865;6&&&&{"indexs":"[\"uuid\",\"instance_id\"]","pollingTime":"60","className":"com.aliyun.filter.namelist.DBNameList","configureName":"30.240.98.174;1616994930865;6","userName":"aegis.sca.esc.userName","type":"nameList","version":"1.0","url":"aegis.sca.esc.url","sql":"select ali_uid,uuid,instance_id,instance_network_type,security_group_id,public_ip_address,allocation_id,eip_ip_address,private_ip_address,vpc_id,instance_st [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_apsara_stack_network_accessibility_sca_union_10002_script_10003&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_apsara_stack_network_accessibility_sca_union_10002_script_10003","type":"script","value":"___dim_10006=left_join('dipper.private.blink.rules','30.240.98.174;1616994930865;6','(uuid,==,uuid)',ecs_info,'',ali_uid,uuid,instance_id, [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_apsara_stack_network_accessibility_sca_union_10002_rule_10002&&&&{"expressionStr":"ecs_info.instance_status,==,'running'","scriptNames":"[]","expressionName":"10015","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_apsara_stack_network_accessibility_sca_union_10002_rule_10002","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[] [...]
-dipper.private.blink.rules&&&&express&&&&10015&&&&{"aesFlag":1,"varName":"ecs_info.instance_status","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"rfJUsZyCSw5+VvHkYYf4dA=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_apsara_stack_network_accessibility_sca_union_10002_script_10004&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_apsara_stack_network_accessibility_sca_union_10002_script_10004","type":"script","value":"vpc_id=ecs_info.vpc_id;\ninstance_id=ecs_info.instance_id;\nprivate_ip_address=ecs_info.private_ip_address;\nsecurity_group_id=ecs_info.se [...]
-dipper.private.blink.rules&&&&nameList&&&&30.240.98.174;1616994930867;7&&&&{"indexs":"[\"server_id;backend_server_port;listener_protocol\"]","pollingTime":"60","className":"com.aliyun.filter.namelist.DBNameList","configureName":"30.240.98.174;1616994930867;7","userName":"aegis.sca.slb.userName","type":"nameList","version":"1.0","url":"aegis.sca.slb.url","sql":"select ali_uid,load_balancer_id,address,address_type,listener_port,listener_protocol,listener_status,backend_server_port,server_i [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_apsara_stack_network_accessibility_sca_union_10002_script_10005&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_apsara_stack_network_accessibility_sca_union_10002_script_10005","type":"script","value":"___dim_10007=left_join('dipper.private.blink.rules','30.240.98.174;1616994930867;7','((instance_id,==,server_id)&(port,==,backend_server_p [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_apsara_stack_network_accessibility_sca_union_10002_rule_10003&&&&{"expressionStr":"(slb_info.address_type,==,'internet')&(slb_info.listener_status,==,'running')","scriptNames":"[]","expressionName":"10018","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_apsara_stack_network_accessibility_sca_union_10002_rule_10003","ruleStatus":"0","type": [...]
-dipper.private.blink.rules&&&&express&&&&10018&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10016\",\"10017\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10017&&&&{"aesFlag":1,"varName":"slb_info.listener_status","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"rfJUsZyCSw5+VvHkYYf4dA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10016&&&&{"aesFlag":1,"varName":"slb_info.address_type","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"dbB21QlapXHUB4tgeOH8Ww=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_apsara_stack_network_accessibility_sca_union_10002_script_10006&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_apsara_stack_network_accessibility_sca_union_10002_script_10006","type":"script","value":"internet_ip=slb_info.address;\nport=slb_info.listener_port;\nprotocol=listenProtocol;\nconnect_type='slb';\nnetwork_type=instance_network_ [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_apsara_stack_network_accessibility_sca_union_10002_script_10007&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_apsara_stack_network_accessibility_sca_union_10002_script_10007","type":"script","value":"retainField(bizType,processUser,processStarted,security_group_id,pid,type,uuid,internet_ip,path,protocol,cmdline,containerName,proof,image [...]
-dipper.private.blink.rules&&&&rule&&&&apsara_stack_network_accessibility_sca_rule_10001&&&&{"expressionStr":"internet_ip,<>,''","scriptNames":"[]","expressionName":"10019","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"apsara_stack_network_accessibility_sca_rule_10001","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[]"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10019&&&&{"aesFlag":1,"varName":"internet_ip","functionName":"<>","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"wc8fi0p7qwBHnY1x2c6Qsw=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&apsara_stack_network_accessibility_sca_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"apsara_stack_network_accessibility_sca_script_10001","type":"script","value":"retainField(bizType,processUser,processStarted,security_group_id,pid,type,uuid,internet_ip,path,protocol,cmdline,containerName,proof,imageName,listen_ports,forwarding_instance_id,bizTy [...]
-dipper.private.blink.rules&&&&channel&&&&apsara_stack_network_accessibility_sca_channel_10001&&&&{"offset":"0","isJsonData":"true","maxThread":"12","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.MetaqChannel","configureName":"apsara_stack_network_accessibility_sca_channel_10001","type":"channel","version":"1.0","timeout":"30000","activtyTimeOut":"3000","tags":"T_MSG_SCA_EXPOSURE","syncTimeout":"5000","pullIntervalMs":"100","isBatchMessage":"true","isAutoFlush [...]
-dipper.private.blink.rules&&&&pipline&&&&blink.direct.source.aegis_asset&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.SubPiplineChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"nextStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"filterMsgSwitch\\\\\\\":\\\\\\\"blink.direct.source.aegis_asset.filter.switch\\\\\\\",\\\\\\\"cancelAfterConfigurableRefreshListerner\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"className\\\\\\\":\\\\\\\"com [...]
-dipper.private.blink.rules&&&&channel&&&&blink.direct.source.aegis_asset;channel&&&&{"offset":"0","isJsonData":"true","maxThread":"12","concurrentCount":"12","className":"com.aliyun.yundun.dipper.channel.adapter.MetaqChannel","configureName":"blink.direct.source.aegis_asset;channel","type":"channel","version":"1.0","timeout":"30000","activtyTimeOut":"3000","tags":"blink.direct.source.aegis_asset.tags","syncTimeout":"5000","groupName":"blink.direct.source.aegis_asset.group","isBatchMessag [...]
-dipper.private.blink.rules&&&&pipline&&&&blink_source_aegis_client_aegis_client&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"nextStageLables\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"blink_source_aegis_client_aegis_client_script_10002\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"cancelAfterConfigurableRefreshListerner\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"ownerSqlNodeTableName\\\\\\\":\\\\\\\ [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client;json_concat_10001&&&&{"fullClassName":"com.aliyun.sec.lyra.udf.ext.JsonConcat","initMethodName":"open","functionName":"json_concat","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client;json_concat_10001","type":"script","version":"1.0","closeMethodName":"close", [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client;explode_json_10001&&&&{"setCollectorMethodName":"setCollector","fullClassName":"com.aliyun.sec.sas.explode_json","initMethodName":"open","functionName":"explode_json","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udtf.BlinkUDTFScript","configureName":"blink_source_aegis_client_aegis_client;explode_json_10001","type":"script","version":"1.0","isURL":"false","nameSpace":"dipper.private.b [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client;action_json_10001&&&&{"fullClassName":"com.aliyun.sec.sas.action_json","initMethodName":"open","functionName":"action_json","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client;action_json_10001","type":"script","version":"1.0","closeMethodName":"close","extendF [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client;hex2clean_10001&&&&{"fullClassName":"com.aliyun.sec.sas.hex2clean","initMethodName":"open","functionName":"hex2clean","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client;hex2clean_10001","type":"script","version":"1.0","closeMethodName":"close","extendField":"[ [...]
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_client_aegis_client&&&&{"offset":"0","isJsonData":"true","maxThread":"12","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.MetaqChannel","configureName":"blink_source_aegis_client_aegis_client","type":"channel","version":"1.0","timeout":"30000","activtyTimeOut":"3000","tags":"T_MSG_VIRUS_EVENT_INFO","syncTimeout":"5000","groupName":"blink_source_aegis_client_aegis_client","pullIntervalMs":"100","isBatc [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10001","type":"script","value":"___json_get_10001=json_get(data,'$.check_msg');\ncheck_msg=___json_get_10001;rm('___json_get_10001');\n___json_get_10002=json_get(meta_conf,'$.aliUid');\naliuid=___json_get_10002;rm('_ [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10002","type":"script","value":"retainField(traceid,aliuid,check_msg,uuid,logtime);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10003&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10003","type":"script","value":"___EXPLODE_JSON_10001=EXPLODE_JSON(check_msg);T.check_msg_one=udtf.0;","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10004&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10004","type":"script","value":"___json_get_10003=json_get(T.check_msg_one,'$.module_id');\nmodule_id=___json_get_10003;rm('___json_get_10003');\n___json_get_10004=json_get(T.check_msg_one,'$.execinfo');\nexec_info=_ [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10005&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10005","type":"script","value":"check_msg_one=T.check_msg_one;\nretainField(traceid,check_msg_one,exec_info,module_id,aliuid,check_msg,uuid,logtime);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10006&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10006","type":"script","value":"___EXPLODE_JSON_10002=EXPLODE_JSON(exec_info);T.exec_info_one=udtf.0;","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10007&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10007","type":"script","value":"___json_get_10005=json_get(T.exec_info_one,'$.data');\nexec_info_datalist=___json_get_10005;rm('___json_get_10005');\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10008&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10008","type":"script","value":"exec_info_one=T.exec_info_one;\nretainField(traceid,exec_info,module_id,exec_info_datalist,aliuid,check_msg,uuid,logtime,exec_info_one);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10009&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10009","type":"script","value":"___EXPLODE_JSON_10003=EXPLODE_JSON(exec_info_datalist);T.exec_info_data=udtf.0;","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10010&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10010","type":"script","value":"exec_info_data=T.exec_info_data;\n___json_get_10006=json_get(T.exec_info_data,'$.type');\nexec_info_type=___json_get_10006;rm('___json_get_10006');\n","version":"1.0","extendField":"[] [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10011&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10011","type":"script","value":"retainField(traceid,exec_info_type,exec_info,module_id,exec_info_datalist,exec_info_data,aliuid,check_msg,uuid,logtime,exec_info_one);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10012&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10012","type":"script","value":"___!null_10001=!null(exec_info_data);\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_client_aegis_client_rule_10001&&&&{"expressionStr":"___!null_10001&(exec_info_data,<>,'')","scriptNames":"[]","expressionName":"10021","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_rule_10001","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[]"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10020&&&&{"aesFlag":1,"varName":"exec_info_data","functionName":"<>","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"wc8fi0p7qwBHnY1x2c6Qsw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10021&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"___!null_10001\",\"10020\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10013&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10013","type":"script","value":"retainField(traceid,module_id,exec_info_data,aliuid,logtime,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10014&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10014","type":"script","value":"___json_get_10007=json_get(exec_info_data,'$.cmdline');\ndata_cmdline=___json_get_10007;rm('___json_get_10007');\n___json_get_10008=json_get(exec_info_data,'$.name');\ndata_name=___jso [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10015&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10015","type":"script","value":"retainField(traceid,module_id,data_name,data_cmdline,exec_info_data,data_type,aliuid,data_files,logtime,uuid,data_raw_info,data_from);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10016&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10016","type":"script","value":"___null_10001=null(data_raw_info);\n___json_get_10013=json_get(data_raw_info,'$.filepath');\nif(___null_10001){___case_10004='';}else{___case_10004=___json_get_10013;};\nraw_info_filep [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10017&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10017","type":"script","value":"retainField(traceid,raw_info_filepath,raw_info_procname,raw_info_md5,data_files,uuid,data_raw_info,data_from,raw_info_line,raw_info_create_time,module_id,raw_info_dynamic,data_name,raw [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10018&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10018","type":"script","value":"___compare_10006=equals(data_type,'CrontabItem');\n___compare_10007=equals(data_type,'ld_so_preload');\n___compare_10008=equals(data_type,'LinuxPubKey');\nif(___compare_10006){___case_ [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10019&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10019","type":"script","value":"retainField(traceid,raw_info_filepath,the_name,raw_info_procname,raw_info_md5,data_files,uuid,data_raw_info,data_from,raw_info_line,raw_info_create_time,module_id,raw_info_dynamic,data [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10020&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10020","type":"script","value":"___lower_cmd_10001=lower(cmd);\nlower_cmd=___lower_cmd_10001;rm('___lower_cmd_10001');\n___lower_cmd_10002=lower(cmd);\n___REGEXP_REPLACE_10001=REGEXP_REPLACE(___lower_cmd_10002,'[~~~~ [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10021&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10021","type":"script","value":"retainField(traceid,the_name,raw_info_procname,raw_info_md5,clean_cmd,uuid,data_raw_info,data_from,raw_info_line,raw_info_create_time,data_name,raw_info_procpath,data_cmdline,aliuid,ra [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10022&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10022","type":"script","value":"retainField(traceid,the_name,raw_info_procname,raw_info_md5,clean_cmd,uuid,data_raw_info,data_from,raw_info_line,raw_info_create_time,data_name,raw_info_procpath,data_cmdline,aliuid,ra [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10023&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10023","type":"script","value":"___json_get_10023=json_get(exec_info_data,'$.name');\ndata_name=___json_get_10023;rm('___json_get_10023');\n___json_get_10024=json_get(exec_info_data,'$.type');\ndata_type=___json_get_ [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10024&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10024","type":"script","value":"retainField(traceid,module_id,data_name,exec_info_data,data_type,raw_info,aliuid,cmd,logtime,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_client_aegis_client_rule_10002&&&&{"expressionStr":"(data_type,==,'ProcessGeneral')|(data_type,==,'LinuxHiddenProcess')","scriptNames":"[]","expressionName":"10024","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_rule_10002","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames": [...]
-dipper.private.blink.rules&&&&express&&&&10022&&&&{"aesFlag":1,"varName":"data_type","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"XiZqpAF5OpdGedX7s8cLFg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10024&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10022\",\"10023\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10023&&&&{"aesFlag":1,"varName":"data_type","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"PCWwrcOjmeBbuLYPHG0JS3tTNU/U90AuQne3hzRtQio="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10025&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10025","type":"script","value":"___json_get_10027=json_get(raw_info,'$.procpath');\nprocpath=___json_get_10027;rm('___json_get_10027');\n___json_get_10028=json_get(raw_info,'$.cwd');\ncwd=___json_get_10028;rm('___jso [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10026&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10026","type":"script","value":"retainField(traceid,procpath,create_time,pid,uuid,cwd,module_id,data_name,process_status,proc_exe_raw,procfs_fd,exec_info_data,data_type,raw_info,aliuid,cmd,procname,logtime,md5,userna [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10027&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10027","type":"script","value":"___lower_cmd_10003=lower(cmd);\n___REGEXP_REPLACE_10002=REGEXP_REPLACE(___lower_cmd_10003,'[,#~~~~~\\^\\\\`]','');\nclean_cmd_win=___REGEXP_REPLACE_10002;rm('___REGEXP_REPLACE_10002'); [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10028&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10028","type":"script","value":"retainField(traceid,procpath,create_time,pid,uuid,cwd,module_id,data_name,process_status,proc_exe_raw,procfs_fd,exec_info_data,data_type,raw_info,aliuid,cmd,procname,clean_cmd_win,logt [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10029&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10029","type":"script","value":"___json_get_10040=json_get(exec_info_data,'$.name');\ndata_name=___json_get_10040;rm('___json_get_10040');\n___json_get_10041=json_get(exec_info_data,'$.type');\ndata_type=___json_get_ [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10030&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10030","type":"script","value":"retainField(traceid,msg,module_id,data_name,exec_info_data,data_type,aliuid,cmd,logtime,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10031&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10031","type":"script","value":"___!null_10005=!null(msg);\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_client_aegis_client_rule_10003&&&&{"expressionStr":"((data_type,==,'file_info')&___!null_10005)&(msg,<>,'')","scriptNames":"[]","expressionName":"10028","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_rule_10003","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[]"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10026&&&&{"aesFlag":1,"varName":"msg","functionName":"<>","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":""}&&&&null
-dipper.private.blink.rules&&&&express&&&&10025&&&&{"aesFlag":1,"varName":"data_type","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"RroTetz+RBliM0NFZ/cGug=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10028&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10027\",\"10026\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10027&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10025\",\"___!null_10005\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10032&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10032","type":"script","value":"___json_get_10044=json_get(msg,'$.data.bin_path');\nbin_path=___json_get_10044;rm('___json_get_10044');\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10033&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10033","type":"script","value":"retainField(traceid,msg,module_id,data_name,exec_info_data,data_type,bin_path,aliuid,cmd,logtime,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10034&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10034","type":"script","value":"___json_get_10045=json_get(exec_info_data,'$.name');\ndata_name=___json_get_10045;rm('___json_get_10045');\n___json_get_10046=json_get(exec_info_data,'$.type');\ndata_type=___json_get_ [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10035&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10035","type":"script","value":"retainField(traceid,module_id,data_name,exec_info_data,data_type,raw_info,aliuid,cmd,logtime,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10036&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10036","type":"script","value":"___!null_10006=!null(raw_info);\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_client_aegis_client_rule_10004&&&&{"expressionStr":"((data_type,==,'LinuxHiddenProcess')&___!null_10006)&(raw_info,<>,'')","scriptNames":"[]","expressionName":"10032","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_rule_10004","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames [...]
-dipper.private.blink.rules&&&&express&&&&10029&&&&{"aesFlag":1,"varName":"data_type","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"PCWwrcOjmeBbuLYPHG0JS3tTNU/U90AuQne3hzRtQio="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10031&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10029\",\"___!null_10006\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10030&&&&{"aesFlag":1,"varName":"raw_info","functionName":"<>","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":""}&&&&null
-dipper.private.blink.rules&&&&express&&&&10032&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10031\",\"10030\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10037&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10037","type":"script","value":"___json_get_10049=json_get(raw_info,'$.procpath');\nprocpath=___json_get_10049;rm('___json_get_10049');\n___json_get_10050=json_get(raw_info,'$.cwd');\ncwd=___json_get_10050;rm('___jso [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10038&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10038","type":"script","value":"retainField(traceid,procpath,create_time,pid,uuid,ppid,cwd,module_id,data_name,process_status,exec_info_data,data_type,raw_info,aliuid,cmd,procname,logtime,md5,username);","version":"1 [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10039&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10039","type":"script","value":"___cast_10114=cast(pid,'long');\n___cast_10115=cast(ppid,'long');\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_client_aegis_client_rule_10005&&&&{"expressionStr":"((((process_status,<>,'zombie')&(___cast_10114,<>,double,2))&(___cast_10115,<>,double,2))&(procpath,like,'/%'))&(procpath,!like,'/usr/local/aegis/%')","scriptNames":"[]","expressionName":"10041","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_rule_10005 [...]
-dipper.private.blink.rules&&&&express&&&&10040&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10039\",\"10036\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10041&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10040\",\"10037\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10033&&&&{"aesFlag":1,"varName":"process_status","functionName":"<>","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"++CE4AvZpMMICfexMf0dbA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10035&&&&{"varName":"___cast_10115","functionName":"<>","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"2.0"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10034&&&&{"varName":"___cast_10114","functionName":"<>","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"2.0"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10037&&&&{"aesFlag":1,"varName":"procpath","functionName":"!like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"AntpwjEjRwNnz5a5rRm8YqFVrYIUducfiHSWcu5oxuY="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10036&&&&{"aesFlag":1,"varName":"procpath","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"M1psS2sL+qyjR6s0zjGxqw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10039&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10038\",\"10035\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10038&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10033\",\"10034\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10040&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10040","type":"script","value":"___ACTION_JSON_10019=ACTION_JSON('3','name','kill_process','pid',pid,'procname',procname);\naction_json_str=___ACTION_JSON_10019;rm('___ACTION_JSON_10019');\n","version":"1.0","extendF [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10041&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10041","type":"script","value":"retainField(traceid,procpath,create_time,pid,action_json_str,uuid,ppid,cwd,module_id,data_name,process_status,exec_info_data,data_type,raw_info,aliuid,cmd,procname,logtime,md5,username [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10042&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10042","type":"script","value":"___json_get_10058=json_get(exec_info_data,'$.name');\ndata_name=___json_get_10058;rm('___json_get_10058');\n___json_get_10059=json_get(exec_info_data,'$.type');\ndata_type=___json_get_ [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10043&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10043","type":"script","value":"retainField(traceid,module_id,data_name,exec_info_data,data_type,raw_info,aliuid,cmd,logtime,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10044&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10044","type":"script","value":"___json_get_10062=json_get(raw_info,'$.buffer');\nraw_buffer=___json_get_10062;rm('___json_get_10062');\n___json_get_10063=json_get(raw_info,'$._');\ntag=___json_get_10063;rm('___json_ [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10045&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10045","type":"script","value":"retainField(traceid,uuid,raw_buffer,check_rules,module_id,data_name,exec_info_data,data_type,raw_info,aliuid,cmd,tag,logtime);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_client_aegis_client_rule_10006&&&&{"expressionStr":"(data_type,==,'linux_hidden_modules_check')&((check_rules,==,'diff_kcore_proc')&(raw_buffer,like,'0000000000000000000100000000adde000200000000adde%'))","scriptNames":"[]","expressionName":"10046","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_rule_1000 [...]
-dipper.private.blink.rules&&&&express&&&&10042&&&&{"aesFlag":1,"varName":"data_type","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"gIUAh6Pk1RIMC88K2POd61aFv5iTdm91J843qDMPKi0="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10044&&&&{"aesFlag":1,"varName":"raw_buffer","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"swpZu3EgJWtT7PnH+5uMmoXU7Ioqx8hDPW0CxnjWPPht2Ln1RwDx1bH3KsV+nSRHZ7UyZvu+rhILQvVCa2LXgA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10043&&&&{"aesFlag":1,"varName":"check_rules","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"u1CqNGOuadoGXwfrY4IxRQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10046&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10042\",\"10045\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10045&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10043\",\"10044\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10046&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10046","type":"script","value":"retainField(traceid,uuid,raw_buffer,check_rules,module_id,data_name,exec_info_data,data_type,raw_info,aliuid,cmd,tag,logtime);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10047&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10047","type":"script","value":"___json_get_10065=json_get(exec_info_data,'$.name');\ndata_name=___json_get_10065;rm('___json_get_10065');\n___json_get_10066=json_get(exec_info_data,'$.type');\ndata_type=___json_get_ [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10048&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10048","type":"script","value":"retainField(traceid,module_id,data_name,exec_info_data,data_type,raw_info,aliuid,cmd,logtime,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_client_aegis_client_rule_10007&&&&{"expressionStr":"data_type,==,'ActiveScriptEventConsumer'","scriptNames":"[]","expressionName":"10047","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_rule_10007","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[]"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10047&&&&{"aesFlag":1,"varName":"data_type","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"cwoTHTD/TNX6siEEbMw3A56xFF6iqvT5S6ES7bZSrPM="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10049&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10049","type":"script","value":"___json_get_10069=json_get(raw_info,'$.ScriptText');\nScriptText=___json_get_10069;rm('___json_get_10069');\n___json_get_10070=json_get(raw_info,'$.ScriptingEngine');\nScriptingEngine= [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10050&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10050","type":"script","value":"retainField(traceid,module_id,data_name,exec_info_data,data_type,raw_info,aliuid,cmd,ScriptText,ScriptingEngine,logtime,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10051&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10051","type":"script","value":"___len_10001=len(ScriptText);\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_client_aegis_client_rule_10008&&&&{"expressionStr":"___len_10001,>,double,20","scriptNames":"[]","expressionName":"10048","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_rule_10008","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[]"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10048&&&&{"varName":"___len_10001","functionName":">","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"20.0"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10052&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10052","type":"script","value":"___ACTION_JSON_10020=ACTION_JSON('2','name','exec_command','cmd','%SystemRoot%/System32/wbem/wmic.exe /NAMESPACE:------\\\\root\\subscription------ PATH ActiveScriptEventConsumer DELET [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10053&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10053","type":"script","value":"retainField(traceid,action_json_str,uuid,module_id,data_name,exec_info_data,data_type,raw_info,aliuid,cmd,ScriptText,ScriptingEngine,logtime);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10054&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10054","type":"script","value":"___json_get_10071=json_get(exec_info_data,'$.name');\ndata_name=___json_get_10071;rm('___json_get_10071');\n___json_get_10072=json_get(exec_info_data,'$.type');\ndata_type=___json_get_ [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10055&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10055","type":"script","value":"retainField(traceid,module_id,data_name,exec_info_data,data_type,raw_info,aliuid,cmd,logtime,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_client_aegis_client_rule_10009&&&&{"expressionStr":"data_type,in,'\\'HiddenPreloadDynamicLibrary\\',\\'HiddenPreloadFile\\''","scriptNames":"[]","expressionName":"10049","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_rule_10009","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNa [...]
-dipper.private.blink.rules&&&&express&&&&10049&&&&{"aesFlag":1,"varName":"data_type","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"LVgTkdA+DPbT+7GBLM6kDQIJMpzRafxq7vLqDdBJfJSG/fNxBKde7P5Y6T+6S/G0g2fPsJKMkGJZoB7p50itCw=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10056&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10056","type":"script","value":"___json_get_10075=json_get(raw_info,'$.filepath');\nld_filepath=___json_get_10075;rm('___json_get_10075');\n___json_get_10076=json_get(raw_info,'$.subfilepath');\nsubfilepath=___json_g [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10057&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10057","type":"script","value":"retainField(traceid,uuid,ld_filepath,ld_content,module_id,data_name,subfilepath,exec_info_data,data_type,raw_info,aliuid,cmd,logtime);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10058&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10058","type":"script","value":"___ACTION_JSON_10021=ACTION_JSON('2','name','truncate_and_lock_file','filepath','/etc/ld.so.preload','2','name','exec_command','cmd','echo > /etc/ld.so.preload; rm -f /etc/ld.so.preloa [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10059&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10059","type":"script","value":"retainField(traceid,action_json_str,uuid,ld_filepath,ld_content,module_id,data_name,subfilepath,exec_info_data,data_type,raw_info,aliuid,cmd,logtime);","version":"1.0","extendField":"[ [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10060&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10060","type":"script","value":"ali_uid='';\nclient_ip='';\nevent_type='持久化后门';\nevent_name='异常计划任务(体检)';\nlevel='high';\n___cast_10140=cast(uuid,'string');\n___cast_10141=cast(logtime,'string');\n___!null_10007=!nul [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10061&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10061","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_client_aegis_client_channel_10001&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink_source_aegis_client_aegis_client_channel_10001","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeInOrder":"true","maxFetchLogGroupSize":"100","syn [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10062&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10062","type":"script","value":"___json_get_10078=json_get(exec_info_data,'$.name');\ndata_name=___json_get_10078;rm('___json_get_10078');\n___json_get_10079=json_get(exec_info_data,'$.type');\ndata_type=___json_get_ [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10063&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10063","type":"script","value":"retainField(traceid,module_id,data_name,exec_info_data,data_type,raw_info,aliuid,cmd,logtime,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_client_aegis_client_rule_10010&&&&{"expressionStr":"data_type,==,'CrontabItem'","scriptNames":"[]","expressionName":"10050","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_rule_10010","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[]"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10050&&&&{"aesFlag":1,"varName":"data_type","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"xHHZWDjQ0GhmZS+s1WEivg=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10064&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10064","type":"script","value":"___json_get_10082=json_get(raw_info,'$.head_content');\nhead_content=___json_get_10082;rm('___json_get_10082');\n___json_get_10083=json_get(raw_info,'$.filepath');\nraw_filepath=___jso [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10065&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10065","type":"script","value":"retainField(traceid,head_content,module_id,data_name,exec_info_data,data_type,raw_filepath,raw_info,aliuid,cmd,logtime,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10066&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10066","type":"script","value":"___HEX2CLEAN_10001=HEX2CLEAN(head_content);\nclean_text=___HEX2CLEAN_10001;rm('___HEX2CLEAN_10001');\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10067&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10067","type":"script","value":"retainField(traceid,head_content,uuid,module_id,data_name,exec_info_data,data_type,raw_filepath,raw_info,aliuid,cmd,logtime,clean_text);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_client_aegis_client_rule_10011&&&&{"expressionStr":"head_content,like,'5245444953%'","scriptNames":"[]","expressionName":"10051","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_rule_10011","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[]"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10051&&&&{"aesFlag":1,"varName":"head_content","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"zAN6I/RDzTx3dgDFvFgeFw=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10068&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10068","type":"script","value":"___ACTION_JSON_10022=ACTION_JSON('2','name','truncate_and_lock_file','filepath',raw_filepath);\naction_json_str=___ACTION_JSON_10022;rm('___ACTION_JSON_10022');\n","version":"1.0","ext [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10069&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10069","type":"script","value":"retainField(traceid,head_content,action_json_str,uuid,module_id,data_name,exec_info_data,data_type,raw_filepath,raw_info,aliuid,cmd,logtime,clean_text);","version":"1.0","extendField": [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10070&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10070","type":"script","value":"ali_uid='';\nclient_ip='';\nevent_type='持久化后门';\nevent_name='异常计划任务(体检)';\nlevel='high';\n___cast_10142=cast(uuid,'string');\n___cast_10143=cast(logtime,'string');\n___JSON_CONCAT_1001 [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10071&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10071","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_client_aegis_client_channel_10002&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink_source_aegis_client_aegis_client_channel_10002","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeInOrder":"true","maxFetchLogGroupSize":"100","syn [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10072&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10072","type":"script","value":"___json_get_10084=json_get(exec_info_data,'$.name');\ndata_name=___json_get_10084;rm('___json_get_10084');\n___json_get_10085=json_get(exec_info_data,'$.type');\ndata_type=___json_get_ [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10073&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10073","type":"script","value":"retainField(traceid,module_id,data_name,exec_info_data,data_type,raw_info,aliuid,logtime,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_client_aegis_client_rule_10012&&&&{"expressionStr":"data_type,==,'Services'","scriptNames":"[]","expressionName":"10052","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_rule_10012","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[]"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10052&&&&{"aesFlag":1,"varName":"data_type","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"yaK344taTb0rT/mrxZPPJQ=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10074&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10074","type":"script","value":"___json_get_10087=json_get(raw_info,'$.ImagePath');\ncmdline=___json_get_10087;rm('___json_get_10087');\n___json_get_10088=json_get(raw_info,'$.ServiceName');\nServiceName=___json_get_ [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10075&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10075","type":"script","value":"retainField(traceid,module_id,cmdline,data_name,ServiceName,exec_info_data,data_type,raw_info,aliuid,ServiceDll,logtime,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10076&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10076","type":"script","value":"___lower_cmdline_10001=lower(cmdline);\n___REGEXP_REPLACE_10003=REGEXP_REPLACE(___lower_cmdline_10001,'[\\^,#~~~~~\\\\]','');\nclean_cmd=___REGEXP_REPLACE_10003;rm('___REGEXP_REPLACE_1 [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10077&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10077","type":"script","value":"retainField(traceid,clean_cmd,ServiceDll,uuid,module_id,cmdline,data_name,ServiceName,exec_info_data,data_type,raw_info,aliuid,logtime);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_client_aegis_client_rule_10013&&&&{"expressionStr":"((clean_cmd,regex,'powershell(\\.exe)?\\s.*?(downloadstring|downloaddata)')|(clean_cmd,regex,'powershell(\\.exe)?\\s.*?\\-(e|en|ec|enc|enco|encod|encode|encoded\\w*)\\s[\\w/\\+]{10,}'))|(clean_cmd,regex,'\\b(mshta)(\\.exe)?\\s+https?://')","scriptNames":"[]","expressionName":"10057","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dippe [...]
-dipper.private.blink.rules&&&&express&&&&10053&&&&{"aesFlag":1,"varName":"clean_cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"z8yLDlsTggzmzcjTt76NpSnu882Zxn0sLxoESbjcKerhyynLIFAn+A2djHJ+Ac/Q/uNxuRa0g2ycveHD2xU13w=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10055&&&&{"aesFlag":1,"varName":"clean_cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"BIaPtBCWgcGbMYsrf33TW4lv3W84RA2B0XbvLcYOCac="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10054&&&&{"aesFlag":1,"varName":"clean_cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"z8yLDlsTggzmzcjTt76Npb0xH68Um53ifSHfmghWGcH7c28JW+KNRcYrfH4caAPafO5ak21ko2ELBABNxEAtz26dwcIOrZADeJEmQGHmeF1y+TkFBjVhO+xaGQ/KSasp"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10057&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10056\",\"10055\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10056&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10053\",\"10054\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10078&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10078","type":"script","value":"___ACTION_JSON_10023=ACTION_JSON('2','name','disable_services','servicename',ServiceName);\naction_json_str=___ACTION_JSON_10023;rm('___ACTION_JSON_10023');\n","version":"1.0","extendF [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10079&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10079","type":"script","value":"retainField(traceid,clean_cmd,ServiceDll,action_json_str,uuid,module_id,cmdline,data_name,ServiceName,exec_info_data,data_type,raw_info,aliuid,logtime);","version":"1.0","extendField": [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10080&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10080","type":"script","value":"ali_uid='';\nclient_ip='';\nevent_type='持久化后门';\nevent_name='异常计划任务(体检)';\nlevel='high';\n___cast_10144=cast(uuid,'string');\n___cast_10145=cast(logtime,'string');\n___JSON_CONCAT_1001 [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10081&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10081","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_client_aegis_client_channel_10003&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink_source_aegis_client_aegis_client_channel_10003","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeInOrder":"true","maxFetchLogGroupSize":"100","syn [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10082&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10082","type":"script","value":"___json_get_10090=json_get(exec_info_data,'$.name');\ndata_name=___json_get_10090;rm('___json_get_10090');\n___json_get_10091=json_get(exec_info_data,'$.type');\ndata_type=___json_get_ [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10083&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10083","type":"script","value":"retainField(traceid,module_id,data_name,exec_info_data,data_type,raw_info,aliuid,logtime,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_client_aegis_client_rule_10014&&&&{"expressionStr":"data_type,==,'RegistryGeneral'","scriptNames":"[]","expressionName":"10058","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_rule_10014","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[]"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10058&&&&{"aesFlag":1,"varName":"data_type","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"ISXhri5DWavQ1E9PYIrdqg=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10084&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10084","type":"script","value":"___json_get_10093=json_get(raw_info,'$.regpath');\nregpath=___json_get_10093;rm('___json_get_10093');\n___json_get_10094=json_get(raw_info,'$.regvalue');\nregvalue=___json_get_10094;rm [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10085&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10085","type":"script","value":"retainField(traceid,module_id,data_name,regpath,regvalue,exec_info_data,data_type,raw_info,aliuid,logtime,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10086&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10086","type":"script","value":"___lower_regvalue_10001=lower(regvalue);\n___REGEXP_REPLACE_10004=REGEXP_REPLACE(___lower_regvalue_10001,'[\\^,#~~~~~\\\\]','');\nclean_regvalue=___REGEXP_REPLACE_10004;rm('___REGEXP_R [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10087&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10087","type":"script","value":"retainField(traceid,module_id,data_name,regpath,regvalue,exec_info_data,data_type,raw_info,aliuid,clean_regvalue,logtime,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_client_aegis_client_rule_10015&&&&{"expressionStr":"(((((((((((((((((clean_regvalue,regex,'regsvr32.+?/i:.+?scrobj\\.dll')|(clean_regvalue,regex,'msiexec(\\.exe)?\\s+/i\\s+https?://'))|(clean_regvalue,regex,'mshta(\\.exe)?\\s+https?://'))|(clean_regvalue,regex,'powershell.+?\\-e\\s+[a-z0-9/\\+]{70,}'))|(clean_regvalue,regex,'powershell.+iex.+\\.downloadstring\\('))|(clean_regvalue,regex,'powershell.+\\b(downloadfile|downloaddata)\\ [...]
-dipper.private.blink.rules&&&&express&&&&10091&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10090\",\"10074\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10090&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10089\",\"10073\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10071&&&&{"aesFlag":1,"varName":"clean_regvalue","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"rGbRaHb1gzHBJCkkEtkmfQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10093&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10092\",\"10076\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10070&&&&{"aesFlag":1,"varName":"clean_regvalue","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"2n+Wuwe9F3dp+DlBZXjzJKlOHk1M/vceN4wGP39FpBU="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10092&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10091\",\"10075\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10073&&&&{"aesFlag":1,"varName":"clean_regvalue","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"dFDEJIEmvW0olWKO1nO/n/NToP4NHtk0ILqfhwv4RDk="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10072&&&&{"aesFlag":1,"varName":"clean_regvalue","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"K8/ZIADsrlSHR6UcJcGm0R1CYQXvfX2LbXDKFXbbaOs="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10075&&&&{"aesFlag":1,"varName":"clean_regvalue","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"lA+s/2pjBgZylmdKCCk9f9gHAxmoaLeAMkSb05249fM="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10074&&&&{"aesFlag":1,"varName":"clean_regvalue","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"GkO9QHwKTBmmAMjCF3nvkLyrtSw64eX5drU2fZt5fKw="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10077&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10059\",\"10060\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10076&&&&{"aesFlag":1,"varName":"regpath","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"d2Z4d4+xxxc/RvqW0b7JkTGVrNq5Yr1KV7gsQDo62ik="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10079&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10078\",\"10062\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10078&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10077\",\"10061\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10059&&&&{"aesFlag":1,"varName":"clean_regvalue","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"XFqPbku4H9xUfMrp1IQY1YAjxm9nRo2fub66YWLpzO0="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10080&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10079\",\"10063\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10060&&&&{"aesFlag":1,"varName":"clean_regvalue","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"Ff1TeJfJS8k3qIuPll4HmWzovg6vUR1ozG+8OV9ZFplgy+vLq7ICH29K211Y93WS"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10082&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10081\",\"10065\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10081&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10080\",\"10064\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10062&&&&{"aesFlag":1,"varName":"clean_regvalue","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"AxvKmdL3nsguw9Oz1IbOdwtVLieg88TtZto1eiiv/TAT0vdUHGX/rBDT06HCSy77"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10084&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10083\",\"10067\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10061&&&&{"aesFlag":1,"varName":"clean_regvalue","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"4VwxUCRBFapbl1aw8xZCdVj8P+rSqPDZ6DwjK7XhWeA="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10083&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10082\",\"10066\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10064&&&&{"aesFlag":1,"varName":"clean_regvalue","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"IqTfIM/H3bwn+VlZ6JdZEPpuJcJ+gDSAwSZL2vJ8vh9sy593aSVReX6iZB3WiJ2a"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10086&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10085\",\"10069\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10063&&&&{"aesFlag":1,"varName":"clean_regvalue","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"K7G0GkInMpEXBQXNc0VDt7KN4byxfQp9KL1tduNL6paMVjI1vgK0ReKUgf+PwiON"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10085&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10084\",\"10068\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10066&&&&{"aesFlag":1,"varName":"clean_regvalue","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"m6zyqDhKVApl4GgJREslRF4DWkt9/S1Sd6pjznLQIBU="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10088&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10087\",\"10071\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10065&&&&{"aesFlag":1,"varName":"clean_regvalue","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"j2a3YippAFdAbrToOmedNMUjQzS7q+iIhy4RhZZg2TQM3AlbEkwBfCyHC5igc5fT+Fb7xGz/cbj/ThfQ6rRCHqtwXseuXzIBd5EfETK/EEXFd51Q2/R/ILR8K0I3n4pIV4372Lb4kv22bS0CslHyxMAtsuxwwPO/xjJc9xGQlHw="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10087&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10086\",\"10070\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10068&&&&{"aesFlag":1,"varName":"clean_regvalue","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"4mg+vxX+MZbNfIPcD7gJO/H/bHrD7VKwUBY0qOmsWSTAR9sCBJgMK8qgUjmMtsjBmp8h4W6JRbR7Aujec0l2xw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10067&&&&{"aesFlag":1,"varName":"clean_regvalue","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"AYC9rVVanS3Jeomd0SAeaXy+V7Zv4m3sSnImuJ7XMQI="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10089&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10088\",\"10072\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10069&&&&{"aesFlag":1,"varName":"clean_regvalue","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"vHpCF0SXL5Xqgf+lUmb7cm+VbZb4pRtg+1XJ974IN0OqNhqHVWEudoaWEeqEHH8PO36+jok7Z5oPprxgPlvGsksmAfwoy0+5FMxlQJeeGnPsRVn2ab9Ur2Vo/EJy39l9SUUGA4PaVJGNyFaGNe+RYxeDvOty6hi/icEw+uUwOXz13fr9Q9SwXecW3nhybJgDyVYBJ+cJnIdl6u+rs9DTwy3JKXjXSnt [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10088&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10088","type":"script","value":"___ACTION_JSON_10024=ACTION_JSON('4','name','modify_regedit','key',regpath,'type','REG_SZ','value','#');\naction_json_str=___ACTION_JSON_10024;rm('___ACTION_JSON_10024');\n","version": [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10089&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10089","type":"script","value":"retainField(traceid,regpath,action_json_str,uuid,module_id,data_name,regvalue,exec_info_data,data_type,raw_info,aliuid,clean_regvalue,logtime);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10090&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10090","type":"script","value":"ali_uid=aliuid;\nclient_ip='';\nevent_type='持久化后门';\nevent_name='异常计划任务(体检)';\nlevel='high';\n___cast_10146=cast(uuid,'string');\n___cast_10147=cast(logtime,'string');\n___JSON_CONCAT_ [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10091&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10091","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_client_aegis_client_channel_10004&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink_source_aegis_client_aegis_client_channel_10004","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeInOrder":"true","maxFetchLogGroupSize":"100","syn [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_client_aegis_client_rule_10016&&&&{"expressionStr":"(data_type,==,'LinuxPubKey')&(((((((((raw_info_line,like,'%AAAAB3NzaC1yc2EAAAADAQABAAABAQDfxLBb/eKbi0TVVULI8ILVtbv2iaGM+eZbZoCWcD3v/eF1B/VkHAC1YwIhfqkUYudwhxVfQzsOZYQmKyapWzgp3tBAxcl82Al++VQc36mf/XFnECHndJS1JZB429/w/Ao+KlASl/qzita61D2VsXyejIQIeYR7Ro+ztLSTXjx+70CvzgOae3oayunL/hGX8qORIkG5YR3R1Jefhxy1NhGxEd6GaR7fZA5QWGfM17IcSXi2Q876JL8U7Aq8cjQyN/kGT2jWiiQiOZzqbjVJVICiwk0KvtrTwppV6FLt [...]
-dipper.private.blink.rules&&&&express&&&&10095&&&&{"aesFlag":1,"varName":"raw_info_line","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"BZyzV+vOpM9E4GE/C1+u1R3hAW1fOH9aQGx9ahg0yNudFHWwPe2FE44JaD64ScqNAWl0kpKYqdSFxtIU7kljwMkPVASi7+bRhuv0V+0kfUNuJE5/tKhO8VTxroNeOFvuHaoxBocCdBX9AGUnl++odyjQc5TuDEwvZgWk2edDpC2mSIUj8UbvbCaTKCnvQGxo+abHOEfyDywjmGUZePU/L3FLcRsSiTY+r [...]
-dipper.private.blink.rules&&&&express&&&&10094&&&&{"aesFlag":1,"varName":"data_type","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"L+tfStNGCgbXQV5HUvPbBQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10097&&&&{"aesFlag":1,"varName":"raw_info_line","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"BZyzV+vOpM9E4GE/C1+u1dG/roJI7dy8yHHcvUlv3L8LCMKGsui0CYCVBkxKX6uqhMEwuQm3lAiyVgDXrHLFrjitLM7hzpxgj7MuNTYuZnPklTLy5Al1COo8bZtdU+x+nK32Ajspo4qHO+/w/L2PyniZq9WaVBtE/LZ4yTrnif2sjAdv0lBCuqZIiKT9SHf0DjVrX4oWzCj+IznfUA43IxdLzp7rvkl5j [...]
-dipper.private.blink.rules&&&&express&&&&10096&&&&{"aesFlag":1,"varName":"raw_info_line","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"BZyzV+vOpM9E4GE/C1+u1etnswILW8xZkCu2PhDMMSj61KJYuJqqkMSGwlgyyfwtHYbZEDB6YoMfPyRHP0gsM9M63nKtIp8aDr65/Mubu/e28eRSV+e5/nhY8cya/LYo9HNYcAuKIKaaWAyoebsAmLcJH5CU4PUJ1w8DSTiV9UZbEJtSiUUZL3qCnxjyrasqkE+O4fKP1XsxZ1n0/cSLmbri3NC5AVmk1 [...]
-dipper.private.blink.rules&&&&express&&&&10107&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10106\",\"10099\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10106&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10105\",\"10098\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10109&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10108\",\"10101\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10108&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10107\",\"10100\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10099&&&&{"aesFlag":1,"varName":"raw_info_line","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"BZyzV+vOpM9E4GE/C1+u1etnswILW8xZkCu2PhDMMSgbKKfTXQ+LEQA116iA0ZfpqwiVnmK3nLloSbbOZgXNLOKc/EppO12yH8JphnHznUBiZ9lNYZtMldox0QBsvEzKgOUNLxHhHNFv1ICPYVBVOI3Ug4J1AIhwEZFJdPRirk/xIx15b96rtHt3UD4D8Brnvh+2VHFFdU96S2OSKt5TZfDKtmqovotFI [...]
-dipper.private.blink.rules&&&&express&&&&10110&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10109\",\"10102\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10098&&&&{"aesFlag":1,"varName":"raw_info_line","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"BZyzV+vOpM9E4GE/C1+u1etnswILW8xZkCu2PhDMMShO2vWDXoHDyQ1dCl1s2me5m8XYhv781gNSHLgzdXUjQVAL5WfpgMm6FasOOcML5iQSISAwcUkn+VuxPL3yY7xq81dP/qnYH+zs/kfLtsl0/T4rZ+jRG/dJhd4kzDI8MTPRIQBC56xhAjN7myFgL2X/FOiTNtnr2RMdepptIeZN586QyAV61jZNE [...]
-dipper.private.blink.rules&&&&express&&&&10101&&&&{"aesFlag":1,"varName":"raw_info_line","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"BZyzV+vOpM9E4GE/C1+u1R3hAW1fOH9aQGx9ahg0yNt8dN6xyUIPvRTC9ZxV1HmLyIXAIFDFLNihY8yyfy57+B0wEM5icSCuVXBAx/t4PRz+hULepYjgent35th+F7DNXPtOZPi9mqjM2b/ZogoGVk5gIN1Da8Sf8fReXezb0/1/jHNZ2J5VUcp0WxGepNZVQlBAQDScJ7TgSTGih9sZVDWCVkOSKRBS8 [...]
-dipper.private.blink.rules&&&&express&&&&10112&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10094\",\"10111\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10100&&&&{"aesFlag":1,"varName":"raw_info_line","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"BZyzV+vOpM9E4GE/C1+u1R3hAW1fOH9aQGx9ahg0yNt82om/CvGg1RVoPa9yicD9DpX/CL3lrWNYCrVePRdPodqQJSk+/GiZvUSxm4VMmTGdR/AaHZKXqVFOOlkyiUusVUHhgtr67pmHxlkc4mPMBDTZYZXKO7mMDscK2DozrGQ/2jzTuY0DyBsoia+xMBFoiV6AzSU4gpfW6+B7rrA0l2tcrc/HsEzS9 [...]
-dipper.private.blink.rules&&&&express&&&&10111&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10110\",\"10103\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10103&&&&{"aesFlag":1,"varName":"raw_info_line","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"BZyzV+vOpM9E4GE/C1+u1bkkDFrPHhbxCC/B8z3puSbzjTCG7R4kt3felgsOX9DzYpbc8BBRCzbmp4y8gzVEidhhJQHlJpy2jVJYq4V9onswKddBtjwvDiHFIdXZhufE/DzQd+pwdNuERsqS5srrB6QAXN+Cq1Me3luoo2TiH1kcBFAIuHBuhFfL9nvpP/Kut7Rw2sqOaLPuBnoRFnoDcjX9TS2lBv9H1 [...]
-dipper.private.blink.rules&&&&express&&&&10102&&&&{"aesFlag":1,"varName":"raw_info_line","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"BZyzV+vOpM9E4GE/C1+u1RL4jtf9LpjbgPbnoNdnKaB8plVaD7XXdsnFWqfmrfkuzHLef7y34QfHA4x4xUjDYRtbznkKSAHn+pXY4acczAQlU11b9T+wejb/YriadpGGgy0jxqGYcy796E3gpR7aDI8lNYGb4ekKVd/IuOVd+Fpa9GHK0poBZzIIcVHtsNe559CxUVlqCfpBe66IcUfsJVmcQ1lkwqg77 [...]
-dipper.private.blink.rules&&&&express&&&&10105&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10104\",\"10097\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10104&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10095\",\"10096\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10092&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10092","type":"script","value":"ali_uid='';\nclient_ip='';\nevent_type='持久化后门';\nevent_name='SSH后门公钥(体检)';\nlevel='high';\n___cast_10010=cast(uuid,'string');\n___cast_10011=cast(logtime,'string');\n___cast_10012=cast [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10093&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10093","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_client_aegis_client_channel_10005&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink_source_aegis_client_aegis_client_channel_10005","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeInOrder":"true","maxFetchLogGroupSize":"100","syn [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10094&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10094","type":"script","value":"___lower_raw_info_procname_10001=lower(raw_info_procname);\n___lower_raw_info_procname_10002=lower(raw_info_procname);\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_client_aegis_client_rule_10017&&&&{"expressionStr":"(data_type,==,'ProcessGeneral')&((((clean_cmd,like,'%powershell%')&(((((((((((clean_cmd,regex,'(downloadfile|downloaddata)')|(clean_cmd,regex,'\\-(e|en|ec|enc|enco|encod|encode|encoded|encodedc|encodedco|encodedcom|encodedcomm|encodedcomma|encodedcomman)\\s+[a-z0-9]'))|(clean_cmd,regex,'\\-ep\\s+bypass'))|(clean_cmd,regex,'\\-nop\\s'))|(clean_cmd,regex,'\\-w\\s+(1|hidden)'))|(clea [...]
-dipper.private.blink.rules&&&&express&&&&10150&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10132\",\"10133\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10130&&&&{"aesFlag":1,"varName":"clean_cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"4VwxUCRBFapbl1aw8xZCdVj8P+rSqPDZ6DwjK7XhWeA="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10152&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10149\",\"10151\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10151&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10131\",\"10150\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10118&&&&{"aesFlag":1,"varName":"clean_cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"ouQjhbeAwkNjrRvJ8UMqmw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10117&&&&{"aesFlag":1,"varName":"clean_cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"DxS0iDNXMTm3R/bUrlI4Ew=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10139&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10138\",\"10121\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10119&&&&{"aesFlag":1,"varName":"clean_cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"pJf/w2pXTWmKHUew8iYxt1ZbZjQzO/biGjN8DMtK0/o="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10132&&&&{"aesFlag":1,"varName":"clean_cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"jnLRKPaJrt+U/7jhC9V7+Q=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10131&&&&{"aesFlag":1,"varName":"___lower_raw_info_procname_10002","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"4z7wKva/e4zfA08jEM4BaA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10153&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10113\",\"10152\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10134&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10115\",\"10116\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10133&&&&{"aesFlag":1,"varName":"clean_cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"vnQBvkfNvdlhhMP/hu6jdQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10114&&&&{"aesFlag":1,"varName":"clean_cmd","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"GxCS7TuHXaBMYH4e8eW32w=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10136&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10135\",\"10118\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10113&&&&{"aesFlag":1,"varName":"data_type","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"XiZqpAF5OpdGedX7s8cLFg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10135&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10134\",\"10117\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10116&&&&{"aesFlag":1,"varName":"clean_cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"OrnzgTI9Ufwp1/kZzZG+NzAQmbLyWvVqTUuRQiSUJJ+Xu05+Kdd7XJemezuep44L2yssNQnog5RfHSCxdwF1JpTlW2sjBGnQvHE3JVlfRhdB5um86EHf52W2G9FARcI88aE0Z0X7uKgc55vNT7zOtKPGt6c5WNk1gLqlVt4Pg+g="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10138&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10137\",\"10120\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10115&&&&{"aesFlag":1,"varName":"clean_cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"sC2O3jJ3TKisOINmoF7BmJBR/jrgpxBsxIi3LdYHCfo="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10137&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10136\",\"10119\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10141&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10140\",\"10123\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10140&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10139\",\"10122\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10129&&&&{"aesFlag":1,"varName":"___lower_raw_info_procname_10001","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"XbhcTcRCF+f3m8FZ8EY4gg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10128&&&&{"aesFlag":1,"varName":"clean_cmd","functionName":"!like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"f7LNrebShYHIo0rK/UJqkVex4bSD8ACQvziOSP6/25g="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10121&&&&{"aesFlag":1,"varName":"clean_cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"k6jXhDndhQ9goN1tiPShLyBfOCCzqkpKPqWOlKA1KwM="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10143&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10126\",\"10127\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10120&&&&{"aesFlag":1,"varName":"clean_cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"D0Uuthi0CJWNxNAaIXZirg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10142&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10141\",\"10124\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10123&&&&{"aesFlag":1,"varName":"clean_cmd","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"Jb/cCVTj1iXKNMzN7/gFFg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10145&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10144\",\"10128\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10122&&&&{"aesFlag":1,"varName":"clean_cmd","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"MjJAk95L4fizoLA150avTw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10144&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10125\",\"10143\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10125&&&&{"aesFlag":1,"varName":"clean_cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"zcpvnfW4flEhijke/ZjFAg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10147&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10114\",\"10146\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10124&&&&{"aesFlag":1,"varName":"clean_cmd","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"KWZxigPMMzTu/5XO8na/Cw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10146&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10142\",\"10145\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10127&&&&{"aesFlag":1,"varName":"clean_cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"p1undPH4vbk2O0DqzLEhBQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10149&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10147\",\"10148\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10126&&&&{"aesFlag":1,"varName":"clean_cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"hEEQwa8sroGi8ZGLurAJLfyULT9St0Y5l2H0P2yeHwg="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10148&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10129\",\"10130\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10095&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10095","type":"script","value":"ali_uid='';\nclient_ip='';\nevent_type='持久化后门';\nevent_name='异常进程(体检)';\nlevel='high';\n___cast_10027=cast(uuid,'string');\n___cast_10028=cast(logtime,'string');\n___cast_10029=cast(cm [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10096&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10096","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_client_aegis_client_channel_10006&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink_source_aegis_client_aegis_client_channel_10006","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeInOrder":"true","maxFetchLogGroupSize":"100","syn [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10097&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10097","type":"script","value":"___lower_raw_info_line_10001=lower(raw_info_line);\n___lower_raw_info_line_10002=lower(raw_info_line);\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_client_aegis_client_rule_10018&&&&{"expressionStr":"(data_type,==,'CrontabItem')&(((((((((((((((raw_info_filepath,regex,'/var/spool/cron/(tomcat|www|www\\-data|apache|httpd|jenkins|weblogic|mysql|zabbix|postgres|redis)$')&(raw_info_line,regex,'(curl |wget |://).+(\\|\\s*(bash|sh)|(&&|;)\\s*(bash|sh))'))|(___lower_raw_info_line_10001,regex,'(pastebin\\.com|\\.tor2web\\.|\\.onion\\.\\w+/)'))|((raw_info_line,regex,' \\-fsSL ')&(raw_in [...]
-dipper.private.blink.rules&&&&express&&&&10170&&&&{"aesFlag":1,"varName":"cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"AyovPj97MVHY9x62K+ZoLQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10172&&&&{"aesFlag":1,"varName":"cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"ka69jAAiNWnzRgL0a48Cnw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10171&&&&{"aesFlag":1,"varName":"cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"2lH4GJEiIIHCo5tBHjFPEOslYnaUJcpOXNOs5XbWJHE="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10174&&&&{"aesFlag":1,"varName":"cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"FlORTfzCcyv0VXlFx/24EyLnGPzKfY0PtK0RhzEhmzk="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10173&&&&{"aesFlag":1,"varName":"cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"N619AMCXLmmPL7ATzW/VKw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10217&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10216\",\"10184\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10216&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10215\",\"10183\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10219&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10218\",\"10186\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10218&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10217\",\"10185\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10176&&&&{"aesFlag":1,"varName":"cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"6Ehxe/ysZelO3CsbJZ+s9g=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10175&&&&{"aesFlag":1,"varName":"cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"WIBK8so6rl0DaIJrn+cHrA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10178&&&&{"aesFlag":1,"varName":"raw_info_line","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"ybUzb0GNXxNAcTWAAPvJPzmDdfy/aASkMeqe6rFpxOQmr395aMACKwpBQ4RaHfPM"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10211&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10177\",\"10178\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10177&&&&{"aesFlag":1,"varName":"raw_info_line","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"04Fs9zdLfT/T/2h2zT9wjHwQ1t2ykCLotBYzrT75gGNXOHkfRM5XzhYHVc/JfF7H"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10210&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10208\",\"10209\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10213&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10212\",\"10180\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10179&&&&{"aesFlag":1,"varName":"raw_info_line","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"CJ8RVh3s4NkyMMceJ2ivAvjBBnb4FBzVtYe6CaX44BQ="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10212&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10211\",\"10179\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10215&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10214\",\"10182\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10214&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10213\",\"10181\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10181&&&&{"aesFlag":1,"varName":"raw_info_line","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"a7S9HBSu7DhJWJECYfsB2Q=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10180&&&&{"aesFlag":1,"varName":"raw_info_line","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"qhBbui9oktTasoU9RVBm11UIGHuPM81CCqhdnoCq4xo="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10183&&&&{"aesFlag":1,"varName":"raw_info_line","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"apoIbsLmNBxxeJCZbkh8yMIz6qmMnvduOA2LwxRUB9Y="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10182&&&&{"aesFlag":1,"varName":"raw_info_line","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"cKixOsIQ7FrAjU3C/q3usw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10185&&&&{"aesFlag":1,"varName":"raw_info_line","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"TD7IYlS5dZjfwO5AoHBjT4btf2k3CU+2dC6+1B+IgCo="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10184&&&&{"aesFlag":1,"varName":"raw_info_line","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"pEEnBL97WgC3bIC9ijDaUwiIp+heu6NYrM+OQ7go0qk="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10187&&&&{"aesFlag":1,"varName":"raw_info_line","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"rYwAgKtbXbiH8bWYdhoqY6LPMeGjL7kidhj76ytNZVk="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10220&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10219\",\"10187\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10186&&&&{"aesFlag":1,"varName":"raw_info_line","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"BDQshGbWFwEEax9X3au+YA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10189&&&&{"aesFlag":1,"varName":"raw_info_line","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"nxDSGLs0omeJL0BOA7jZzw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10222&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10221\",\"10189\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10188&&&&{"aesFlag":1,"varName":"raw_info_line","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"VqXp6KMOEZOLYJ5AgBObZhe5WBeg5knQLPjPWlO0Nd4="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10221&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10220\",\"10188\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10224&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10154\",\"10223\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10223&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10210\",\"10222\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10190&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10155\",\"10156\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10192&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10158\",\"10159\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10191&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10190\",\"10157\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10194&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10160\",\"10161\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10193&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10191\",\"10192\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10196&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10195\",\"10162\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10195&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10193\",\"10194\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10154&&&&{"aesFlag":1,"varName":"data_type","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"xHHZWDjQ0GhmZS+s1WEivg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10198&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10197\",\"10164\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10197&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10196\",\"10163\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10156&&&&{"aesFlag":1,"varName":"raw_info_line","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"WEMIKIZ571Q4Y/QQpqN7TEfYYuctkW76TRNRMp6/BWXN048NleKFFrqwHqF1elcwndVVd0iQ5Slmnhf3Z6tPsQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10155&&&&{"aesFlag":1,"varName":"raw_info_filepath","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"n23p4wIjNkxbIVURyDrxT5YNcmbNw1/LVGeSSFcbz3K1qtwn4mXSPZBBXyUTM+qvf5FUiQ3Kq9agSZDZCcYha10OjItTqfE3Gov7nr6Et5DPs/16Jd6GDDl04U3R+VTviEHuuELUKyJ7mMT/Cz8Dww=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10199&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10166\",\"10167\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10158&&&&{"aesFlag":1,"varName":"raw_info_line","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"72D2FUR8NOgHMNGaohKA/w=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10157&&&&{"aesFlag":1,"varName":"___lower_raw_info_line_10001","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"y9QWBxsY2OAY2ZznpLEXqhgoHSHYCVrcVUvDICZEeuHvVW01rz+Kfq8MQXqUPnfz"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10159&&&&{"aesFlag":1,"varName":"raw_info_line","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"Fq2bQGe9tGv1Qkyc8Q5PMA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10161&&&&{"aesFlag":1,"varName":"raw_info_line","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"vYIzbHNZQFejinB8Ru5YFnuS7DkIhcONHUl88tunJG0="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10160&&&&{"aesFlag":1,"varName":"raw_info_line","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"DS9OcRjwEq27ARivXv0rFqQBapesfw1T5cDImS1poPU="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10163&&&&{"aesFlag":1,"varName":"raw_info_line","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"DipVb4enDRvPCfKcV5R6f3rx5jSjKeseoqbFBaOLYYZ+nfvXWbCiMeyztKjveoUv"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10162&&&&{"aesFlag":1,"varName":"___lower_raw_info_line_10002","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"WEMIKIZ571Q4Y/QQpqN7TAyAnuA4Rf9GTXS/iTA8IU9gXadF+5P1WKmFDBVKgGtNV6jhdpZFbeTYZVut72HUXQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10206&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10172\",\"10173\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10205&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10202\",\"10204\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10208&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10207\",\"10174\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10207&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10205\",\"10206\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10209&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10175\",\"10176\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10165&&&&{"aesFlag":1,"varName":"raw_info_line","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"Tcg8as3TPnYJKB38oRhTKA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10164&&&&{"aesFlag":1,"varName":"raw_info_line","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"aUTk6gy9vmynBzG6n7soacHLailGLMFWVJKtYmuA8bk="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10167&&&&{"aesFlag":1,"varName":"raw_info_line","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"+2F4pj/+Wl5dV5xNp2noqg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10200&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10165\",\"10199\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10166&&&&{"aesFlag":1,"varName":"raw_info_line","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"SJG15mQS56+JNuCUdfjm7A=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10169&&&&{"aesFlag":1,"varName":"cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"ka69jAAiNWnzRgL0a48Cnw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10202&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10201\",\"10168\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10168&&&&{"aesFlag":1,"varName":"raw_info_line","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"2BTOjUy8J0x4M0w3P5HlQyaCmG/P5Bw8xl9dHz3pCC0="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10201&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10198\",\"10200\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10204&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10203\",\"10171\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10203&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10169\",\"10170\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10098&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10098","type":"script","value":"ali_uid='';\nclient_ip='';\nevent_type='持久化后门';\nevent_name='异常计划任务(体检)';\nlevel='high';\n___cast_10044=cast(uuid,'string');\n___cast_10045=cast(logtime,'string');\n___cast_10046=cast( [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10099&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10099","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_client_aegis_client_channel_10007&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink_source_aegis_client_aegis_client_channel_10007","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeInOrder":"true","maxFetchLogGroupSize":"100","syn [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_client_aegis_client_rule_10019&&&&{"expressionStr":"(data_type,==,'ScheduleTask')&((((clean_cmd,regex,'echo\\s.*?get\\s.*?ftp.*\\-s:')|((clean_cmd,like,'%powershell%')&(((((((((((clean_cmd,regex,'(downloadfile|downloaddata)')|(clean_cmd,regex,'\\-(e|en|ec|enc|enco|encod|encode|encoded|encodedc|encodedco|encodedcom|encodedcomm|encodedcomma|encodedcomman)\\s+[a-z0-9]'))|(clean_cmd,regex,'\\-ep\\s+bypass'))|(clean_cmd,regex,'\\-nop\\s [...]
-dipper.private.blink.rules&&&&express&&&&10251&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10250\",\"10236\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10250&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10249\",\"10235\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10239&&&&{"aesFlag":1,"varName":"clean_cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"hEEQwa8sroGi8ZGLurAJLfyULT9St0Y5l2H0P2yeHwg="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10238&&&&{"aesFlag":1,"varName":"clean_cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"zcpvnfW4flEhijke/ZjFAg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10231&&&&{"aesFlag":1,"varName":"clean_cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"ouQjhbeAwkNjrRvJ8UMqmw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10253&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10239\",\"10240\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10230&&&&{"aesFlag":1,"varName":"clean_cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"DxS0iDNXMTm3R/bUrlI4Ew=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10252&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10251\",\"10237\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10233&&&&{"aesFlag":1,"varName":"clean_cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"D0Uuthi0CJWNxNAaIXZirg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10255&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10254\",\"10241\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10232&&&&{"aesFlag":1,"varName":"clean_cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"pJf/w2pXTWmKHUew8iYxt1ZbZjQzO/biGjN8DMtK0/o="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10254&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10238\",\"10253\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10235&&&&{"aesFlag":1,"varName":"clean_cmd","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"MjJAk95L4fizoLA150avTw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10257&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10227\",\"10256\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10234&&&&{"aesFlag":1,"varName":"clean_cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"k6jXhDndhQ9goN1tiPShLyBfOCCzqkpKPqWOlKA1KwM="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10256&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10252\",\"10255\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10237&&&&{"aesFlag":1,"varName":"clean_cmd","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"KWZxigPMMzTu/5XO8na/Cw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10259&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10258\",\"10242\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10236&&&&{"aesFlag":1,"varName":"clean_cmd","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"Jb/cCVTj1iXKNMzN7/gFFg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10258&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10226\",\"10257\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10260&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10259\",\"10243\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10240&&&&{"aesFlag":1,"varName":"clean_cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"p1undPH4vbk2O0DqzLEhBQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10261&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10225\",\"10260\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10228&&&&{"aesFlag":1,"varName":"clean_cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"sC2O3jJ3TKisOINmoF7BmJBR/jrgpxBsxIi3LdYHCfo="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10227&&&&{"aesFlag":1,"varName":"clean_cmd","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"GxCS7TuHXaBMYH4e8eW32w=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10249&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10248\",\"10234\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10229&&&&{"aesFlag":1,"varName":"clean_cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"OrnzgTI9Ufwp1/kZzZG+NzAQmbLyWvVqTUuRQiSUJJ+Xu05+Kdd7XJemezuep44L2yssNQnog5RfHSCxdwF1JpTlW2sjBGnQvHE3JVlfRhdB5um86EHf52W2G9FARcI88aE0Z0X7uKgc55vNT7zOtKPGt6c5WNk1gLqlVt4Pg+g="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10242&&&&{"aesFlag":1,"varName":"clean_cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"4VwxUCRBFapbl1aw8xZCdVj8P+rSqPDZ6DwjK7XhWeA="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10241&&&&{"aesFlag":1,"varName":"clean_cmd","functionName":"!like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"f7LNrebShYHIo0rK/UJqkVex4bSD8ACQvziOSP6/25g="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10244&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10228\",\"10229\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10243&&&&{"aesFlag":1,"varName":"clean_cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"qsxpa9k7JgugBVJOgX0OOU6/ZHZF5QDeEDmLSMirjIY="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10246&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10245\",\"10231\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10245&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10244\",\"10230\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10226&&&&{"aesFlag":1,"varName":"clean_cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"uHktvI6l7A0arVp97UP3LXXWvdClqf67Lf1iXL9Oi90="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10248&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10247\",\"10233\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10225&&&&{"aesFlag":1,"varName":"data_type","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"anUuK2mD3ioF20PxopRM9w=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10247&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10246\",\"10232\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10100&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10100","type":"script","value":"ali_uid='';\nclient_ip='';\nevent_type='持久化后门';\nevent_name='异常计划任务(体检)';\nlevel='high';\n___cast_10061=cast(uuid,'string');\n___cast_10062=cast(logtime,'string');\n___cast_10063=cast( [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10101&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10101","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_client_aegis_client_channel_10008&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink_source_aegis_client_aegis_client_channel_10008","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeInOrder":"true","maxFetchLogGroupSize":"100","syn [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10102&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10102","type":"script","value":"___!null_10002=!null(data_name);\n___!null_10003=!null(cmd);\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_client_aegis_client_rule_10020&&&&{"expressionStr":"((data_type,==,'ld_so_preload')&(raw_info_dynamic,regex,' T readdir'))|(((((data_type,==,'WMI')&___!null_10002)&(data_name,!in,'\\'N/A\\',\\'\\''))&___!null_10003)&(cmd,!in,'\\'N/A\\',\\'\\',\\'cscript KernCap.vbs\\''))","scriptNames":"[]","expressionName":"10272","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rul [...]
-dipper.private.blink.rules&&&&express&&&&10271&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10270\",\"10266\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10270&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10269\",\"___!null_10003\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10262&&&&{"aesFlag":1,"varName":"data_type","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"gLxi5VktFJrwW3OKtdQp3g=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10272&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10267\",\"10271\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10264&&&&{"aesFlag":1,"varName":"data_type","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"9Lz999aE2ngcyYuYoLoV4A=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10263&&&&{"aesFlag":1,"varName":"raw_info_dynamic","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"xt6pxCaaTUHThRq0tfGH+w=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10266&&&&{"aesFlag":1,"varName":"cmd","functionName":"!in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"s4XnGzCV421Rf4tamm1ESwcmCsEQRHx9L15rNSfbdxY="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10265&&&&{"aesFlag":1,"varName":"data_name","functionName":"!in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"QhO8c5+E1GcoCgDIqz67dw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10268&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10264\",\"___!null_10002\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10267&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10262\",\"10263\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10269&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10268\",\"10265\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10103&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10103","type":"script","value":"ali_uid='';\nclient_ip='';\nevent_type='持久化后门';\nevent_name='异常计划任务(体检)';\nlevel='high';\n___cast_10078=cast(uuid,'string');\n___cast_10079=cast(logtime,'string');\n___cast_10080=cast( [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10104&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10104","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_client_aegis_client_channel_10009&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink_source_aegis_client_aegis_client_channel_10009","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeInOrder":"true","maxFetchLogGroupSize":"100","syn [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink_source_aegis_client_aegis_client_union_10001&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"entityName\\\\\\\":\\\\\\\"script\\\\\\\",\\\\\\\"scriptName\\\\\\\":\\\\\\\"subpipline_blink_source_aegis_client_aegis_cl [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink_source_aegis_client_aegis_client_union_10001_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10001_script_10001","type":"script","value":"___json_get_10037=json_get(procfs_fd,'$.0');\n___json_get_10038=json_get(procfs_fd,'$.1');\n___json_get_10039=json_get(procfs_fd,'$.2');\n [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_blink_source_aegis_client_aegis_client_union_10001_rule_10001&&&&{"expressionStr":"((((procpath,regex,'/s?bin/(bash|sh|dash|zsh|csh|ash)$')&(cmd,regex,'^\\s*([/a-z]+/)?(bash|sh|dash)(\\s+\\-{1,2}\\w+){0,3}\\s*$'))&(___json_get_10037,like,'socket:%'))&(___json_get_10038,like,'socket:%'))&(___json_get_10039,like,'socket:%')","scriptNames":"[]","expressionName":"10281","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProc [...]
-dipper.private.blink.rules&&&&express&&&&10280&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10279\",\"10276\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10281&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10280\",\"10277\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10273&&&&{"aesFlag":1,"varName":"procpath","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"OBp1px8RK6IShOh8AAr0iNiACLtpVPuxvjjGqwECe2CFlei/vdLxd0i7pPmkRMA8"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10275&&&&{"aesFlag":1,"varName":"___json_get_10037","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"aBoycb0ObB8S4GNkLFcMBA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10274&&&&{"aesFlag":1,"varName":"cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"fPD9oXUqpQyYZMUxBHXE8eS8ejltc+NAc6ft0uRWZ0xSos/IrEKP8BNMubxm01sWQLQFQ1ERmiLgE9fyi7jxyA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10277&&&&{"aesFlag":1,"varName":"___json_get_10039","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"aBoycb0ObB8S4GNkLFcMBA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10276&&&&{"aesFlag":1,"varName":"___json_get_10038","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"aBoycb0ObB8S4GNkLFcMBA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10279&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10278\",\"10275\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10278&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10273\",\"10274\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_blink_source_aegis_client_aegis_client_union_10001_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10001_script_10002","type":"script","value":"virus_name='kill_process_with_pid';\n___cast_10095=cast(pid,'string');\n___ACTION_JSON_10001=ACTION_JSON('3','name','kill_process','pid',_ [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink_source_aegis_client_aegis_client_union_10001_script_10003&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10001_script_10003","type":"script","value":"retainField(traceid,procpath,create_time,pid,action_json_str,uuid,cwd,module_id,data_name,process_status,proc_exe_raw,procfs_fd,exec_info_ [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink_source_aegis_client_aegis_client_union_10002&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"entityName\\\\\\\":\\\\\\\"script\\\\\\\",\\\\\\\"scriptName\\\\\\\":\\\\\\\"subpipline_blink_source_aegis_client_aegis_cl [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink_source_aegis_client_aegis_client_union_10002_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10002_script_10001","type":"script","value":"___!null_10004=!null(cmd);\n___lower_procname_10001=lower(procname);\n___lower_cmd_10004=lower(cmd);\n___lower_cmd_10005=lower(cmd);\n___l [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_blink_source_aegis_client_aegis_client_union_10002_rule_10001&&&&{"expressionStr":"(((((((((((((((((((((((cmd,like,'%--donate-level=%')|(cmd,like,'%--cpu-max-threads-hint=%'))|(cmd,like,'%--cpu-no-yield%'))|(cmd,like,'%stratum+tcp://%'))|(cmd,like,'%--max-cpu-usage%'))|((procpath,regex,'/(bash|sh|dash)$')&((cmd,regex,'echo[,\\s]+.+?\\|.*?base64[,\\s].*\\-\\w*d.*?\\|.*?sh')|(cmd,regex,'echo[,\\s]+[a-zA-Z0-9/+]{1024,}$'))))|((((cmd,regex,'\\ [...]
-dipper.private.blink.rules&&&&express&&&&10291&&&&{"aesFlag":1,"varName":"cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"cECkFlqt73o1xSPuzzd9z8UjXb5eh3aylRFhVB8A5to="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10290&&&&{"aesFlag":1,"varName":"cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"/Kpj4ers4UMmVZ4mIUFvjw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10293&&&&{"aesFlag":1,"varName":"cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"eP9+Z2NwGbAhTQ7h9nnmiaf/mgkt5YuGhjEDqecB2pc="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10370&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10322\",\"10323\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10292&&&&{"aesFlag":1,"varName":"cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"AyovPj97MVHY9x62K+ZoLQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10295&&&&{"aesFlag":1,"varName":"proc_exe_raw","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0hETEOhdPOhgsHqf9cITJ3uajHBEpIyhvtkptEzWPnE="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10372&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10371\",\"10327\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10294&&&&{"aesFlag":1,"varName":"cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"c6ggrlQrUeyhEqAP2etRwZM2jynxCzg2iUjWE1Dt+LEilGSPJSB98immmDV/Qw9D/R7TAkEEnU44a6fN64zzDjqD2pCD8cugrM+TXt6AkG4="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10371&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10325\",\"10326\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10338&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10337\",\"10292\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10337&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10290\",\"10291\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10339&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10338\",\"10293\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10297&&&&{"aesFlag":1,"varName":"proc_exe_raw","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"M7grO0DWTNleGtk17FTQSw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10330&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10282\",\"10283\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10374&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10373\",\"10328\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10296&&&&{"aesFlag":1,"varName":"cwd","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"uU6tQYF2jEuonNwagoHnAw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10373&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10324\",\"10372\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10299&&&&{"aesFlag":1,"varName":"cmd","functionName":"<>","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"M7grO0DWTNleGtk17FTQSw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10332&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10331\",\"10285\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10376&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10370\",\"10375\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10298&&&&{"aesFlag":1,"varName":"procpath","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":""}&&&&null
-dipper.private.blink.rules&&&&express&&&&10331&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10330\",\"10284\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10375&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10374\",\"10329\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10334&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10288\",\"10289\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10333&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10332\",\"10286\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10377&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10369\",\"10376\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10336&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10333\",\"10335\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10335&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10287\",\"10334\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10305&&&&{"aesFlag":1,"varName":"procpath","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"TEqV2bs1+47RS4TN6RHJoj1smKqFn43+X/2SteL6LKo="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10349&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10348\",\"10301\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10304&&&&{"aesFlag":1,"varName":"cmd","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"NKWvFA2QnoLjObpZvMK9Ng=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10348&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10342\",\"10347\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10307&&&&{"aesFlag":1,"varName":"cmd","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"h5rbOtEln+0UEYHnC+pvln1MRRnW0L5CHKHb7h8SCrK7miRFMJt5qxpa/QYpz8jzLxZ/F8Nk+9NAhR8iRSdEO8h+k5MdrbXv5W/hLuQokzk="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10306&&&&{"aesFlag":1,"varName":"cmd","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"KptcWM/7vcbA4u2dB369SiaIa7maCMGBNahH6RvrFINy/gRcghRvZKzhTwHmwnmNjSJZ/vOZYUVuZb3BpXMIygc1TVTodrzkJmvRlu/1v38="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10309&&&&{"aesFlag":1,"varName":"cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"OhDxw3jw4Yu4JOWTdkAY9+qXmVxhdNlNmmJC9lvdcRghCmwHa2uMQjws9cs3Vfzc1mDgg1PXSITej1dqe231pQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10308&&&&{"aesFlag":1,"varName":"cmd","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"XGLKrhA9Z/Bd0X1CDYerXUMevnjD7nJIU61OTOAYilrZTwPELEiHxsAefUcHeSh32dmYq2IyjooH6/5g75BcMQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10341&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10340\",\"10294\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10340&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10336\",\"10339\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10343&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10296\",\"10297\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10342&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10341\",\"10295\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10301&&&&{"aesFlag":1,"varName":"procpath","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"2s8ZCLPJtD6IHbCBvcw5Tg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10345&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10344\",\"10299\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10300&&&&{"aesFlag":1,"varName":"cmd","functionName":"<>","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":""}&&&&null
-dipper.private.blink.rules&&&&express&&&&10344&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10343\",\"10298\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10303&&&&{"aesFlag":1,"varName":"procname","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"gYVFp6C3AKJNXiSlMiyn1g=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10347&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10346\",\"10300\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10302&&&&{"aesFlag":1,"varName":"cwd","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"MBK6L9UnWZkIYO80zNn2Zg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10346&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10345\",\"___!null_10004\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10350&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10302\",\"10303\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10316&&&&{"aesFlag":1,"varName":"___lower_cmd_10005","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"SIy5ekVAX0umC4T0H6Tku3aAM5B7YJvBT2LDhjONlRo="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10315&&&&{"aesFlag":1,"varName":"___lower_cmd_10004","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"IjxWrw82Xske4nRCYo8UV97x/afiUuaW5lZajqRJ1Ds="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10359&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10357\",\"10358\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10318&&&&{"aesFlag":1,"varName":"___lower_procpath_10001","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"rN9IlE/BjPaN2S1ls3SlH1YLvOtXcGAebMswW+dLs08qfq9bqrWrvn1gn6mITQLL"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10317&&&&{"aesFlag":1,"varName":"cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"iV6zd7UPQOWHgIpg1LG8cg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10319&&&&{"aesFlag":1,"varName":"procpath","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"LCnXjMaCbqDbnK1Oj0OljtgA3fBu/RGc/mYyhElKjgg="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10352&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10304\",\"10305\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10351&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10349\",\"10350\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10310&&&&{"aesFlag":1,"varName":"proc_exe_raw","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"M7grO0DWTNleGtk17FTQSw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10354&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10353\",\"10306\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10353&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10351\",\"10352\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10312&&&&{"aesFlag":1,"varName":"cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0rPu9dG7UvuiwjHloX2m3sHDmM6WMKz5jM8LAjBvPlwcgs2PzHRESIilzRQoPEmZ"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10356&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10355\",\"10308\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10311&&&&{"aesFlag":1,"varName":"cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"d2sco0Kid1RoQGaTbPRlkVSgIH04kRyyfMrVfXrGrk0="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10355&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10354\",\"10307\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10314&&&&{"aesFlag":1,"varName":"___lower_procname_10001","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"YGeHkwJkc4xJFjK1BtvAneYgtClT+Scfhae2DwOvwGrr5A9UU9NjpebRHHceTXHW"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10358&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10310\",\"10311\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10313&&&&{"aesFlag":1,"varName":"procpath","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"zk4q0251XUrIlRMMU/zZCFB67WwsYOloCTDYG4l2Uyo="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10357&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10356\",\"10309\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10282&&&&{"aesFlag":1,"varName":"cmd","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"nyaTdtZJYIShnwseCSkd6LG6H5pAsfDea3Yawq4lnas="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10284&&&&{"aesFlag":1,"varName":"cmd","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"VFT9DBd6dHwcTQ0dMJpRhrBEUkm2m9HD237K7wlBROA="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10361&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10359\",\"10360\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10283&&&&{"aesFlag":1,"varName":"cmd","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"OzIqxPzg6EFVHblrX6CeEdgBzuPbK5DDsXem5ZLyncs="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10360&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10312\",\"10313\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10327&&&&{"aesFlag":1,"varName":"clean_cmd_win","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"xoCy+NuDLdOSwd+izIcwpw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10326&&&&{"aesFlag":1,"varName":"clean_cmd_win","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"W3MqYeVHD84ZKewsibuGrA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10329&&&&{"aesFlag":1,"varName":"clean_cmd_win","functionName":"!like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"Y1xP5Zqof/nortA8qXFEv9khDvHx94PTFoE0OYMRHDU="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10328&&&&{"aesFlag":1,"varName":"clean_cmd_win","functionName":"!like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0uaghsmK0u8Q52lMr37TsQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10286&&&&{"aesFlag":1,"varName":"cmd","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"NLedzJoebdPrcoYLy1zPR641j0TO96t9l/uXXpkeVVM="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10363&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10316\",\"10317\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10285&&&&{"aesFlag":1,"varName":"cmd","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"Ta9wXKILR6hPtsj6Iv6v5YWW2bAcPNoB1mMSavZGUvA="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10362&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10314\",\"10315\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10288&&&&{"aesFlag":1,"varName":"cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"oeY+QJJj+JZDExolyrT6l7g1ozDutxCJ69vGlryXXC+pmrzzMtUFeRhMy6PlUt8F"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10321&&&&{"aesFlag":1,"varName":"cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"I7iaJaLsWkgVXDJq0WUckhysoyhGEQvoV74LSntGxWk="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10365&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10361\",\"10364\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10287&&&&{"aesFlag":1,"varName":"procpath","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"aos5iXMtHdnUxxKfkend4b2fhnOtpThh7+Ehz5U2864="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10320&&&&{"aesFlag":1,"varName":"cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"FlORTfzCcyv0VXlFx/24E3Ck/p5AqCXGIjsP1FTejnw="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10364&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10362\",\"10363\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10323&&&&{"aesFlag":1,"varName":"clean_cmd_win","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"GxCS7TuHXaBMYH4e8eW32w=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10367&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10319\",\"10320\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10289&&&&{"aesFlag":1,"varName":"cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"NaqG6xeJ1iXZ1FDA0ZpCEB1Yfp8dIl2fppXuCFY213E="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10322&&&&{"aesFlag":1,"varName":"___lower_procname_10002","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"YGeHkwJkc4xJFjK1BtvAndrjJb4sJfqN+JRKfoTDfew="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10366&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10365\",\"10318\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10325&&&&{"aesFlag":1,"varName":"clean_cmd_win","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"lmbRrPZHxX1BtO2HURD8N7EgEkV34P2k3x48NXgWDqs="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10369&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10368\",\"10321\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10324&&&&{"aesFlag":1,"varName":"clean_cmd_win","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"41IrpuxnUu5AS0El6RwFfzrh+qiY+dlJrJRRvimB2ik="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10368&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10366\",\"10367\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_blink_source_aegis_client_aegis_client_union_10002_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10002_script_10002","type":"script","value":"virus_name='general_linux_kill_process';\n___ACTION_JSON_10002=ACTION_JSON('2','name','kill_process','cmdline',cmd);\naction_json_str=___A [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink_source_aegis_client_aegis_client_union_10002_script_10003&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10002_script_10003","type":"script","value":"retainField(traceid,procpath,create_time,pid,action_json_str,uuid,cwd,module_id,data_name,process_status,proc_exe_raw,procfs_fd,exec_info_ [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink_source_aegis_client_aegis_client_union_10003&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"entityName\\\\\\\":\\\\\\\"script\\\\\\\",\\\\\\\"scriptName\\\\\\\":\\\\\\\"subpipline_blink_source_aegis_client_aegis_cl [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink_source_aegis_client_aegis_client_union_10003_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10003_script_10001","type":"script","value":"___lower_procpath_10002=lower(procpath);\n___lower_procpath_10003=lower(procpath);\n___lower_procpath_10004=lower(procpath);\n___!_10004=! [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_blink_source_aegis_client_aegis_client_union_10003_rule_10001&&&&{"expressionStr":"(((((((((((((((((((((((cmd,like,'% --library-path %')&((((procpath,like,'/tmp/%')|(procpath,like,'/dev/shm/%'))|(procpath,like,'/var/tmp/%'))|(procpath,like,'%/tsm')))|(procpath,in,'\\'/tmp/tcpp\\',\\'/root/.configrc/a/kswapd0\\',\\'/usr/lib/libiacpkmn.so.3\\''))|((procpath,like,'%/pnscan')&(cmd,like,'% 6379')))|(___lower_procpath_10002,regex,'^.:\\\\windows [...]
-dipper.private.blink.rules&&&&express&&&&10415&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10414\",\"10383\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10414&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10378\",\"10413\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10417&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10415\",\"10416\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10416&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10384\",\"10385\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10419&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10418\",\"10387\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10418&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10417\",\"10386\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10378&&&&{"aesFlag":1,"varName":"cmd","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"gXA7C5ro+OQEX8nekzQ73Ntc58AjIKSHsn61009/oBA="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10411&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10379\",\"10380\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10410&&&&{"aesFlag":1,"varName":"procpath","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"XdxudAULAmFe2aAciQT+Ng=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10413&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10412\",\"10382\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10379&&&&{"aesFlag":1,"varName":"procpath","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"3aOQXwy39398HHum+b3ipg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10412&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10411\",\"10381\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10381&&&&{"aesFlag":1,"varName":"procpath","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"r5NpQArPBuUSy1du++0y2w=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10380&&&&{"aesFlag":1,"varName":"procpath","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"jrFV0b18se7v5HvK/3mHoA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10383&&&&{"aesFlag":1,"varName":"procpath","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"YvAI2oVALaV1OYXwAzQv992qrKqmDd6rbuxvy158Odey+tvKX8S11kS1hSNlImqfjTwygi7S+96XNPf23h044xTC1o9QDLTkWqPU2cSExag="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10382&&&&{"aesFlag":1,"varName":"procpath","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"lyAoUZQJf7Bx/R6ARtXB0Q=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10426&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10425\",\"___!_10004\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10425&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10393\",\"10394\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10428&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10427\",\"10395\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10427&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10426\",\"___!_10005\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10429&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10428\",\"10396\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10385&&&&{"aesFlag":1,"varName":"cmd","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"sBu6Jkl7YhMgu+Ag2W+zAg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10384&&&&{"aesFlag":1,"varName":"procpath","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"AYhbWNuJBTNOoQVJdPPh4w=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10387&&&&{"aesFlag":1,"varName":"___lower_procpath_10003","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"c89Gm7+etClLETr9cLddnP286cOdPI59WNu+C8ZR9zY="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10420&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10419\",\"10388\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10386&&&&{"aesFlag":1,"varName":"___lower_procpath_10002","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"ZUgxfkA0JoNQ1CkC+qjYb+16vUZ72jCs0Wgz2jYtqFY="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10389&&&&{"aesFlag":1,"varName":"procpath","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"s+KKcequ2+C9/WMGShw+/0rPcNn5VjA7MJGyg7L0rr+QNxjb5l8i1BzaTxvbEnM0z/fp0ihRPI1PnD6vk4P2ExQ0ZgRI1v4em7B6KSRGeRIBmbgHNvMmVU1Tjgb8sb75"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10422&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10421\",\"10390\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10388&&&&{"aesFlag":1,"varName":"___lower_procpath_10004","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"zqq7tYSsUchhQo7ZgmBrHT50gP4076BjnMkNX98WtP70Pw7astgarWOakhtbNMWAapWEeN6TO3KYf+/wM1WKFA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10421&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10420\",\"10389\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10424&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10423\",\"10392\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10423&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10422\",\"10391\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10390&&&&{"aesFlag":1,"varName":"procpath","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"xwxXPeTIIGYgBNoCj4Tz2w=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10392&&&&{"aesFlag":1,"varName":"proc_exe_raw","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"b0JNlcUDI5pF6nlIR5ZT9WxKibHVf0RKzWjwx9OMVXg="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10391&&&&{"aesFlag":1,"varName":"proc_exe_raw","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"Sv+DlLjkFDExc/g3C+8Yw9Q1GWWT567G7tAdLDXprPM="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10394&&&&{"aesFlag":1,"varName":"procpath","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"M1psS2sL+qyjR6s0zjGxqw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10393&&&&{"aesFlag":1,"varName":"cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"4z1m5KmwuLagpcIi7x0BUw/NvnJWcbMuNk1ENM37JF7Hwcada6rTPRoZSFQ1gXC9"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10437&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10436\",\"10403\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10436&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10435\",\"10402\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10439&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10405\",\"10406\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10438&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10437\",\"10404\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10396&&&&{"aesFlag":1,"varName":"procpath","functionName":"!like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"AntpwjEjRwNnz5a5rRm8YqFVrYIUducfiHSWcu5oxuY="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10395&&&&{"aesFlag":1,"varName":"procpath","functionName":"!like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"gN2T7BT12UpBs4qlFHQP1g=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10398&&&&{"aesFlag":1,"varName":"procpath","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"xgQftohsZqKpjh+aT6Oro4ZbZs9DxgQziRq/Cci6Z/o="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10431&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10424\",\"10430\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10397&&&&{"aesFlag":1,"varName":"procpath","functionName":"!like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"EZiPvvNVKMZ4YNxZZs8ezyxSMpSrz/KUywvbAE2qQXk="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10430&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10429\",\"10397\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10433&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10432\",\"10399\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10399&&&&{"aesFlag":1,"varName":"procpath","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"YPoacOnv1U039xGtwiuEtYDAhsiAgEj/QIQMhKCPAlo="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10432&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10431\",\"10398\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10435&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10434\",\"10401\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10434&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10433\",\"10400\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10404&&&&{"aesFlag":1,"varName":"procpath","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"ZF5VYXwu5Klqsku4kMumVkzNrliHW3MLbrr0z+ruX5Q="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10403&&&&{"aesFlag":1,"varName":"procpath","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"wpFYYgrMnwvZwLSpzCZoyA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10406&&&&{"aesFlag":1,"varName":"procpath","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"dkWLfONiuSRH+CTtHFBCQw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10405&&&&{"aesFlag":1,"varName":"cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"41Nxcy7EoXonQGe5IA+efQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10408&&&&{"aesFlag":1,"varName":"procpath","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"phXAw3UZBY7ZdJY1nIntuA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10407&&&&{"aesFlag":1,"varName":"procpath","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"EWuqDik+o0wlOxvlkTI/PA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10409&&&&{"aesFlag":1,"varName":"procpath","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"/5lzyEaJTL5hYRI63j6GAqqJCeao7Qlq5GmtFKYu/ao="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10440&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10438\",\"10439\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10442&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10441\",\"10408\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10441&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10440\",\"10407\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10400&&&&{"aesFlag":1,"varName":"procpath","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"ZMJzLac21jevE3turxfhzg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10444&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10443\",\"10410\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10443&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10442\",\"10409\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10402&&&&{"aesFlag":1,"varName":"procpath","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"BWfp5wd5XtQbMHh+kZQ5NQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10401&&&&{"aesFlag":1,"varName":"procpath","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"p7O+46Y1YkU+XEYD+kI+zw=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_blink_source_aegis_client_aegis_client_union_10003_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10003_script_10002","type":"script","value":"virus_name='general_linux_kill_and_lock';\n___ACTION_JSON_10003=ACTION_JSON('3','name','kill_process','procpath',procpath,'lock_file','1') [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink_source_aegis_client_aegis_client_union_10003_script_10003&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10003_script_10003","type":"script","value":"retainField(traceid,procpath,create_time,pid,action_json_str,uuid,cwd,module_id,data_name,process_status,proc_exe_raw,procfs_fd,exec_info_ [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink_source_aegis_client_aegis_client_union_10004&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"entityName\\\\\\\":\\\\\\\"script\\\\\\\",\\\\\\\"scriptName\\\\\\\":\\\\\\\"subpipline_blink_source_aegis_client_aegis_cl [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink_source_aegis_client_aegis_client_union_10004_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10004_script_10001","type":"script","value":"___lower_procpath_10005=lower(procpath);\n___lower_cmd_10006=lower(cmd);\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&subpipline_blink_source_aegis_client_aegis_client_union_10004_rule_10001&&&&{"expressionStr":"(___lower_procpath_10005,regex,'\\\\rundll32\\.exe$')&(___lower_cmd_10006,regex,':\\\\programdata\\\\sxs\\.dll')","scriptNames":"[]","expressionName":"10447","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10004_r [...]
-dipper.private.blink.rules&&&&express&&&&10447&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10445\",\"10446\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10446&&&&{"aesFlag":1,"varName":"___lower_cmd_10006","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"xdRRx/VhKDnwulvIxkkiTV2+1A4k+90ZlF0w4oU8COw="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10445&&&&{"aesFlag":1,"varName":"___lower_procpath_10005","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"lHnZRFSx+HjvuZiiJxdKeh2GwOU/HX06xZ0p+TIMJZU="}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_blink_source_aegis_client_aegis_client_union_10004_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10004_script_10002","type":"script","value":"virus_name='SXS';\n___ACTION_JSON_10004=ACTION_JSON('3','name','kill_process','procpath',procpath,'cmd',cmd,'2','name','del_file','filepat [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink_source_aegis_client_aegis_client_union_10004_script_10003&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10004_script_10003","type":"script","value":"retainField(traceid,procpath,create_time,pid,action_json_str,uuid,cwd,module_id,data_name,process_status,proc_exe_raw,procfs_fd,exec_info_ [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink_source_aegis_client_aegis_client_union_10005&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"entityName\\\\\\\":\\\\\\\"script\\\\\\\",\\\\\\\"scriptName\\\\\\\":\\\\\\\"subpipline_blink_source_aegis_client_aegis_cl [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink_source_aegis_client_aegis_client_union_10005_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10005_script_10001","type":"script","value":"___lower_procpath_10006=lower(procpath);\n___lower_procpath_10007=lower(procpath);\n___lower_procpath_10008=lower(procpath);\n___lower_cmd [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_blink_source_aegis_client_aegis_client_union_10005_rule_10001&&&&{"expressionStr":"((___lower_procpath_10006,in,'\\'c:\\windows\\temp\\u.exe\\',\\'c:\\windows\\temp\\conhou.exe\\',\\'c:\\windows\\temp\\conhos.exe\\',\\'c:\\windows\\temp\\conhot.exe\\',\\'c:\\windows\\inf\\aspnet\\lsma12.exe\\'')|(___lower_procpath_10007,regex,'^c:\\\\windows\\\\temp\\\\conho\\w\\.exe$'))|((___lower_procpath_10008,regex,'\\\\rundll32\\.exe$')&((___lower_cmd [...]
-dipper.private.blink.rules&&&&express&&&&10448&&&&{"aesFlag":1,"varName":"___lower_procpath_10006","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"XMepMDrqrqXDI6D5eaCzhSRFZhxNQZEL80s2U8ogVyDQpcaP87fMGmIibAGXHp6qU9QR78ctrfhdn0NC6KCV5Tdd4A3ZDVficTeIYQ1Zb/KW0SFy89exnE5iBvYbQU0PtR7F7kOt3Ty5x+l5xdZq3DPLpIrUe+Z598TQeFYseLbP4sK/s6jbr5sJ1BKyxeBXBfxFDRfJ/C9HzC7dRCyoBg==" [...]
-dipper.private.blink.rules&&&&express&&&&10449&&&&{"aesFlag":1,"varName":"___lower_procpath_10007","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"tKbPsNK0wJpqVNh22l4YvyFE0TLH/6HjaH70lUXJETg4Hx5+EIyI8dccS6jLsrYH"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10451&&&&{"aesFlag":1,"varName":"___lower_cmd_10007","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0zKRH6X+31K+2nkU+d4Vw/kwRcTGEjunCUBijpo+aI0="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10450&&&&{"aesFlag":1,"varName":"___lower_procpath_10008","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"lHnZRFSx+HjvuZiiJxdKeh2GwOU/HX06xZ0p+TIMJZU="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10453&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10448\",\"10449\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10452&&&&{"aesFlag":1,"varName":"___lower_cmd_10008","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"D8wQgWTOKdbUosrAyre+Oo7HsiIaXc+E+KzquEX6jxI="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10455&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10450\",\"10454\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10454&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10451\",\"10452\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10456&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10453\",\"10455\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_blink_source_aegis_client_aegis_client_union_10005_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10005_script_10002","type":"script","value":"virus_name='conhou';\n___ACTION_JSON_10005=ACTION_JSON('2','name','exec_command','cmd','%SystemRoot%/System32/wbem/wmic.exe /NAMESPACE:--- [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink_source_aegis_client_aegis_client_union_10005_script_10003&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10005_script_10003","type":"script","value":"retainField(traceid,procpath,create_time,pid,action_json_str,uuid,cwd,module_id,data_name,process_status,proc_exe_raw,procfs_fd,exec_info_ [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink_source_aegis_client_aegis_client_union_10006&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"entityName\\\\\\\":\\\\\\\"script\\\\\\\",\\\\\\\"scriptName\\\\\\\":\\\\\\\"subpipline_blink_source_aegis_client_aegis_cl [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink_source_aegis_client_aegis_client_union_10006_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10006_script_10001","type":"script","value":"___!_10006=!((procpath,regex,'/usr/bin/[^/]+;\\w+$'));\n___!_10007=!((cmd,like,'Titan%'));\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&subpipline_blink_source_aegis_client_aegis_client_union_10006_rule_10001&&&&{"expressionStr":"(((((cmd,regex,'^[a-zA-Z0-9]{6,8}\\s{9,}$')|(cmd,regex,'^tracepath\\s{9,}$'))&((((proc_exe_raw,==,'N/A')|(proc_exe_raw,like,'/tmp/%'))|(proc_exe_raw,like,'/dev/shm/%'))|(proc_exe_raw,like,'% (deleted)')))|(procpath,==,'/tmp/.X11-unix/sshd/ssh'))&___!_10006)&___!_10007","scriptNames":"[]","expressionName":"10471","varNames":"[]","className":"com.aliyun.filter [...]
-dipper.private.blink.rules&&&&express&&&&10460&&&&{"aesFlag":1,"varName":"proc_exe_raw","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"3aOQXwy39398HHum+b3ipg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10471&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10470\",\"___!_10007\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10470&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10469\",\"___!_10006\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10459&&&&{"aesFlag":1,"varName":"proc_exe_raw","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"M7grO0DWTNleGtk17FTQSw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10458&&&&{"aesFlag":1,"varName":"cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"nUHI7Ordwx7ITv+ihmpefTTyySL/0pVGErmV+cfbHes="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10469&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10468\",\"10463\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10462&&&&{"aesFlag":1,"varName":"proc_exe_raw","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"L185Aioyc9BQCz0KBWnP4Q=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10461&&&&{"aesFlag":1,"varName":"proc_exe_raw","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"jrFV0b18se7v5HvK/3mHoA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10464&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10457\",\"10458\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10463&&&&{"aesFlag":1,"varName":"procpath","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"qyBWFcxtj3CKvRZY6qH6PzYI+Y12rMz+xdheWAqAcU8="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10466&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10465\",\"10461\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10465&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10459\",\"10460\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10457&&&&{"aesFlag":1,"varName":"cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0O8sVvfX7WQoLJxmDFf0WtGjJYkaR1eP6vJHEvxTjCo="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10468&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10464\",\"10467\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10467&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10466\",\"10462\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_blink_source_aegis_client_aegis_client_union_10006_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10006_script_10002","type":"script","value":"virus_name='20200819_dev_shm';\n___ACTION_JSON_10006=ACTION_JSON('3','name','kill_process','cmdline',cmd,'lock_file','1','3','name','kill_ [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink_source_aegis_client_aegis_client_union_10006_script_10003&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10006_script_10003","type":"script","value":"retainField(traceid,procpath,create_time,pid,uuid,cwd,___ACTION_JSON_10006,module_id,data_name,process_status,proc_exe_raw,procfs_fd,exec_ [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink_source_aegis_client_aegis_client_union_10007&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.FilterChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"names\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"subpipline_blink_source_aegis_client_aegis_client_union_10007_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\ [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_blink_source_aegis_client_aegis_client_union_10007_rule_10001&&&&{"expressionStr":"procpath,in,'\\'/var/tmp/pty3\\',\\'/var/lock/pty3\\',\\'/var/run/pty3\\',\\'/dev/shm/pty3\\',\\'/tmp/pty3\\''","scriptNames":"[]","expressionName":"10472","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10007_rul [...]
-dipper.private.blink.rules&&&&express&&&&10472&&&&{"aesFlag":1,"varName":"procpath","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"PYes9JUjllc1/j2duafDMw6eKBXFfQrN18HDTaN7h1sqjiLKsUQWVdrIGOGkmL9BwsPakkIH/BsSuxXAGdvG4QqgZTvkRus1GSgCW09CYNY="}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_blink_source_aegis_client_aegis_client_union_10007_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10007_script_10001","type":"script","value":"virus_name='pty3';\n___ACTION_JSON_10007=ACTION_JSON('3','name','kill_process','procpath','/var/tmp/pty3','lock_file','1','3','name','kill [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink_source_aegis_client_aegis_client_union_10007_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10007_script_10002","type":"script","value":"retainField(traceid,procpath,create_time,pid,action_json_str,uuid,cwd,module_id,data_name,process_status,proc_exe_raw,procfs_fd,exec_info_ [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink_source_aegis_client_aegis_client_union_10008&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.FilterChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"names\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"subpipline_blink_source_aegis_client_aegis_client_union_10008_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\ [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_blink_source_aegis_client_aegis_client_union_10008_rule_10001&&&&{"expressionStr":"((procpath,in,'\\'/usr/bin/kaudited\\',\\'/usr/bin/kswaped\\',\\'/usr/bin/irqbalanced\\',\\'/usr/bin/rctlcli\\',\\'/usr/bin/systemd-network\\',\\'/usr/bin/pamdicks\\',\\'/var/lib/cc\\'')|(cmd,in,'\\'/usr/bin/kaudited\\',\\'/usr/bin/kswaped\\',\\'/usr/bin/irqbalanced\\',\\'/usr/bin/rctlcli\\',\\'/usr/bin/systemd-network\\',\\'/usr/bin/pamdicks\\',\\'/var/lib/ [...]
-dipper.private.blink.rules&&&&express&&&&10473&&&&{"aesFlag":1,"varName":"procpath","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"HIpgKMyocjkX/1j+gc8nexpjKKR6jI54oY9hQywt8EQo5sKYu/rinv7k0hx+eDdKLPO8I9otp13VmNrFF3bVxp0jKFaOFExQjPmuFLvKrKao1WRbpxrHOQ/tVVLeHOo6694sseDu30Mx3oJUpWlX1RFz/985NDTzj4Y2pci90So8gYg+AOcUl8Rv8XCqg8aT"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10475&&&&{"aesFlag":1,"varName":"procpath","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"JTrRys0LzjfPNCaa582pLHItc8I71ObQdGcCqqo+wQI="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10474&&&&{"aesFlag":1,"varName":"cmd","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"HIpgKMyocjkX/1j+gc8nexpjKKR6jI54oY9hQywt8EQo5sKYu/rinv7k0hx+eDdKLPO8I9otp13VmNrFF3bVxp0jKFaOFExQjPmuFLvKrKao1WRbpxrHOQ/tVVLeHOo6694sseDu30Mx3oJUpWlX1RFz/985NDTzj4Y2pci90So8gYg+AOcUl8Rv8XCqg8aT"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10477&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10473\",\"10474\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10476&&&&{"aesFlag":1,"varName":"data_type","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"PCWwrcOjmeBbuLYPHG0JS3tTNU/U90AuQne3hzRtQio="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10479&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10477\",\"10478\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10478&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10475\",\"10476\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_blink_source_aegis_client_aegis_client_union_10008_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10008_script_10001","type":"script","value":"virus_name='pamdicks';\n___ACTION_JSON_10008=ACTION_JSON('2','name','truncate_and_lock_file','filepath','/usr/bin/pamdicks','2','name','tr [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink_source_aegis_client_aegis_client_union_10008_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10008_script_10002","type":"script","value":"retainField(traceid,procpath,create_time,pid,action_json_str,uuid,cwd,module_id,data_name,process_status,proc_exe_raw,procfs_fd,exec_info_ [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink_source_aegis_client_aegis_client_union_10009&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.FilterChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"names\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"subpipline_blink_source_aegis_client_aegis_client_union_10009_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\ [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_blink_source_aegis_client_aegis_client_union_10009_rule_10001&&&&{"expressionStr":"(procpath,==,'/usr/bin/bsd-port/getty')|(procpath,==,'/usr/bin/.sshd')","scriptNames":"[]","expressionName":"10482","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10009_rule_10001","ruleStatus":"0","type":"rule", [...]
-dipper.private.blink.rules&&&&express&&&&10480&&&&{"aesFlag":1,"varName":"procpath","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"U+cFpj3/fJdKAU2ArXG0NLYcbURKbp7Yon7RYKdYfQY="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10482&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10480\",\"10481\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10481&&&&{"aesFlag":1,"varName":"procpath","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"9DGpeFFf3c6C/q3li5wn/Q=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_blink_source_aegis_client_aegis_client_union_10009_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10009_script_10001","type":"script","value":"virus_name='getty';\n___concat_10001=concat('cp -f /usr/bin/dpkgd/ps /bin/ps','; cp -f /usr/bin/dpkgd/ps /usr/bin/ps','; cp -f /usr/bin/dp [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink_source_aegis_client_aegis_client_union_10009_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10009_script_10002","type":"script","value":"retainField(traceid,procpath,create_time,pid,action_json_str,uuid,cwd,module_id,data_name,process_status,proc_exe_raw,procfs_fd,exec_info_ [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink_source_aegis_client_aegis_client_union_10010&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.FilterChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"names\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"subpipline_blink_source_aegis_client_aegis_client_union_10010_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\ [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_blink_source_aegis_client_aegis_client_union_10010_rule_10001&&&&{"expressionStr":"procpath,regex,'/kinsing\\w*$'","scriptNames":"[]","expressionName":"10483","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10010_rule_10001","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","acti [...]
-dipper.private.blink.rules&&&&express&&&&10483&&&&{"aesFlag":1,"varName":"procpath","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"K2PO7IOFsF9fkS/f85etAA=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_blink_source_aegis_client_aegis_client_union_10010_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10010_script_10001","type":"script","value":"virus_name='kinsing1';\n___ACTION_JSON_10010=ACTION_JSON('3','name','kill_process','procpath','/tmp/kdevtmpfsi','lock_file','1','3','name' [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink_source_aegis_client_aegis_client_union_10010_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10010_script_10002","type":"script","value":"retainField(traceid,procpath,create_time,pid,action_json_str,uuid,cwd,module_id,data_name,process_status,proc_exe_raw,procfs_fd,exec_info_ [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink_source_aegis_client_aegis_client_union_10011&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.FilterChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"names\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"subpipline_blink_source_aegis_client_aegis_client_union_10011_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\ [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_blink_source_aegis_client_aegis_client_union_10011_rule_10001&&&&{"expressionStr":"procpath,==,'/tmp/kdevtmpfsi'","scriptNames":"[]","expressionName":"10484","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10011_rule_10001","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actio [...]
-dipper.private.blink.rules&&&&express&&&&10484&&&&{"aesFlag":1,"varName":"procpath","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"36099YLZTfSCzr3csLZN2A=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_blink_source_aegis_client_aegis_client_union_10011_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10011_script_10001","type":"script","value":"virus_name='kinsing2';\n___ACTION_JSON_10011=ACTION_JSON('3','name','kill_process','procpath','/tmp/kdevtmpfsi','lock_file','1','3','name' [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink_source_aegis_client_aegis_client_union_10011_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10011_script_10002","type":"script","value":"retainField(traceid,procpath,create_time,pid,action_json_str,uuid,cwd,module_id,data_name,process_status,proc_exe_raw,procfs_fd,exec_info_ [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink_source_aegis_client_aegis_client_union_10012&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.FilterChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"names\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"subpipline_blink_source_aegis_client_aegis_client_union_10012_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\ [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_blink_source_aegis_client_aegis_client_union_10012_rule_10001&&&&{"expressionStr":"procpath,regex,'^/(tmp|etc)/(sysguard|sysupdate|networkservice)$'","scriptNames":"[]","expressionName":"10485","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10012_rule_10001","ruleStatus":"0","type":"rule","vers [...]
-dipper.private.blink.rules&&&&express&&&&10485&&&&{"aesFlag":1,"varName":"procpath","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"4sP/o0gxAlw/Pxyze7Ge4Va7ITYujK84yjmaaOP2vEz/Nww2OfVYURkEKjFBfLcctvmFPQBAq4jbfkWBbxIojw=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_blink_source_aegis_client_aegis_client_union_10012_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10012_script_10001","type":"script","value":"virus_name='update';\n___ACTION_JSON_10012=ACTION_JSON('3','name','kill_process','procpath','/etc/sysguard','lock_file','1','3','name','ki [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink_source_aegis_client_aegis_client_union_10012_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10012_script_10002","type":"script","value":"retainField(traceid,procpath,create_time,pid,action_json_str,uuid,cwd,module_id,data_name,process_status,proc_exe_raw,procfs_fd,exec_info_ [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink_source_aegis_client_aegis_client_union_10013&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.FilterChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"names\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"subpipline_blink_source_aegis_client_aegis_client_union_10013_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\ [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_blink_source_aegis_client_aegis_client_union_10013_rule_10001&&&&{"expressionStr":"(cmd,like,'[ksoftirqd/0]%')&(procpath,like,'/%')","scriptNames":"[]","expressionName":"10488","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10013_rule_10001","ruleStatus":"0","type":"rule","version":"1.0","exten [...]
-dipper.private.blink.rules&&&&express&&&&10486&&&&{"aesFlag":1,"varName":"cmd","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"iyc8SOI2CL4N7qyg1Oyfjw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10488&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10486\",\"10487\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10487&&&&{"aesFlag":1,"varName":"procpath","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"M1psS2sL+qyjR6s0zjGxqw=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_blink_source_aegis_client_aegis_client_union_10013_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10013_script_10001","type":"script","value":"virus_name='ksoftirqd';\n___ACTION_JSON_10013=ACTION_JSON('3','name','kill_process','procpath','/etc/pam.d/zabbix_agent','lock_file','1',' [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink_source_aegis_client_aegis_client_union_10013_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10013_script_10002","type":"script","value":"retainField(traceid,procpath,create_time,pid,action_json_str,uuid,cwd,module_id,data_name,process_status,proc_exe_raw,procfs_fd,exec_info_ [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink_source_aegis_client_aegis_client_union_10014&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.FilterChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"names\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"subpipline_blink_source_aegis_client_aegis_client_union_10014_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\ [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_blink_source_aegis_client_aegis_client_union_10014_rule_10001&&&&{"expressionStr":"(procpath,==,'/etc/.vhost/netvhost')|(procpath,==,'/etc/.etcservice/linuxservice')","scriptNames":"[]","expressionName":"10491","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10014_rule_10001","ruleStatus":"0","t [...]
-dipper.private.blink.rules&&&&express&&&&10491&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10489\",\"10490\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10490&&&&{"aesFlag":1,"varName":"procpath","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"glBpTffqmEn1NiABsTOPtM8FOPOKFl1gr2ApTxWqzm0="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10489&&&&{"aesFlag":1,"varName":"procpath","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"7z3k388y892KZDrVE2hV2nvcQCxennFMBNyPOw2Ri04="}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_blink_source_aegis_client_aegis_client_union_10014_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10014_script_10001","type":"script","value":"virus_name='vhost';\n___ACTION_JSON_10014=ACTION_JSON('3','name','kill_process','procpath','/etc/.etcservice/linuxservice','lock_file','1' [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink_source_aegis_client_aegis_client_union_10014_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10014_script_10002","type":"script","value":"retainField(traceid,procpath,create_time,pid,action_json_str,uuid,cwd,module_id,data_name,process_status,proc_exe_raw,procfs_fd,exec_info_ [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink_source_aegis_client_aegis_client_union_10015&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"entityName\\\\\\\":\\\\\\\"script\\\\\\\",\\\\\\\"scriptName\\\\\\\":\\\\\\\"subpipline_blink_source_aegis_client_aegis_cl [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink_source_aegis_client_aegis_client_union_10015_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10015_script_10001","type":"script","value":"___lower_procpath_10009=lower(procpath);\n___lower_procpath_10010=lower(procpath);\n___lower_procpath_10011=lower(procpath);\n___lower_pro [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_blink_source_aegis_client_aegis_client_union_10015_rule_10001&&&&{"expressionStr":"(((___lower_procpath_10009,regex,'\\\\windows\\\\(syswow64|system32)\\\\drivers\\\\(svchost|taskmgr)\\.exe$')|(___lower_procpath_10010,regex,'\\\\windows\\\\(syswow64|system32)\\\\(wmiex|svhost)\\.exe$'))|(___lower_procpath_10011,regex,'\\\\windows\\\\temp\\\\(svchost|ttt)\\.exe$'))|(___lower_procpath_10012,==,'c:\\installed.exe')","scriptNames":"[]","expres [...]
-dipper.private.blink.rules&&&&express&&&&10493&&&&{"aesFlag":1,"varName":"___lower_procpath_10010","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"VPEtPZrTcTR05ttjrG+6M29pAzzfeRYW+z3vUGDGTj6fjQXttJfBbGpqmvOK8OuOqXxd9PCXE2mXeiBWQPEcZQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10492&&&&{"aesFlag":1,"varName":"___lower_procpath_10009","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"VPEtPZrTcTR05ttjrG+6M29pAzzfeRYW+z3vUGDGTj5uKWxzL2pROIxE1kThx2DJ9KFaVWBTOikSepQJhDbgZT1K20jlDLev+sDCZWnuoSE="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10495&&&&{"aesFlag":1,"varName":"___lower_procpath_10012","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"7K96DZJ+oabhNsXJy6/wnXb2w2h/79Nen3eyGN/yATk="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10494&&&&{"aesFlag":1,"varName":"___lower_procpath_10011","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"+bqqVvDs9ewuct9S+91tt68Erx9MHYjWflggrMsRuqGAlCbNFU7apzPpq6O84gdA"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10497&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10496\",\"10494\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10496&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10492\",\"10493\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10498&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10497\",\"10495\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_blink_source_aegis_client_aegis_client_union_10015_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10015_script_10002","type":"script","value":"virus_name='qudongrensheng';\n___ACTION_JSON_10015=ACTION_JSON('3','name','kill_process','procpath',procpath,'lock_file','1','2','name','d [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink_source_aegis_client_aegis_client_union_10015_script_10003&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10015_script_10003","type":"script","value":"retainField(traceid,procpath,create_time,pid,action_json_str,uuid,cwd,module_id,data_name,process_status,proc_exe_raw,procfs_fd,exec_info_ [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink_source_aegis_client_aegis_client_union_10016&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.FilterChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"names\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"subpipline_blink_source_aegis_client_aegis_client_union_10016_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\ [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_blink_source_aegis_client_aegis_client_union_10016_rule_10001&&&&{"expressionStr":"((((((((procpath,==,'/tmp/x64b')|(procpath,==,'/tmp/x32b'))|((cmd,==,'python ')&(procpath,==,'')))|(cmd,==,'/bin/bash /tmp/go'))|(procpath,regex,'^(/dev/shm/|/var/run/|/home/[^/]+/|/root/|/tmp/)dbusex$'))|(cmd,regex,'curl\\s.+?wget\\s.+?http://[\\w\\.\\d:]+/xms.+?\\|\\s*bash\\b.+echo\\s'))|(cmd,like,'%http://5.196.247.12/%'))|(cmd,like,'%http://198.98.57.2 [...]
-dipper.private.blink.rules&&&&express&&&&10503&&&&{"aesFlag":1,"varName":"cmd","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"XdhvFm5VMl3JbJHyrtXYiQc36/gyDbor9Vz7ULHp2rg="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10514&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10513\",\"10505\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10502&&&&{"aesFlag":1,"varName":"procpath","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":""}&&&&null
-dipper.private.blink.rules&&&&express&&&&10513&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10512\",\"10504\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10505&&&&{"aesFlag":1,"varName":"cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"bZOt4FZ8P2CRnl9EcyI7dXT01UKjIlvuQxQlQ7XRZb0UyuCBDXj3RXRUHK1/+lmix0tgY07KWny7X4gRn2lIIg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10516&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10515\",\"10507\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10504&&&&{"aesFlag":1,"varName":"procpath","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"8w6r8h0lFafedUbpPD6um9MpIqG8XRe6VvhleFxtwVcqVV8/NHBkOonKlo48o5KEkal6ndokDe2CJqKNhDPGVA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10515&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10514\",\"10506\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10507&&&&{"aesFlag":1,"varName":"cmd","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"k4Hcqn1H9CyrXRfk5XZLKMj3WZRnk3tHf26lVxjxet0="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10506&&&&{"aesFlag":1,"varName":"cmd","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"FFNB320gPfSnaI+74L1/o+W7EhrtOECKfHmk6txzyvA="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10517&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10516\",\"10508\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10509&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10499\",\"10500\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10508&&&&{"aesFlag":1,"varName":"cmd","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"ATBQXyDX23wLSQk7xmuQ02Z20dL4jVz7SafrWvDq+6w="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10499&&&&{"aesFlag":1,"varName":"procpath","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"mKeuL1cbS+bmGA0NJdNOUQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10510&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10501\",\"10502\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10501&&&&{"aesFlag":1,"varName":"cmd","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"xC6yS//Tp5Lm7ZynRuL3MA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10512&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10511\",\"10503\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10500&&&&{"aesFlag":1,"varName":"procpath","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"PyVWxHTv8xgKdzDgdVK+Yw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10511&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10509\",\"10510\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_blink_source_aegis_client_aegis_client_union_10016_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10016_script_10001","type":"script","value":"virus_name='dbusex';\n___ACTION_JSON_10016=ACTION_JSON('2','name','exec_command','cmd','chattr -ia -R /etc/cron.* ; chattr -ia -R /var/spo [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink_source_aegis_client_aegis_client_union_10016_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10016_script_10002","type":"script","value":"retainField(traceid,procpath,create_time,pid,action_json_str,uuid,cwd,module_id,data_name,process_status,proc_exe_raw,procfs_fd,exec_info_ [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink_source_aegis_client_aegis_client_union_10017&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.FilterChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"names\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"subpipline_blink_source_aegis_client_aegis_client_union_10017_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\ [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_blink_source_aegis_client_aegis_client_union_10017_rule_10001&&&&{"expressionStr":"(cmd,regex,'.+\\s{9,}$')&(procpath,regex,'^/usr/bin/[a-z]{10}$')","scriptNames":"[]","expressionName":"10520","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10017_rule_10001","ruleStatus":"0","type":"rule","versi [...]
-dipper.private.blink.rules&&&&express&&&&10518&&&&{"aesFlag":1,"varName":"cmd","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"H5GlG6Bg+t3LK6Su0vyvoA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10519&&&&{"aesFlag":1,"varName":"procpath","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"XWm/XonJ1inMbJDH1eEQukKv+/8rzpMqz6ml9x6k+gI="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10520&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10518\",\"10519\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_blink_source_aegis_client_aegis_client_union_10017_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10017_script_10001","type":"script","value":"virus_name='usr_bin_changing';\n___ACTION_JSON_10017=ACTION_JSON('2','name','exec_command','cmd','\n for pid in `ls /proc/` \n [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink_source_aegis_client_aegis_client_union_10017_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink_source_aegis_client_aegis_client_union_10017_script_10002","type":"script","value":"retainField(traceid,procpath,create_time,pid,action_json_str,uuid,cwd,module_id,data_name,process_status,proc_exe_raw,procfs_fd,exec_info_ [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_client_aegis_client_rule_10021&&&&{"expressionStr":"virus_name,in,'\\'pty3\\',\\'pamdicks\\',\\'getty\\',\\'kinsing1\\',\\'kinsing2\\',\\'update\\',\\'ksoftirqd\\',\\'vhost\\',\\'20200819_dev_shm\\',\\'general_linux_kill_process\\',\\'general_linux_kill_and_lock\\',\\'conhou\\',\\'SXS\\',\\'kill_process_with_pid\\',\\'qudongrensheng\\',\\'dbusex\\''","scriptNames":"[]","expressionName":"10521","varNames":"[]","className":"com.aliyu [...]
-dipper.private.blink.rules&&&&express&&&&10521&&&&{"aesFlag":1,"varName":"virus_name","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"8GCrWbqPt2TLsVAelAldIWw7m/fSeYhSmt+Q7C7tDHuvrn7g3TOWneTfzx7OTfjtrxF2CmIJuot3xJrhbmIxijAUc8eIP6029nsQ5XBNwqlwljknzTgqqrf59XnGDyAtU/2a74zlhga2gLiZ4kqhbXpz5Kv+xcBVilVtqb1fHmAQbicS++MiBNZoxlxmHqFoMQ82Zukwwx/sfTR/7ECaK1xp8NB3xkNf9Xopq1 [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10105&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10105","type":"script","value":"ali_uid='';\nclient_ip='';\nevent_type='持久化后门';\nevent_name='异常进程(体检)';\nlevel='high';\n___cast_10096=cast(uuid,'string');\n___cast_10097=cast(logtime,'string');\n___cast_10098=cast(cm [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10106&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10106","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_client_aegis_client_channel_10010&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink_source_aegis_client_aegis_client_channel_10010","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeInOrder":"true","maxFetchLogGroupSize":"100","syn [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_client_aegis_client_rule_10022&&&&{"expressionStr":"bin_path,regex,'^/(tmp/|bin/|usr/bin/)(initdr|bprofr|dbused|dbusex|xms|sysdr|crondr)$'","scriptNames":"[]","expressionName":"10522","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_rule_10022","ruleStatus":"0","type":"rule","version":"1.0","extendField": [...]
-dipper.private.blink.rules&&&&express&&&&10522&&&&{"aesFlag":1,"varName":"bin_path","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"+RWJU6SvYfw4fXHCca19b+wmqyn5a2yxxPxYMKBm6Cu0wiifpqHjXcN0pClCIF3JrfQxwpFFRQTgZSrgcPLBYiAdb08yNdYwWUcZ9SIbRiI="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10107&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10107","type":"script","value":"___ACTION_JSON_10018=ACTION_JSON('2','name','truncate_and_lock_file','filepath',bin_path,'2','name','truncate_and_lock_file','filepath','/usr/bin/initdr','2','name','truncate_and_lock_ [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10108&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10108","type":"script","value":"retainField(traceid,msg,module_id,data_name,exec_info_data,data_type,bin_path,aliuid,cmd,action_json_str,logtime,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10109&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10109","type":"script","value":"ali_uid='';\nclient_ip='';\nevent_type='持久化后门';\nevent_name='异常计划任务(体检)';\nlevel='high';\n___cast_10108=cast(uuid,'string');\n___cast_10109=cast(logtime,'string');\n___cast_10110=cast( [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10110&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10110","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_client_aegis_client_channel_10011&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink_source_aegis_client_aegis_client_channel_10011","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeInOrder":"true","maxFetchLogGroupSize":"100","syn [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10111&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10111","type":"script","value":"ali_uid='';\nclient_ip='';\nevent_type='持久化后门';\nevent_name='Rootkit后门(体检)';\nlevel='high';\n___cast_10116=cast(uuid,'string');\n___cast_10117=cast(logtime,'string');\n___cast_10118=ca [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10112&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10112","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_client_aegis_client_channel_10012&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink_source_aegis_client_aegis_client_channel_10012","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeInOrder":"true","maxFetchLogGroupSize":"100","syn [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10113&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10113","type":"script","value":"ali_uid='';\nclient_ip='';\nevent_type='持久化后门';\nevent_name='隐藏的内核模块';\nlevel='high';\n___cast_10129=cast(uuid,'string');\n___cast_10130=cast(raw_buffer,'string');\n___cast_10131=cast( [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10114&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10114","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_client_aegis_client_channel_10013&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink_source_aegis_client_aegis_client_channel_10013","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeInOrder":"true","maxFetchLogGroupSize":"100","syn [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10115&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10115","type":"script","value":"ali_uid='';\nclient_ip='';\nevent_type='持久化后门';\nevent_name='异常计划任务(体检)';\nlevel='high';\n___cast_10134=cast(uuid,'string');\n___cast_10135=cast(logtime,'string');\n___cast_10136=cast( [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_client_aegis_client_script_10116&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_client_aegis_client_script_10116","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_client_aegis_client_channel_10014&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink_source_aegis_client_aegis_client_channel_10014","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeInOrder":"true","maxFetchLogGroupSize":"100","syn [...]
-dipper.private.blink.rules&&&&pipline&&&&blink.source.aegis.client&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.SubPiplineChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"nextStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"filterMsgSwitch\\\\\\\":\\\\\\\"blink.source.aegis.client.filter.switch\\\\\\\",\\\\\\\"cancelAfterConfigurableRefreshListerner\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"className\\\\\\\":\\\\\\\"com.aliyun.yund [...]
-dipper.private.blink.rules&&&&channel&&&&blink.source.aegis.client;channel&&&&{"offset":"0","isJsonData":"true","maxThread":"12","concurrentCount":"12","className":"com.aliyun.yundun.dipper.channel.adapter.MetaqChannel","configureName":"blink.source.aegis.client;channel","type":"channel","version":"1.0","timeout":"30000","activtyTimeOut":"3000","tags":"blink.source.aegis.client.tags","syncTimeout":"5000","groupName":"blink.source.aegis.client.group","isBatchMessage":"true","isAutoFlush": [...]
-dipper.private.blink.rules&&&&pipline&&&&blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"nextStageLables\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"cancelAfterConfigurableRefreshListerner\\\\\\\":\\\\\\\"false\\\\\\\",\ [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert;json_concat_10002&&&&{"fullClassName":"com.aliyun.sec.lyra.udf.ext.JsonConcat","initMethodName":"open","functionName":"json_concat","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert;json_concat_10002","type":"scri [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert;salt_hash_10001&&&&{"fullClassName":"com.aliyun.sec.lyra.hsh.udf.ext.SaltHash","initMethodName":"open","functionName":"salt_hash","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert;salt_hash_10001","type":"script", [...]
-dipper.private.blink.rules&&&&channel&&&&blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert&&&&{"isJsonData":"true","project":"ali-beaver-net-log","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert","type":"channel","timeout":"30000","accessId":"replace_accessId_sls","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeInOrder":"true","maxF [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert_script_10001","type":"script","value":"___!null_10008=!null(qname);\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert_rule_10001&&&&{"expressionStr":"___!null_10008&(qname,!in,'\\'fget-career.com\\',\\'tv.2345.com\\',\\'img.sobot.com\\',\\'rocksat.ddns.net\\'')","scriptNames":"[]","expressionName":"10524","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert_ru [...]
-dipper.private.blink.rules&&&&express&&&&10524&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"___!null_10008\",\"10523\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10523&&&&{"aesFlag":1,"varName":"qname","functionName":"!in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"hNKGAVqoWpMgEUKB7jaj4md1Tb4nBp1l7iFQTsSKN1OZKMy6uPZv+x8c8qvsoQ1srCCsGvGbprGzr/X3rE9nMb/xES0OKcksBBITIzSLe0c="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert_script_10002","type":"script","value":"___REGEXP_EXTRACT_10001=REGEXP_EXTRACT(answer,'([0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3})',1);\nmalicious_ip=___REGEXP_EXTRACT_10 [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert_script_10003&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert_script_10003","type":"script","value":"retainField(query_name,dns_host,client_ip,malicious_ip,query_time,tunnel);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert_script_10004&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert_script_10004","type":"script","value":"___len_10002=len(malicious_ip);\n___null_10011=null(malicious_ip);\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert_rule_10002&&&&{"expressionStr":"((malicious_ip,regex,'^([0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3})$')&(___len_10002,>,double,8))|___null_10011","scriptNames":"[]","expressionName":"10528","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_a [...]
-dipper.private.blink.rules&&&&express&&&&10525&&&&{"aesFlag":1,"varName":"malicious_ip","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"o+lYaT32Ycuad4ukPH0R75ILq6S5FS30z+RNbKvU85d1yM/FEqcwYQt+Q9ukp9xnAqsb9WZyTY3E1yicqpL7NQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10527&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10525\",\"10526\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10526&&&&{"varName":"___len_10002","functionName":">","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"8.0"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10528&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10527\",\"___null_10011\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert_script_10005&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert_script_10005","type":"script","value":"retainField(query_name,dns_host,client_ip,malicious_ip,query_time,tunnel);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert_script_10006&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert_script_10006","type":"script","value":"___SALT_HASH_10001=SALT_HASH(query_name);;","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&dataSource&&&&blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert_dataSource_10001&&&&{"className":"com.aliyun.yundun.dipper.configurable.http.resource.JDBCDataSource","configureName":"blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert_dataSource_10001","userName":"xxxxxxxxxx","type":"dataSource","version":"1.0","url":"intelligence.rds.jdbc.url","timeout":"30000","activtyTimeOut":"3000","password":"xxxxxxxxx","isAutoFlush":"false","outpu [...]
-dipper.private.blink.rules&&&&intelligence&&&&blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert_intelligence_10001&&&&{"className":"com.aliyun.filter.intelligence.DomainIntelligenceCache","pollingTimeMintue":"30","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert_intelligence_10001","idFieldName":"id","batchSize":"3000","datasourceName":"blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert_dataSourc [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert_script_10007&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert_script_10007","type":"script","value":"intelligence('dipper.private.blink.rules','blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert_intelligence_10001',___SALT_HASH_ [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert_script_10008&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert_script_10008","type":"script","value":"aliuid='';\nuuid='';\n___concat_10002=concat('aliyun_dns : ',dns_host);\ndns_server=___concat_10002;rm('___concat_10002');\ndns_port='53'; [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert_script_10009&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert_script_10009","type":"script","value":"is_c2=p.is_c2;\nis_malicious_source=p.is_malicious_source;\nis_mining_pool=p.is_mining_pool;\nis_phishing=p.is_phishing;\nretainField(dns_ [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert_script_10010&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert_script_10010","type":"script","value":"retainField(is_malicious_source,is_mining_pool,query_name,dns_server,dns_port,is_c2,client_ip,malicious_ip,query_time,tunnel,is_phishing); [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert_script_10011&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert_script_10011","type":"script","value":"aliuid='';\nuuid='';\ndst_port='';\nproc_path='';\ncmdline='';\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert_script_10012&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert_script_10012","type":"script","value":"retainField(proc_path,dns_server,uuid,is_malicious_source,cmdline,is_mining_pool,query_name,dns_port,dst_port,is_c2,client_ip,aliuid,malic [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert_script_10013&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert_script_10013","type":"script","value":"ali_uid=aliuid;\nevent_type='威胁情报';\n___in_10001=contain(is_malicious_source,'1')\n;___in_10002=contain(is_phishing,'1')\n;___in_10003=con [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert_script_10014&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert_script_10014","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&channel&&&&blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert_channel_10001&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink_source_bevear_dns_adl_sas_apsara_intelligence_dns_alert_channel_10001","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeInO [...]
-dipper.private.blink.rules&&&&pipline&&&&blink.source.bevear.dns&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.SubPiplineChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"nextStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"filterMsgSwitch\\\\\\\":\\\\\\\"blink.source.bevear.dns.filter.switch\\\\\\\",\\\\\\\"cancelAfterConfigurableRefreshListerner\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"className\\\\\\\":\\\\\\\"com.aliyun.yundun.d [...]
-dipper.private.blink.rules&&&&channel&&&&blink.source.bevear.dns;channel&&&&{"isJsonData":"true","project":"blink.source.bevear.dns.project","concurrentCount":"4","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink.source.bevear.dns;channel","type":"channel","timeout":"30000","accessId":"blink.source.bevear.dns.accessId","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeInOrder":"true","maxFetchLogGroupSize":"100","syncCount" [...]
-dipper.private.blink.rules&&&&pipline&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"nextStageLables\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"cancelAfterConfigurableRefreshListerner\\\\\\\":\\\\\\\ [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation;json_concat_10003&&&&{"fullClassName":"com.aliyun.sec.lyra.udf.ext.JsonConcat","initMethodName":"open","functionName":"json_concat","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation;json_concat_100 [...]
-dipper.private.blink.rules&&&&channel&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeInOrder":"true" [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_script_10001","type":"script","value":"___lower_uid_name_10001=lower(uid_name);\n___lower_file_name_10001=lower(file_name);\n___!null_10009=!null(cmdline);\n___b [...]
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_rule_10001&&&&{"expressionStr":"((((((dockercontainerid,==,'N/A')&((___lower_uid_name_10001,==,'root')|(uid,==,double,0)))&(___lower_file_name_10001,in,'\\'bash\\',\\'sh\\',\\'dash\\',\\'ash\\',\\'tcsh\\',\\'csh\\',\\'ksh\\',\\'zsh\\''))&___!null_10009)&(cmdline,!in,'\\'N/A\\',\\'\\''))&(((___blink_instr_10001,==,double,0)|((___lower_cmdline_10002,regex,'^[a-z\\/]+\\s+-c')&(___reve [...]
-dipper.private.blink.rules&&&&express&&&&10536&&&&{"aesFlag":1,"varName":"___reverse_10002","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"BBydeM9HIhzVgDNQknyTW2YqZypw1YMRsPuo4g1fQFsA+K07lbLjD4VKjxg4fnCylP7M7H1sAtd2sn2NNJf7Bw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10535&&&&{"aesFlag":1,"varName":"___lower_cmdline_10002","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"eJuXezsg41SClgVzsqU6rQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10557&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10551\",\"10556\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10538&&&&{"aesFlag":1,"varName":"dockercontainerid","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"M7grO0DWTNleGtk17FTQSw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10537&&&&{"aesFlag":1,"varName":"cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"Zco97o6UrrSZ0elsH7Q0Z81dTm4v+WG9f5wJTk8gquPLsPdY/WyRBliLUr69uA61Xk2v4lpU+OCt+L4ruCbfwA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10539&&&&{"aesFlag":1,"varName":"uid_name","functionName":"!in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"QhO8c5+E1GcoCgDIqz67dw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10550&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10549\",\"10537\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10530&&&&{"aesFlag":1,"varName":"___lower_uid_name_10001","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"AYWKBDw0xxoXMPJIP1jDuQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10552&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10538\",\"10539\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10551&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10547\",\"10550\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10532&&&&{"aesFlag":1,"varName":"___lower_file_name_10001","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"QEp6CiJU27OLzwRtBGXIEhVTK6eSTiKR31dK4XVFkz9u3lRpcCT2PS6e+eCuoy5slti7BVH9KFO+ljo720o5lg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10554&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10553\",\"10540\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10531&&&&{"varName":"uid","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0.0"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10553&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10552\",\"___!null_10010\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10534&&&&{"varName":"___blink_instr_10001","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0.0"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10556&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10555\",\"10542\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10533&&&&{"aesFlag":1,"varName":"cmdline","functionName":"!in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"QhO8c5+E1GcoCgDIqz67dw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10555&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10554\",\"10541\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10547&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10546\",\"10533\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10546&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10545\",\"___!null_10009\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10549&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10534\",\"10548\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10548&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10535\",\"10536\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10529&&&&{"aesFlag":1,"varName":"dockercontainerid","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"M7grO0DWTNleGtk17FTQSw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10541&&&&{"aesFlag":1,"varName":"uid","functionName":"<>","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"4ROl+e4g5JZAI9EFfD9CUQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10540&&&&{"aesFlag":1,"varName":"uid_name","functionName":"<>","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"AYWKBDw0xxoXMPJIP1jDuQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10543&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10530\",\"10531\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10542&&&&{"aesFlag":1,"varName":"file_name","functionName":"!in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"h5esfc211d7HPaABmIo2RKX9LiQVTyONPcxb+6/EuuG8oPglo1aUxNrKM6Yoh3/5wwMi6keLnLkGByygcyh+fCA1eR4snN/3ZB4Mnw9vQpL9TWWxK3TldEkn8PlDVWUjNfmNfHXj+wyWvcStCeAIFjqZyWLjH02Upvv7yFRD+KPtht6iL3RIMp60WvTGOUOF7LgAh+NFz9yg9h1sN5PHTlUui0WvS7KCZ6oEgQ [...]
-dipper.private.blink.rules&&&&express&&&&10545&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10544\",\"10532\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10544&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10529\",\"10543\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_script_10002","type":"script","value":"retainField(k8spodname,file_path,dockerimageid,gid,perm,pid,k8snodeid,host_uuid,file_gid_name,sid,uid,file_gid,k8snodename [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_script_10003&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_script_10003","type":"script","value":"___lower_uid_name_10002=lower(uid_name);\n___lower_file_name_10002=lower(file_name);\n___!null_10011=!null(cmdline);\n___b [...]
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_rule_10002&&&&{"expressionStr":"(((((dockercontainerid,==,'N/A')&((___lower_uid_name_10002,==,'root')|(uid,==,double,0)))&(___lower_file_name_10002,in,'\\'bash\\',\\'sh\\',\\'dash\\',\\'ash\\',\\'tcsh\\',\\'csh\\',\\'ksh\\',\\'zsh\\''))&___!null_10011)&(cmdline,!in,'\\'N/A\\',\\'\\''))&(((___blink_instr_10002,==,double,0)|((___lower_cmdline_10003,regex,'^[a-z\\/]+\\s+-c')&(___rever [...]
-dipper.private.blink.rules&&&&express&&&&10570&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10569\",\"___!null_10011\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10558&&&&{"aesFlag":1,"varName":"dockercontainerid","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"M7grO0DWTNleGtk17FTQSw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10569&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10568\",\"10561\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10568&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10558\",\"10567\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10559&&&&{"aesFlag":1,"varName":"___lower_uid_name_10002","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"AYWKBDw0xxoXMPJIP1jDuQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10561&&&&{"aesFlag":1,"varName":"___lower_file_name_10002","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"QEp6CiJU27OLzwRtBGXIEhVTK6eSTiKR31dK4XVFkz9u3lRpcCT2PS6e+eCuoy5slti7BVH9KFO+ljo720o5lg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10572&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10564\",\"10565\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10560&&&&{"varName":"uid","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0.0"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10571&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10570\",\"10562\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10563&&&&{"varName":"___blink_instr_10002","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0.0"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10574&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10573\",\"10566\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10562&&&&{"aesFlag":1,"varName":"cmdline","functionName":"!in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"QhO8c5+E1GcoCgDIqz67dw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10573&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10563\",\"10572\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10565&&&&{"aesFlag":1,"varName":"___reverse_10004","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"BBydeM9HIhzVgDNQknyTW2YqZypw1YMRsPuo4g1fQFsA+K07lbLjD4VKjxg4fnCylP7M7H1sAtd2sn2NNJf7Bw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10564&&&&{"aesFlag":1,"varName":"___lower_cmdline_10003","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"eJuXezsg41SClgVzsqU6rQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10575&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10571\",\"10574\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10567&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10559\",\"10560\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10566&&&&{"aesFlag":1,"varName":"cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"Zco97o6UrrSZ0elsH7Q0Z81dTm4v+WG9f5wJTk8gquPLsPdY/WyRBliLUr69uA61Xk2v4lpU+OCt+L4ruCbfwA=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_script_10004&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_script_10004","type":"script","value":"filename=file_name;\nfilepath=file_path;\npcmdline=parent_cmd_line;\npfilename=parent_file_name;\npfilepath=parent_file_pa [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_script_10005&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_script_10005","type":"script","value":"retainField(k8spodname,dockerimageid,perm,pid,k8snodeid,host_uuid,sid,uid,k8snodename,cmdline,pfilename,filepath,time_win, [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_script_10006&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_script_10006","type":"script","value":"___!null_10012=!null(uid_name);\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_rule_10003&&&&{"expressionStr":"(((((dockercontainerid,==,'N/A')&(uid_name,!in,'\\'N/A\\',\\'\\''))&___!null_10012)&(uid_name,<>,'root'))&(uid,<>,'0'))&(file_name,!in,'\\'sudo\\',\\'su\\',\\'dzdo\\',\\'expr\\',\\'modprobe\\',\\'systemd-cgroups-agent\\',\\'java\\',\\'php\\',\\'php-fpm\\',\\'python\\',\\'N/A\\',\\'grep\\',\\'kmod\\',\\'busybox\\',\\'rm\\',\\'conhost.exe\\',\\'date\\' [...]
-dipper.private.blink.rules&&&&express&&&&10579&&&&{"aesFlag":1,"varName":"uid","functionName":"<>","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"4ROl+e4g5JZAI9EFfD9CUQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10581&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10576\",\"10577\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10580&&&&{"aesFlag":1,"varName":"file_name","functionName":"!in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"h5esfc211d7HPaABmIo2RKX9LiQVTyONPcxb+6/EuuG8oPglo1aUxNrKM6Yoh3/5wwMi6keLnLkGByygcyh+fCA1eR4snN/3ZB4Mnw9vQpL9TWWxK3TldEkn8PlDVWUjNfmNfHXj+wyWvcStCeAIFjqZyWLjH02Upvv7yFRD+KPtht6iL3RIMp60WvTGOUOF7LgAh+NFz9yg9h1sN5PHTlUui0WvS7KCZ6oEgQ [...]
-dipper.private.blink.rules&&&&express&&&&10583&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10582\",\"10578\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10582&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10581\",\"___!null_10012\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10585&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10584\",\"10580\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10584&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10583\",\"10579\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10576&&&&{"aesFlag":1,"varName":"dockercontainerid","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"M7grO0DWTNleGtk17FTQSw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10578&&&&{"aesFlag":1,"varName":"uid_name","functionName":"<>","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"AYWKBDw0xxoXMPJIP1jDuQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10577&&&&{"aesFlag":1,"varName":"uid_name","functionName":"!in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"QhO8c5+E1GcoCgDIqz67dw=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_script_10007&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_script_10007","type":"script","value":"___unixtime_10002=unixtime(scan_time,'yyyy-MM-dd HH:mm:ss');\n___division_10002=division(___unixtime_10002,3600);\n___floo [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_script_10008&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_script_10008","type":"script","value":"retainField(uid_name,file_path,cmdline,perm,time_win,pid,parent_file_path,parent_cmd_line,scan_time,host_uuid,sid,ppid);", [...]
-dipper.private.blink.rules&&&&window&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_window_10001&&&&{"groupByFieldName":"host_uuid;cmdline;filename;filepath;pcmdline;pfilename;pfilepath;perm;pid;pid_start_time;ppid;scan_time;sid;tty;uid;uid_name;docker_file_path;dockercontainerid;dockerimageid;dockerimagename;k8sclusterid;k8snamespace;k8snodeid;k8snodename;k8spodname;time_win","waterMarkMinute":"0","sizeAdjust":"0","className":"com.aliyun.yundun.dipper.window.mo [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_script_10009&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_script_10009","type":"script","value":"retainField(k8spodname,dockerimageid,perm,pid,k8snodeid,host_uuid,sid,uid,k8snodename,cmdline,pfilename,filepath,time_win, [...]
-dipper.private.blink.rules&&&&window&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_join_window_10001&&&&{"waterMarkMinute":"0","className":"com.aliyun.yundun.dipper.window.model.JoinWindow","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_join_window_10001","type":"window","retainWindowCount":"6","windowType":"tumble","timeout":"30000","sizeInterval":"8","isAutoFlush":"false","nameSpace":"dipper.private.blink.rules","having [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_join_left_10001&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.WindowChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"entityName\\\\\\\":\\\\\\\"window\\\\\\\",\\\\\\\"lable\\\\\\\":\\\\\\\"blink.source.ae [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_join_right_10001&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"entityName\\\\\\\":\\\\\\\"script\\\\\\\",\\\\\\\"scriptName\\\\\\\":\\\\\\\"subpiplin [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_join_right_10001_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_join_right_10001_script_10001","type":"script","value":"retainField(uid_name,file_path,cmdline,perm,time_win,pid,parent_fi [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_script_10010&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_script_10010","type":"script","value":"___unixtime_10003=unixtime(scan_time);\n___unixtime_10004=unixtime(parent.scan_time);\n___subtraction_10001=subtraction(__ [...]
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_rule_10004&&&&{"expressionStr":"___abs_10001,<,double,15","scriptNames":"[]","expressionName":"10586","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_rule_10004","ruleStatus":"0","type":"rule","version":"1.0","extendField": [...]
-dipper.private.blink.rules&&&&express&&&&10586&&&&{"varName":"___abs_10001","functionName":"<","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"15.0"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_script_10011&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_script_10011","type":"script","value":"pperm=parent.perm;\npusername=parent.uid_name;\npppid=parent.ppid;\npppfilepath=parent.parent_file_path;\npppcmdline=paren [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_script_10012&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_script_10012","type":"script","value":"scan_time=parent.scan_time;\nsid=parent.sid;\npid=parent.pid;\ntime_win=parent.time_win;\ncmdline=parent.cmdline;\nhost_uu [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_script_10013&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_script_10013","type":"script","value":"___regexp_10001=regex(cmdline,'virustotal');\n___compare_10012=equals(___regexp_10001,true);\nif(___compare_10012){___case [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_script_10014&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_script_10014","type":"script","value":"retainField(is_white,k8spodname,file_path,dockerimageid,pppfilepath,perm,pid,k8snodeid,host_uuid,sid,uid,k8snodename,cmdli [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_script_10015&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_script_10015","type":"script","value":"ali_uid='';\nclient_ip='';\nevent_type='进程异常行为';\nevent_name='疑似权限提升';\nlevel='high';\n___cast_10154=cast(host_uuid,'strin [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_script_10016&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_script_10016","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,host_uuid);","version":"1.0" [...]
-dipper.private.blink.rules&&&&channel&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_channel_10001&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_privilege_escalation_channel_10001","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"fa [...]
-dipper.private.blink.rules&&&&pipline&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"nextStageLables\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"cancelAfterConfigurableRefreshListerner\\\\\\\":\\\\\\\"false\\\\\\\" [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature;sas_black_rule_v2_10001&&&&{"fullClassName":"com.lyra.xs.udf.ext.sas_black_rule_v2","initMethodName":"open","functionName":"sas_black_rule_v2","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature;sas_black_rule_v2_ [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature;json_concat_10004&&&&{"fullClassName":"com.aliyun.sec.lyra.udf.ext.JsonConcat","initMethodName":"open","functionName":"json_concat","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature;json_concat_10004","type":"sc [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature;b64_auto_10001&&&&{"fullClassName":"com.lyra.xs.udf.ext.b64_auto","initMethodName":"open","functionName":"b64_auto","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature;b64_auto_10001","type":"script","version":"1. [...]
-dipper.private.blink.rules&&&&channel&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature&&&&{"offset":"0","isJsonData":"true","maxThread":"12","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.MetaqChannel","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature","type":"channel","version":"1.0","timeout":"30000","activtyTimeOut":"3000","syncTimeout":"5000","groupName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_black [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature_script_10001","type":"script","value":"___!null_10013=!null(cmdline);\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature_rule_10001&&&&{"expressionStr":"___!null_10013&(cmdline,<>,'')","scriptNames":"[]","expressionName":"10588","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature_rule_10001","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","ac [...]
-dipper.private.blink.rules&&&&express&&&&10587&&&&{"aesFlag":1,"varName":"cmdline","functionName":"<>","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"wc8fi0p7qwBHnY1x2c6Qsw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10588&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"___!null_10013\",\"10587\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature_script_10002","type":"script","value":"uuid=host_uuid;\nfilename=file_name;\npfilename=parent_file_name;\npcmdline=parent_cmd_line;\nfilepath=file_path;\npfilepath=parent_file [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature_script_10003&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature_script_10003","type":"script","value":"retainField(k8spodname,dockerimageid,dockercontainerid,dockerimagename,pid,k8snodeid,k8sclusterid,uuid,ppid,sid,uid_name,b64_cmdline,k8s [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature_script_10004&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature_script_10004","type":"script","value":"retainField(k8spodname,dockerimageid,dockercontainerid,dockerimagename,pid,k8snodeid,k8sclusterid,uuid,ppid,sid,uid_name,b64_cmdline,k8s [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature_script_10005&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature_script_10005","type":"script","value":"___lower_cmdline_10004=lower(cmdline);\n___REGEXP_REPLACE_10006=REGEXP_REPLACE(___lower_cmdline_10004,'\\s+',' ');\nstd_cmdline=___REGEX [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature_script_10006&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature_script_10006","type":"script","value":"retainField(k8spodname,dockerimageid,std_filepath,pid,k8snodeid,uuid,std_pfilename,sid,k8snodename,cmdline,pfilename,filepath,std_filena [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature_script_10007&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature_script_10007","type":"script","value":"___SAS_BLACK_RULE_V2_10001=SAS_BLACK_RULE_V2(257,1,'std_cmd_line',std_cmdline,'cmd_line',cmdline,'cmdline',cmdline,'filename',filename,' [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature_script_10008&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature_script_10008","type":"script","value":"retainField(k8spodname,dockerimageid,std_filepath,pid,k8snodeid,uuid,std_pfilename,sid,k8snodename,cmdline,pfilename,filepath,std_filena [...]
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature_rule_10002&&&&{"expressionStr":"(sas_black_rule_ob_result,>,double,0)|(sas_black_rule_online_result,>,double,0)","scriptNames":"[]","expressionName":"10591","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature_rule_10002","ruleStatus":"0","ty [...]
-dipper.private.blink.rules&&&&express&&&&10590&&&&{"varName":"sas_black_rule_online_result","functionName":">","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0.0"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10591&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10589\",\"10590\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10589&&&&{"varName":"sas_black_rule_ob_result","functionName":">","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0.0"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature_script_10009&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature_script_10009","type":"script","value":"retainField(k8spodname,dockerimageid,std_filepath,pid,k8snodeid,uuid,std_pfilename,sid,k8snodename,cmdline,pfilename,filepath,std_filena [...]
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature_rule_10003&&&&{"expressionStr":"((sas_black_rule_online_result,>,double,0)|(sas_black_rule_ob_result,in,'954,956'))|(((sas_black_rule_ob_result,in,'919')&(filename,in,'\\'ssh\\''))|((sas_black_rule_ob_result,>=,double,972)&(sas_black_rule_ob_result,<=,double,1003)))","scriptNames":"[]","expressionName":"10602","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor [...]
-dipper.private.blink.rules&&&&express&&&&10602&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10598\",\"10601\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10601&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10599\",\"10600\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10592&&&&{"varName":"sas_black_rule_online_result","functionName":">","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0.0"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10594&&&&{"aesFlag":1,"varName":"sas_black_rule_ob_result","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"yMqD2bBfahlBo0PPn2vLJg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10593&&&&{"aesFlag":1,"varName":"sas_black_rule_ob_result","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"x41L6PfNXwd5JNipSA5WCw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10596&&&&{"varName":"sas_black_rule_ob_result","functionName":">=","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"972.0"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10595&&&&{"aesFlag":1,"varName":"filename","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"qVChOq21zoNjgEZ6Rf+lqQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10598&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10592\",\"10593\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10597&&&&{"varName":"sas_black_rule_ob_result","functionName":"<=","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"1003.0"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10600&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10596\",\"10597\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10599&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10594\",\"10595\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature_script_10010&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature_script_10010","type":"script","value":"___in_10005=contain(sas_black_rule_online_result,602,603,618)\n;___REGEXP_REPLACE_10012=REGEXP_REPLACE(std_filename,'([^\\\\\\:\\>\\-\\& [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature_script_10011&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature_script_10011","type":"script","value":"retainField(k8spodname,dockerimageid,k8snodename,docker_file_path,dockercontainerid,dockerimagename,k8spodname,dockerimageid,std_filepat [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature_script_10012&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature_script_10012","type":"script","value":"___md5_10004=md5(abk_raw);\nabk=___md5_10004;rm('___md5_10004');\n___regexp_10002=regex(std_cmdline,'virustotal');\n___compare_10014=equ [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature_script_10013&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature_script_10013","type":"script","value":"retainField(is_white,k8spodname,dockerimageid,std_filepath,pid,k8snodeid,uuid,std_pfilename,sid,k8snodename,cmdline,pfilename,filepath,s [...]
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature_rule_10004&&&&{"expressionStr":"is_white,==,double,0","scriptNames":"[]","expressionName":"10603","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature_rule_10004","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames" [...]
-dipper.private.blink.rules&&&&express&&&&10603&&&&{"varName":"is_white","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0.0"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature_script_10014&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature_script_10014","type":"script","value":"retainField(k8spodname,dockerimageid,std_filepath,pid,k8snodeid,uuid,std_pfilename,sid,k8snodename,cmdline,pfilename,filepath,std_filena [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature_script_10015&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature_script_10015","type":"script","value":"ali_uid='';\nclient_ip='';\nevent_type='进程异常行为';\nevent_name='linux可疑命令执行';\nlevel='high';\n___cast_10187=cast(scan_time,'string');\n___ [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature_script_10016&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature_script_10016","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField":"[]" [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature_script_10017&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature_script_10017","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField":"[]" [...]
-dipper.private.blink.rules&&&&channel&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature_channel_10001&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_black_feature_channel_10001","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeI [...]
-dipper.private.blink.rules&&&&pipline&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"nextStageLables\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10002\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"cancelAfterConfigurableRefreshListerner\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"o [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2;sas_black_rule_v3_10001&&&&{"fullClassName":"com.lyra.xs.udf.ext.sas_black_rule_v3","initMethodName":"open","functionName":"sas_black_rule_v3","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2;sas_black_rule_v3_10001","type [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2;b64_auto_10002&&&&{"fullClassName":"com.lyra.xs.udf.ext.b64_auto","initMethodName":"open","functionName":"b64_auto","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2;b64_auto_10002","type":"script","version":"1.0","closeMet [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2;instr_10001&&&&{"fullClassName":"com.lyra.xs.udf.ext.instr","initMethodName":"open","functionName":"instr","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2;instr_10001","type":"script","version":"1.0","closeMethodName":"cl [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2;json_concat_10005&&&&{"fullClassName":"com.aliyun.sec.lyra.udf.ext.JsonConcat","initMethodName":"open","functionName":"json_concat","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2;json_concat_10005","type":"script","versi [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2;sas_black_rule_v2_10002&&&&{"fullClassName":"com.lyra.xs.udf.ext.sas_black_rule_v2","initMethodName":"open","functionName":"sas_black_rule_v2","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2;sas_black_rule_v2_10002","type [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2;b64_auto_10003&&&&{"fullClassName":"com.lyra.xs.udf.ext.b64_auto","initMethodName":"open","functionName":"b64_auto","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2;b64_auto_10003","type":"script","version":"1.0","closeMet [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2;instr_10002&&&&{"fullClassName":"com.lyra.xs.udf.ext.instr","initMethodName":"open","functionName":"instr","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2;instr_10002","type":"script","version":"1.0","closeMethodName":"cl [...]
-dipper.private.blink.rules&&&&channel&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeInOrder":"true","maxFetchLogGroupSize":"1 [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10001","type":"script","value":"logtime=scan_time;\nuuid=host_uuid;\nproc_name=file_name;\ncmd=cmdline;\npproc_name=parent_file_name;\npcmd=parent_cmd_line;\npexe=file_path;\nppexe [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10002","type":"script","value":"retainField(pexe,pid,cmd,aliuid,proc_name,ppexe,logtime,uuid,pproc_name,pcmd,ppid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10003&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10003","type":"script","value":"___lower_proc_name_10001=lower(proc_name);\n___lower_pproc_name_10001=lower(pproc_name);\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_rule_10001&&&&{"expressionStr":"(((((___lower_proc_name_10001,like,'%.exe')|(___lower_pproc_name_10001,like,'%.exe'))|(pexe,like,'_:/%'))|(ppexe,like,'_:/%'))|(pexe,like,'//%'))|(ppexe,like,'//%')","scriptNames":"[]","expressionName":"10614","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aeg [...]
-dipper.private.blink.rules&&&&express&&&&10613&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10612\",\"10608\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10612&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10611\",\"10607\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10604&&&&{"aesFlag":1,"varName":"___lower_proc_name_10001","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"+hpefcjI0jxX3V74t6E7Ow=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10614&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10613\",\"10609\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10606&&&&{"aesFlag":1,"varName":"pexe","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"iEGyh/evZuz6gUfAf8QayQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10605&&&&{"aesFlag":1,"varName":"___lower_pproc_name_10001","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"+hpefcjI0jxX3V74t6E7Ow=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10608&&&&{"aesFlag":1,"varName":"pexe","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"dZof4o+5XqLmguDMF556hA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10607&&&&{"aesFlag":1,"varName":"ppexe","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"iEGyh/evZuz6gUfAf8QayQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10609&&&&{"aesFlag":1,"varName":"ppexe","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"dZof4o+5XqLmguDMF556hA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10611&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10610\",\"10606\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10610&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10604\",\"10605\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10004&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10004","type":"script","value":"data_type='online';\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10005&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10005","type":"script","value":"retainField(pexe,data_type,pid,cmd,aliuid,proc_name,ppexe,logtime,uuid,pproc_name,pcmd,ppid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_rule_10002&&&&{"expressionStr":"data_type,==,'online'","scriptNames":"[]","expressionName":"10615","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_rule_10002","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[]"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10615&&&&{"aesFlag":1,"varName":"data_type","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"MhriRxEga5GjFlRBwhEN4Q=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10006&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10006","type":"script","value":"retainField(pexe,data_type,pid,cmd,aliuid,proc_name,ppexe,logtime,uuid,pproc_name,pcmd,ppid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10007&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10007","type":"script","value":"___unixtime_10006=unixtime(logtime);\nunixtime=___unixtime_10006;rm('___unixtime_10006');\n___lower_proc_name_10002=lower(proc_name);\nclean_proc=__ [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10008&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10008","type":"script","value":"retainField(unixtime,clean_proc,clean_cmd_noquote,clean_pproc,pexe,data_type,pid,cmd,aliuid,proc_name,ppexe,logtime,uuid,pproc_name,pcmd,ppid);","ve [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10009&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10009","type":"script","value":"___multiplication_10001=multiplication(60,60);\n___multiplication_10002=multiplication(___multiplication_10001,24);\n___division_10004=division(unix [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10010&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10010","type":"script","value":"retainField(shot_pcmd,timepart_1hour,log_uid,timepart_1day,unixtime,pid,ppexe,uuid,pproc_name,ppid,clean_proc,clean_cmd_noquote,pexe,clean_pproc,dat [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10001&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.FilterChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"names\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10001_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\ [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10001_rule_10001&&&&{"expressionStr":"data_type,<>,'online'","scriptNames":"[]","expressionName":"10616","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10001_rule_10001","ruleStatus":"0","type":"rule","version":"1. [...]
-dipper.private.blink.rules&&&&express&&&&10616&&&&{"aesFlag":1,"varName":"data_type","functionName":"<>","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"MhriRxEga5GjFlRBwhEN4Q=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10001_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10001_script_10001","type":"script","value":"p2_logtime='';\np2_proc_name='';\np2_cmd='';\np2_pproc_name='';\np2_pcmd='';\np2_pexe='';\np2_ppexe='' [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10001_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10001_script_10002","type":"script","value":"retainField(p2_pid,p2_ppid,p2_pexe,pid,p3_ppexe,ppexe,uuid,p3_pid,pexe,aliuid,p2_proc_name,p2_pcmd,p2_ [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10002&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"entityName\\\\\\\":\\\\\\\"script\\\\\\\",\\\\\\\"scriptName\\\\\\\":\\\\\\\"subpipline_blink.source.aeg [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10002_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10002_script_10001","type":"script","value":"___null_10013=null(p2_logtime);\nif(___null_10013){___if_v3_10001=''}else{___if_v3_10001=p2_logtime};\ [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10002_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10002_script_10002","type":"script","value":"retainField(p2_pid,p2_ppid,p2_pexe,pid,p3_ppexe,ppexe,uuid,p3_pid,pexe,aliuid,p2_proc_name,p2_pcmd,p2_ [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10003&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.FilterChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"names\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10003_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\ [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10003_rule_10001&&&&{"expressionStr":"step_2_continue,==,boolean,false","scriptNames":"[]","expressionName":"10617","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10003_rule_10001","ruleStatus":"0","type":"rule","v [...]
-dipper.private.blink.rules&&&&express&&&&10617&&&&{"varName":"step_2_continue","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Boolean\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"false"}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10003_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10003_script_10001","type":"script","value":"p2_logtime='';\np2_proc_name='';\np2_cmd='';\np2_pproc_name='';\np2_pcmd='';\np2_pexe='';\np2_ppexe='' [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10003_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10003_script_10002","type":"script","value":"retainField(p2_pid,p2_ppid,p2_pexe,pid,p3_ppexe,ppexe,uuid,p3_pid,pexe,aliuid,p2_proc_name,p2_pcmd,p2_ [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10004&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.FilterChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"names\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10004_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\ [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10004_rule_10001&&&&{"expressionStr":"step_3_continue,==,boolean,false","scriptNames":"[]","expressionName":"10618","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10004_rule_10001","ruleStatus":"0","type":"rule","v [...]
-dipper.private.blink.rules&&&&express&&&&10618&&&&{"varName":"step_3_continue","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Boolean\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"false"}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10004_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10004_script_10001","type":"script","value":"___null_10031=null(p2_logtime);\nif(___null_10031){___if_v_tmp_ayd871y7dy12_10001=''}else{___if_v_tmp_ [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10004_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10004_script_10002","type":"script","value":"retainField(p2_pid,p2_ppid,p2_pexe,pid,p3_ppexe,ppexe,uuid,p3_pid,pexe,aliuid,p2_proc_name,p2_pcmd,p2_ [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10005&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.FilterChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"names\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10005_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\ [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10005_rule_10001&&&&{"expressionStr":"chain_level,in,'\\'p2\\',\\'p3\\''","scriptNames":"[]","expressionName":"10619","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10005_rule_10001","ruleStatus":"0","type":"rule", [...]
-dipper.private.blink.rules&&&&express&&&&10619&&&&{"aesFlag":1,"varName":"chain_level","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"H2zd1kq1OIhVTUDIiE4pYw=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10005_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10005_script_10001","type":"script","value":"retainField(p2_pid,p2_ppid,p2_pexe,pid,p3_ppexe,ppexe,uuid,chain_pcmd,p3_pid,pexe,p2_proc_name,p2_pcmd [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10006&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"entityName\\\\\\\":\\\\\\\"script\\\\\\\",\\\\\\\"scriptName\\\\\\\":\\\\\\\"subpipline_blink.source.aeg [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10006_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10006_script_10001","type":"script","value":"p2_logtime='';\np2_proc_name='';\np2_cmd='';\np2_pproc_name='';\np2_pcmd='';\np2_pexe='';\np2_ppexe='' [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10006_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10006_script_10002","type":"script","value":"retainField(p2_pid,p2_ppid,p2_pexe,pid,p3_ppexe,ppexe,uuid,chain_pcmd,p3_pid,pexe,p2_proc_name,p2_pcmd [...]
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_rule_10003&&&&{"expressionStr":"clean_proc,in,'\\'cmd.exe\\',\\'powershell.exe\\''","scriptNames":"[]","expressionName":"10620","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_rule_10003","ruleStatus":"0","type":"rule","version":"1.0","extendField": [...]
-dipper.private.blink.rules&&&&express&&&&10620&&&&{"aesFlag":1,"varName":"clean_proc","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"YGeHkwJkc4xJFjK1BtvAndrjJb4sJfqN+JRKfoTDfew="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10011&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10011","type":"script","value":"retainField(unixtime,shot_pcmd,timepart_1day,pid,ppexe,uuid,pproc_name,ppid,clean_proc,timepart_1hour,clean_cmd_noquote,log_uid,pexe,clean_pproc,dat [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10012&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10012","type":"script","value":"___lower_pproc_name_10003=lower(pproc_name);\n___in_10006=contain(___lower_pproc_name_10003,'cmd.exe','powershell.exe')\n;if(___in_10006){___if_v0_1 [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10013&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10013","type":"script","value":"retainField(step_2_continue,unixtime,shot_pcmd,timepart_1day,pid,ppexe,uuid,pproc_name,ppid,clean_proc,timepart_1hour,clean_cmd_noquote,log_uid,pexe [...]
-dipper.private.blink.rules&&&&window&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_join_window_10001&&&&{"waterMarkMinute":"0","className":"com.aliyun.yundun.dipper.window.model.JoinWindow","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_join_window_10001","type":"window","retainWindowCount":"6","windowType":"tumble","timeout":"30000","sizeInterval":"8","isAutoFlush":"false","nameSpace":"dipper.private.blink.rules","havingMap":"[]","extendField":"[ [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_join_left_10001&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.FilterChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"names\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_join_left_10001_rule_10001\\\\\\\\\\\\\\\"]\\\ [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_join_left_10001_rule_10001&&&&{"expressionStr":"step_2_continue,==,boolean,true","scriptNames":"[]","expressionName":"10621","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_join_left_10001_rule_10001","ruleStatus":"0","type":"r [...]
-dipper.private.blink.rules&&&&express&&&&10621&&&&{"varName":"step_2_continue","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Boolean\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"true"}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_join_left_10001_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_join_left_10001_script_10001","type":"script","value":"___uuid_10002=uuid();\n___concat_10006=concat(uuid,___uuid_10002);\nthe_uuid_dayd71y28y17=___c [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_join_left_10001_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_join_left_10001_script_10002","type":"script","value":"retainField(the_uuid_dayd71y28y17,unixtime,step_2_continue,shot_pcmd,timepart_1day,pid,ppexe,u [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_join_right_10001&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.WindowChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"entityName\\\\\\\":\\\\\\\"window\\\\\\\",\\\\\\\"lable\\\\\\\":\\\\\\\"blink.source.aegis.proc_adl [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10014&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10014","type":"script","value":"p2_logtime=b.logtime;\np2_unixtime=b.unixtime;\np2_uuid=b.uuid;\np2_proc_name=b.proc_name;\np2_cmd=b.cmd;\np2_pproc_name=b.pproc_name;\np2_pcmd=b.pc [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10015&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10015","type":"script","value":"aliuid=b.aliuid;\nclean_cmd_noquote=b.clean_cmd_noquote;\ntimepart_1day=b.timepart_1day;\ntimepart_1hour=b.timepart_1hour;\ndata_type=b.data_type;\n [...]
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_rule_10004&&&&{"expressionStr":"r_Dau8d192yd712yd7,==,double,1","scriptNames":"[]","expressionName":"10622","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_rule_10004","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":" [...]
-dipper.private.blink.rules&&&&express&&&&10622&&&&{"varName":"r_Dau8d192yd712yd7","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"1.0"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10016&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10016","type":"script","value":"retainField(unixtime,p2_pid,shot_pcmd,p2_ppid,p2_pexe,p2_clean_proc,timepart_1day,pid,ppexe,uuid,p2_unixtime,timepart_1hour,pexe,aliuid,p2_proc_name [...]
-dipper.private.blink.rules&&&&window&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_join_window_10002&&&&{"waterMarkMinute":"0","className":"com.aliyun.yundun.dipper.window.model.JoinWindow","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_join_window_10002","type":"window","retainWindowCount":"6","windowType":"tumble","timeout":"30000","sizeInterval":"8","isAutoFlush":"false","nameSpace":"dipper.private.blink.rules","havingMap":"[]","extendField":"[ [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_join_left_10002&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.FilterChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"names\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_join_left_10002_rule_10001\\\\\\\\\\\\\\\"]\\\ [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_join_left_10002_rule_10001&&&&{"expressionStr":"step_3_continue,==,boolean,true","scriptNames":"[]","expressionName":"10623","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_join_left_10002_rule_10001","ruleStatus":"0","type":"r [...]
-dipper.private.blink.rules&&&&express&&&&10623&&&&{"varName":"step_3_continue","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Boolean\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"true"}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_join_left_10002_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_join_left_10002_script_10001","type":"script","value":"retainField(unixtime,p2_pid,shot_pcmd,p2_ppid,p2_pexe,timepart_1day,p2_clean_proc,pid,ppexe,uu [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_join_right_10002&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.WindowChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"entityName\\\\\\\":\\\\\\\"window\\\\\\\",\\\\\\\"lable\\\\\\\":\\\\\\\"blink.source.aegis.proc_adl [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10017&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10017","type":"script","value":"p3_logtime=b.logtime;\np3_unixtime=b.unixtime;\np3_uuid=b.uuid;\np3_proc_name=b.proc_name;\np3_cmd=b.cmd;\np3_pproc_name=b.pproc_name;\np3_pcmd=b.pc [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10018&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10018","type":"script","value":"clean_proc=b.clean_proc;\nclean_cmd_noquote=b.clean_cmd_noquote;\ntimepart_1day=b.timepart_1day;\ndata_type=b.data_type;\nlog_uid=b.log_uid;\naliuid [...]
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_rule_10005&&&&{"expressionStr":"r_Dadu1y2871yd2821tg28,==,double,1","scriptNames":"[]","expressionName":"10624","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_rule_10005","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionName [...]
-dipper.private.blink.rules&&&&express&&&&10624&&&&{"varName":"r_Dadu1y2871yd2821tg28","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"1.0"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10019&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10019","type":"script","value":"retainField(unixtime,p2_pid,shot_pcmd,p2_ppid,p2_pexe,p3_uuid,p3_shot_pcmd,pid,uuid,p2_unixtime,timepart_1hour,p3_pid,step_3_continue,pexe,p2_proc_n [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10020&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10020","type":"script","value":"___lower_p2_pproc_name_10001=lower(p2_pproc_name);\n___in_10007=contain(___lower_p2_pproc_name_10001,'cmd.exe','powershell.exe')\n;if(___in_10007){_ [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10021&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10021","type":"script","value":"retainField(step_3_continue,unixtime,p2_pid,shot_pcmd,p2_ppid,p2_pexe,timepart_1day,p2_clean_proc,pid,ppexe,uuid,p2_unixtime,timepart_1hour,pexe,p2_ [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10007&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"entityName\\\\\\\":\\\\\\\"script\\\\\\\",\\\\\\\"scriptName\\\\\\\":\\\\\\\"subpipline_blink.source.aeg [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10007_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10007_script_10001","type":"script","value":"chain_level='p1';\nchain_pproc_name=pproc_name;\nchain_pcmd=pcmd;\nchain_ppexe=ppexe;\nchain_ppid=ppid [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10007_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10007_script_10002","type":"script","value":"retainField(chain_pcmd,chain_ppid,chain_pproc_name,p2_pid,p2_ppid,p2_pexe,pid,p3_ppexe,uuid,p3_pid,pex [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10008&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.FilterChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"names\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10008_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\ [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10008_rule_10001&&&&{"expressionStr":"p2_pproc_name,<>,''","scriptNames":"[]","expressionName":"10625","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10008_rule_10001","ruleStatus":"0","type":"rule","version":"1.0" [...]
-dipper.private.blink.rules&&&&express&&&&10625&&&&{"aesFlag":1,"varName":"p2_pproc_name","functionName":"<>","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"wc8fi0p7qwBHnY1x2c6Qsw=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10008_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10008_script_10001","type":"script","value":"chain_level='p2';\nchain_pproc_name=p2_pproc_name;\nchain_pcmd=p2_pcmd;\nchain_ppexe=p2_ppexe;\nchain_ [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10008_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10008_script_10002","type":"script","value":"retainField(chain_pcmd,chain_ppid,chain_pproc_name,p2_pid,p2_pexe,pid,p3_ppexe,ppexe,uuid,p3_pid,pexe, [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10009&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.FilterChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"names\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10009_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\ [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10009_rule_10001&&&&{"expressionStr":"p3_pproc_name,<>,''","scriptNames":"[]","expressionName":"10626","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10009_rule_10001","ruleStatus":"0","type":"rule","version":"1.0" [...]
-dipper.private.blink.rules&&&&express&&&&10626&&&&{"aesFlag":1,"varName":"p3_pproc_name","functionName":"<>","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"wc8fi0p7qwBHnY1x2c6Qsw=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10009_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10009_script_10001","type":"script","value":"chain_level='p3';\nchain_pproc_name=p3_pproc_name;\nchain_pcmd=p3_pcmd;\nchain_ppexe=p3_ppexe;\nchain_ [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10009_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_union_10009_script_10002","type":"script","value":"retainField(chain_pcmd,chain_ppid,chain_pproc_name,p2_pid,p2_ppid,p2_pexe,pid,ppexe,uuid,p3_pid,pexe,a [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10022&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10022","type":"script","value":"retainField(chain_ppid,p2_pid,p2_ppid,p2_pexe,pid,p3_ppexe,ppexe,uuid,chain_pcmd,p3_pid,pexe,aliuid,p2_proc_name,p2_pcmd,p2_ppexe,p3_cmd,p3_ppid,p3_ [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10023&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10023","type":"script","value":"___REGEXP_EXTRACT_10003=REGEXP_EXTRACT(cmd,'([a-zA-Z0-9\\/+=]{80,})',1);\ncmd_b64_raw=___REGEXP_EXTRACT_10003;rm('___REGEXP_EXTRACT_10003');\n___low [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10024&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10024","type":"script","value":"retainField(clean_proc,log_uuid,clean_pexe,clean_chain_pproc,clean_chain_pcmd,cmd_b64_raw,clean_pproc,clean_cmd,clean_chain_ppexe,p2_pid,p2_ppid,p2_ [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10025&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10025","type":"script","value":"___null_10040=null(cmd_b64_raw);\nif(___null_10040){___if_proc_log_union_10001=''}else{___if_proc_log_union_10001=cmd_b64_raw};\n___B64_AUTO_10002=B [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10026&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10026","type":"script","value":"retainField(b64_decode_raw,clean_cmd_no_quote,p2_pid,p2_ppid,p2_pexe,clean_cmd,pid,p3_ppexe,ppexe,uuid,chain_pcmd,log_uuid,clean_chain_pproc,clean_c [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10027&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10027","type":"script","value":"___lower_b64_decode_raw_10001=lower(b64_decode_raw);\nb64_decode=___lower_b64_decode_raw_10001;rm('___lower_b64_decode_raw_10001');\n___!null_10014= [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10028&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10028","type":"script","value":"retainField(b64_decode,p2_pid,p2_ppid,p2_pexe,clean_cmd,pid,p3_ppexe,ppexe,uuid,chain_pcmd,log_uuid,clean_chain_pproc,clean_chain_pcmd,p3_pid,pexe,c [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10029&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10029","type":"script","value":"___compare_10016=equals(b64_decode,'');\n___concat_10007=concat(clean_cmd,' / ',b64_decode);\nif(___compare_10016){___if_proc_log_union_10003=clean_ [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10030&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10030","type":"script","value":"retainField(p2_pid,p2_ppid,p2_pexe,clean_cmd,pid,p3_ppexe,ppexe,uuid,chain_pcmd,log_uuid,clean_chain_pproc,clean_chain_pcmd,p3_pid,pexe,cmd_b64_raw, [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10031&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10031","type":"script","value":"_sep1='\\|';\nblack_feature_number='234';\ntmp_1='1';\ntmp_2='2';\ntmp_3='3';\ntmp_4='4';\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10032&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10032","type":"script","value":"retainField(_sep1,black_feature_number,tmp_4,p2_pid,p2_ppid,p2_pexe,pid,uuid,chain_pcmd,p3_pid,pexe,clean_chain_ppexe,p2_proc_name,b64_decode_raw,p2 [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10033&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10033","type":"script","value":"___regexp_10003=regex(clean_cmd_no_quote,'net1?(\\.exe)?\\s+localgroup\\s+administrators.*/add?\\b');\n___regexp_10004=regex(clean_cmd_no_quote,'net [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10034&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10034","type":"script","value":"retainField(sas_module_rule_result_s,_sep1,p2_pid,p2_ppid,p2_pexe,pid,uuid,chain_pcmd,p3_pid,pexe,tmp_4,clean_chain_ppexe,p2_proc_name,tmp_1,tmp_3,t [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10035&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10035","type":"script","value":"___regexp_10035=regex(sas_module_rule_result_s,'^[\\|\\s]*$');\nif(___regexp_10035){___if_new_v3_10001=false}else{___if_new_v3_10001=true};\nhit_mod [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10036&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10036","type":"script","value":"retainField(_sep1,p2_pid,p2_ppid,p2_pexe,pid,uuid,chain_pcmd,p3_pid,pexe,tmp_4,clean_chain_ppexe,p2_proc_name,tmp_1,tmp_3,tmp_2,b64_decode_raw,p2_pp [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10037&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10037","type":"script","value":"___STRING_SPLIT_10001=STRING_SPLIT(sas_module_rule_result_s,_sep1);T.v=udtf.0;","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10038&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10038","type":"script","value":"___trim_10003=trim(T.v);\nfinal_result=___trim_10003;rm('___trim_10003');\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10039&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10039","type":"script","value":"retainField(final_result,_sep1,p2_pid,p2_ppid,p2_pexe,pid,uuid,chain_pcmd,p3_pid,pexe,tmp_4,clean_chain_ppexe,p2_proc_name,tmp_1,tmp_3,tmp_2,b64_dec [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10040&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10040","type":"script","value":"___!null_10015=!null(final_result);\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_rule_10006&&&&{"expressionStr":"___!null_10015&(final_result,regex,'\\S+')","scriptNames":"[]","expressionName":"10628","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_rule_10006","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","ac [...]
-dipper.private.blink.rules&&&&express&&&&10628&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"___!null_10015\",\"10627\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10627&&&&{"aesFlag":1,"varName":"final_result","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"BhPVyKanBl/NbolFClSKRQ=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10041&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10041","type":"script","value":"retainField(_sep1,p2_pid,p2_ppid,p2_pexe,pid,uuid,chain_pcmd,p3_pid,pexe,tmp_4,clean_chain_ppexe,p2_proc_name,tmp_1,tmp_3,tmp_2,b64_decode_raw,p2_pp [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10042&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10042","type":"script","value":"___cast_10222=cast(logtime,'string');\nlogtime=___cast_10222;rm('___cast_10222');\n___cast_10223=cast(uuid,'string');\nuuid=___cast_10223;rm('___cas [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10043&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10043","type":"script","value":"retainField(clean_cmd,final_result,pid,ppexe,uuid,chain_pcmd,log_uuid,clean_chain_pproc,clean_chain_pcmd,pexe,cmd_b64_raw,clean_chain_ppexe,b64_deco [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10044&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10044","type":"script","value":"___regexp_10036=regex(final_result,'^\\d+$');\nif(___regexp_10036){___if_to_online_10001=true}else{___if_to_online_10001=false};\nfrom_black_rule=__ [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10045&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10045","type":"script","value":"retainField(clean_cmd,final_result,pid,ppexe,uuid,chain_pcmd,log_uuid,clean_chain_pproc,clean_chain_pcmd,pexe,cmd_b64_raw,clean_chain_ppexe,b64_deco [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10046&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10046","type":"script","value":"___in_10009=contain(final_result,'839','840','904')\n;___cast_10255=cast('停用安全软件服务','string');\n___in_10010=contain(final_result,'900','901')\n;___c [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10047&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10047","type":"script","value":"retainField(event_name,clean_cmd,final_result,pid,ppexe,uuid,chain_pcmd,log_uuid,clean_chain_pproc,clean_chain_pcmd,pexe,cmd_b64_raw,clean_chain_ppe [...]
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_rule_10007&&&&{"expressionStr":"event_name,<>,''","scriptNames":"[]","expressionName":"10629","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_rule_10007","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[]"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10629&&&&{"aesFlag":1,"varName":"event_name","functionName":"<>","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"wc8fi0p7qwBHnY1x2c6Qsw=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10048&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10048","type":"script","value":"___in_10022=contain(event_name,'可疑的进程路径','可疑的进程文件名')\n;___REGEXP_REPLACE_10018=REGEXP_REPLACE(clean_cmd,'([^/\\s:\\\\\\-]+)','A');\n___concat_10008= [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10049&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10049","type":"script","value":"retainField(clean_cmd,final_result,pid,ppexe,uuid,chain_pcmd,log_uuid,clean_chain_pproc,clean_chain_pcmd,pexe,cmd_b64_raw,clean_chain_ppexe,b64_deco [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10050&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10050","type":"script","value":"___md5_10005=md5(abk_raw);\nabk=___md5_10005;rm('___md5_10005');\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10051&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10051","type":"script","value":"retainField(clean_cmd,final_result,pid,ppexe,uuid,chain_pcmd,log_uuid,clean_chain_pproc,clean_chain_pcmd,pexe,cmd_b64_raw,clean_chain_ppexe,abk_raw, [...]
-dipper.private.blink.rules&&&&window&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_window_10001&&&&{"groupByFieldName":"uuid;event_name;abk;___cast_10276","waterMarkMinute":"0","className":"com.aliyun.yundun.dipper.window.model.OverWindow","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_window_10001","type":"window","version":"1.0","fireDelaySecond":"30","timeout":"30000","activtyTimeOut":"3000","sizeInterval":"60","groupMap":"[]","slideInterval":" [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10052&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10052","type":"script","value":"___unixtime_10007=unixtime(logtime);\n___multiplication_10007=multiplication(30,1);\n___division_10006=division(___unixtime_10007,___multiplication_ [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10053&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10053","type":"script","value":"retainField(r_dasud891ud912,clean_cmd,final_result,pid,ppexe,uuid,chain_pcmd,log_uuid,clean_chain_pproc,clean_chain_pcmd,pexe,cmd_b64_raw,clean_chai [...]
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_rule_10008&&&&{"expressionStr":"r_dasud891ud912,==,double,1","scriptNames":"[]","expressionName":"10630","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_rule_10008","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[]" [...]
-dipper.private.blink.rules&&&&express&&&&10630&&&&{"varName":"r_dasud891ud912","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"1.0"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10054&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10054","type":"script","value":"retainField(clean_cmd,final_result,pid,ppexe,uuid,chain_pcmd,log_uuid,clean_chain_pproc,clean_chain_pcmd,pexe,cmd_b64_raw,clean_chain_ppexe,abk_raw, [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10055&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10055","type":"script","value":"ali_uid='';\nclient_ip='';\nevent_type='进程异常行为';\nlevel='high';\n___cast_10277=cast(logtime,'string');\n___cast_10278=cast(uuid,'string');\n___cast_ [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10056&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_script_10056","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&channel&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_channel_10001&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_v2_channel_10001","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeInOrder":"tru [...]
-dipper.private.blink.rules&&&&pipline&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"nextStageLables\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10002\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"cancelAfterConfigurableRefreshListerner\\\\\\\":\\\\\\\"false [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq;blackrule_10001&&&&{"setCollectorMethodName":"setCollector","fullClassName":"com.sas.zing.blink.udf.BlackRule","initMethodName":"open","functionName":"blackrule","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udtf.BlinkUDTFScript","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq;blackrule_10001","type":"script","version":"1.0" [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq;BlackSeq_10001&&&&{"setCollectorMethodName":"setCollector","fullClassName":"com.aliyun.sec.lyra.hsh.udf.ext.BlackSeq","initMethodName":"open","functionName":"BlackSeq","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udtf.BlinkUDTFScript","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq;BlackSeq_10001","type":"script","version": [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq;json_concat_10006&&&&{"fullClassName":"com.aliyun.sec.lyra.udf.ext.JsonConcat","initMethodName":"open","functionName":"json_concat","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq;json_concat_10006","typ [...]
-dipper.private.blink.rules&&&&channel&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeInOrder":"true","maxFet [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10001","type":"script","value":"___lower_cmdline_10005=lower(cmdline);\nstd_cmdline=___lower_cmdline_10005;rm('___lower_cmdline_10005');\n","version":"1.0","exten [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10002","type":"script","value":"retainField(std_cmdline,k8spodname,file_path,dockerimageid,gid,perm,pid,k8snodeid,host_uuid,file_gid_name,sid,uid,file_gid,k8snode [...]
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_rule_10001&&&&{"expressionStr":"(((std_cmdline,regex,'((\\W+cron)|(^|\\W+)(scp|ssh|crontab|wget|curl|base64|ifconfig|whoami|traceroute|touch|last|history|uname|arp|netstat|useradd|adduser|nslookup|ping|chmod)(\\W+|$)|/rc\\.d|authorized_keys|/etc/passwd|ld\\.so\\.preload|spool/cron|bash_history)')|(file_name,in,'\\'wget\\',\\'ifconfig\\',\\'whoami\\',\\'id\\',\\'curl\\',\\'base64\\',\\' [...]
-dipper.private.blink.rules&&&&express&&&&10635&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10631\",\"10632\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10634&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"twnNM4S7+LShivhp2iyZVb4Xm4ErQA0BvqUDN741JdA="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10637&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10636\",\"10634\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10636&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10635\",\"10633\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10631&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"CwLJQmDR+14h1C68i/ZdDKjlUoeRATpmIGhjlfwX/gvorTB3SGEMkuWKqRHucHZ4g6mLjxqfHKJkH16NBvcUIW9YMuV9x8l1TkETAnaqguwnFY/YJzB3nPQbuRs62XKadorJP+GrKNgFzTw4UesTatNHMHOYprQDT9nWMuUOGSHguXGtZdFFbmCQT9smj8IKCu5Lga17fb93nxud8UPm+SsYrtgGPjx/MQ [...]
-dipper.private.blink.rules&&&&express&&&&10633&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"wuFzc6RuWAdEfPGMyg1TecBnjYtObODlDycixQ7BhZ2+HRmKZZL+SHqnSqZm3R78udkxlc7BUuH7BjmlFauMMiZ3RAuZ0Fhni4mz4y3caG8="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10632&&&&{"aesFlag":1,"varName":"file_name","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"I6Xra4uKOQ4clEQ9mBojYkMaUdfAOQ2dXE7d81U0x9csfvPcc+8lxKZRKrbebd5/JDwUfA8t8ih5S30Yht6bgDIBWdhDUO6wP2N9wP9MQUqZQzEPE6nK6kQ04Z8sBwgl22VW2p6Hyb9cLNWFF3LxQiaUsbXRDNi4fr5RngBrBigzMjSfEvKYay3j7m9XZbdJlMiNRUHyksM4G1utPL73Z+z1YPWkI08BGbnvMv0 [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10003&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10003","type":"script","value":"retainField(k8spodname,file_path,dockerimageid,gid,perm,pid,k8snodeid,host_uuid,file_gid_name,sid,uid,file_gid,k8snodename,cmdline [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10004&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10004","type":"script","value":"___unixtime_10008=unixtime(scan_time);\n___division_10007=division(___unixtime_10008,3600);\n___cast_10301=cast(___division_10007, [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10005&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10005","type":"script","value":"retainField(key_time,k8spodname,file_path,dockerimageid,gid,perm,pid,k8snodeid,host_uuid,file_gid_name,sid,uid,file_gid,k8snodenam [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10006&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10006","type":"script","value":"___lower_parent_file_name_10001=lower(parent_file_name);\n___lower_parent_cmd_line_10001=lower(parent_cmd_line);\n__compare_value_ [...]
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_rule_10002&&&&{"expressionStr":"(((((___lower_parent_file_name_10001,in,'\\'java\\',\\'java (deleted)\\'')|(___lower_parent_cmd_line_10001,regex,'yarn.*?container'))&(((std_cmdline,regex,'((\\W+cron)|(^|\\W+)(scp|ssh|crontab|wget|curl|base64|ifconfig|whoami|traceroute|touch|last|history|uname|arp|netstat|useradd|adduser|nslookup|ping|chmod)(\\W+|$)|/rc\\.d|authorized_keys|/etc/passwd|l [...]
-dipper.private.blink.rules&&&&express&&&&10646&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10645\",\"10642\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10645&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10640\",\"10641\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10648&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10647\",\"__compare_value_10001\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10647&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10644\",\"10646\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10639&&&&{"aesFlag":1,"varName":"___lower_parent_cmd_line_10001","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"UqIEA0mgyxPygn9s2M18j6t59zr/GOjLa11h/KtCj5o="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10638&&&&{"aesFlag":1,"varName":"___lower_parent_file_name_10001","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"7e6pM0MkqkUDyoj+uVhgz90PnvyOrw+8nHng6QFp4uo="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10649&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10648\",\"__compare_value_10002\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10640&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"CwLJQmDR+14h1C68i/ZdDKjlUoeRATpmIGhjlfwX/gvorTB3SGEMkuWKqRHucHZ4g6mLjxqfHKJkH16NBvcUIW9YMuV9x8l1TkETAnaqguwnFY/YJzB3nPQbuRs62XKadorJP+GrKNgFzTw4UesTatNHMHOYprQDT9nWMuUOGSHguXGtZdFFbmCQT9smj8IKCu5Lga17fb93nxud8UPm+SsYrtgGPjx/MQ [...]
-dipper.private.blink.rules&&&&express&&&&10650&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10649\",\"10643\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10642&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"33qbaF1903y7GqiYsI/zCVAb97id/0iB0ALzWF3wGcllWrbNm/9Hk7hK3IcH4cI/LGcxyRAvNy4AYAzkn7j1rg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10641&&&&{"aesFlag":1,"varName":"file_name","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"I6Xra4uKOQ4clEQ9mBojYkMaUdfAOQ2dXE7d81U0x9csfvPcc+8lxKZRKrbebd5/JDwUfA8t8ih5S30Yht6bgDIBWdhDUO6wP2N9wP9MQUqZQzEPE6nK6kQ04Z8sBwgl22VW2p6Hyb9cLNWFF3LxQiaUsbXRDNi4fr5RngBrBigzMjSfEvKYay3j7m9XZbdJlMiNRUHyksM4G1utPL73Z+z1YPWkI08BGbnvMv0 [...]
-dipper.private.blink.rules&&&&express&&&&10644&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10638\",\"10639\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10643&&&&{"varName":"___len_10003","functionName":"<","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"600.0"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10007&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10007","type":"script","value":"retainField(k8spodname,file_path,dockerimageid,gid,perm,pid,k8snodeid,host_uuid,file_gid_name,sid,uid,file_gid,k8snodename,cmdline [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10008&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10008","type":"script","value":"___BLACKRULE_10001=BLACKRULE(file_name,cmdline);T.hit_result=udtf.0;T.score=udtf.1;","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10009&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10009","type":"script","value":"score=T.score;\nhit_result=T.hit_result;\nretainField(score,k8spodname,file_path,dockerimageid,gid,perm,pid,k8snodeid,host_uuid,fi [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10010&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10010","type":"script","value":"___!null_10016=!null(hit_result);\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_rule_10003&&&&{"expressionStr":"___!null_10016&(hit_result,<>,'')","scriptNames":"[]","expressionName":"10652","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_rule_10003","ruleStatus":"0","type":"rule","version":"1.0","extendField" [...]
-dipper.private.blink.rules&&&&express&&&&10651&&&&{"aesFlag":1,"varName":"hit_result","functionName":"<>","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"wc8fi0p7qwBHnY1x2c6Qsw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10652&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"___!null_10016\",\"10651\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&window&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_window_10001&&&&{"groupByFieldName":"host_uuid;ppid;parent_cmd_line;key_time;hit_result","waterMarkMinute":"0","className":"com.aliyun.yundun.dipper.window.model.OverWindow","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_window_10001","type":"window","version":"1.0","fireDelaySecond":"30","timeout":"30000","activtyTimeOut":"3000","sizeInterval":"60 [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10011&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10011","type":"script","value":"___concat_10009=concat(scan_time,' ',pid,' ',cmdline);\ninfo=___concat_10009;rm('___concat_10009');\nrnk=over_parition_10002 [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10012&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10012","type":"script","value":"retainField(uid_name,score,key_time,hit_result,parent_file_path,pid_start_time,parent_cmd_line,scan_time,host_uuid,rnk,ppid,info); [...]
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_rule_10004&&&&{"expressionStr":"rnk,==,double,1","scriptNames":"[]","expressionName":"10653","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_rule_10004","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames [...]
-dipper.private.blink.rules&&&&express&&&&10653&&&&{"varName":"rnk","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"1.0"}&&&&null
-dipper.private.blink.rules&&&&window&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_window_10002&&&&{"groupByFieldName":"host_uuid;ppid;parent_cmd_line;parent_file_path;pid_start_time;uid_name;key_time","waterMarkMinute":"0","sizeAdjust":"0","className":"com.aliyun.yundun.dipper.window.model.WindowMessageProcessor","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_window_10002","type":"window","version":"1.0","windowType":"hop","fire [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10013&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10013","type":"script","value":"retainField(uid_name,score,key_time,min_scan_time,hit_result,cmd_seq_detail_markdown,max_scan_time,parent_file_path,pid_start_time [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10014&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10014","type":"script","value":"___BLACKSEQ_10001=BLACKSEQ(hit_result);T.seq_hit_result=udtf.0;T.seq_score=udtf.1;","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10015&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10015","type":"script","value":"scan_time=max_scan_time;\n___unixtime_10009=unixtime(max_scan_time);\n___unixtime_10010=unixtime(min_scan_time);\n___subtraction_1 [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10016&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10016","type":"script","value":"seq_hit_result=T.seq_hit_result;\nseq_score=T.seq_score;\nretainField(seq_hit_result,seq_score,min_scan_time,hit_result,host_uuid, [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10017&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10017","type":"script","value":"___in_10023=contain(hit_result,'scp,curl,ls','ifconfig,uname,netstat','scp,ls,curl','curl,scp,ls','ls,scp,curl')\n;___regexp_10042 [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10018&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10018","type":"script","value":"retainField(seq_hit_result,hit_result,seq_score,host_uuid,ppid,uid_name,score,key_time,cmd_seq_detail_markdown,is_white,parent_fil [...]
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_rule_10005&&&&{"expressionStr":"((is_white,==,double,0)&(sub_time_delta,>,double,3))&((hit_result,regex,'cron|wget|curl|base64|whoami|traceroute|touch|history|rc\\.d|authorized_keys|etc/passwd|ld\\.so\\.preload|bash_history|useradd|adduser')|(cmd_seq_detail_markdown,regex,'/(shm)(\\W+|$)'))","scriptNames":"[]","expressionName":"10660","varNames":"[]","className":"com.aliyun.filter.proc [...]
-dipper.private.blink.rules&&&&express&&&&10657&&&&{"aesFlag":1,"varName":"cmd_seq_detail_markdown","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"Syjnt42fKMvfydGz7NTIvw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10656&&&&{"aesFlag":1,"varName":"hit_result","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"pH31XIVO87qbawKIzRRpXMtkiIokcOvKNv920+XUbGt8LyMFikVQeSrXyCUYj8Xr94ZujOwbZQcZ3j1DBIooCI3o/dVpIzRR/Q1sF6mXm1vomYWfAwN+PlaXuZBY/OQOkLNyw1VR5od9JkweYCSsDZrxg2j1Hf9qDWrrSlBvfjpMN7LQTXq+9uIed6EKQwAC"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10659&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10656\",\"10657\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10658&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10654\",\"10655\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10660&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10658\",\"10659\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10655&&&&{"varName":"sub_time_delta","functionName":">","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"3.0"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10654&&&&{"varName":"is_white","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0.0"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10019&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10019","type":"script","value":"___REGEXP_REPLACE_10019=REGEXP_REPLACE(parent_cmd_line,'([^\\\\\\:\\>\\-\\&\\@\\=\\%\\s~~~~~\\/\\.\\(\\)\\[\\]]{1})','A');\n___con [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10020&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10020","type":"script","value":"retainField(hit_result,host_uuid,ppid,uid_name,score,key_time,cmd_seq_detail_markdown,is_white,parent_file_path,pid_start_time,par [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10021&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10021","type":"script","value":"uuid=host_uuid;\nali_uid='';\nclient_ip='';\nevent_type='进程异常行为';\nevent_name='linux可疑命令序列';\nlevel='high';\n___cast_10302=cast(sc [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10022&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10022","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10023&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_script_10023","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField [...]
-dipper.private.blink.rules&&&&channel&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_channel_10001&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_java_cmd_seq_channel_10001","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","co [...]
-dipper.private.blink.rules&&&&pipline&&&&blink.source.aegis.proc_windows_proc_alert&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"nextStageLables\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"blink.source.aegis.proc_windows_proc_alert_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"cancelAfterConfigurableRefreshListerner\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"ownerSqlNodeTableName\\\\\\\":\ [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_windows_proc_alert;json_concat_10007&&&&{"fullClassName":"com.aliyun.sec.lyra.udf.ext.JsonConcat","initMethodName":"open","functionName":"json_concat","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_windows_proc_alert;json_concat_10007","type":"script","version":"1.0","closeMethodName": [...]
-dipper.private.blink.rules&&&&channel&&&&blink.source.aegis.proc_windows_proc_alert&&&&{"offset":"0","isJsonData":"true","maxThread":"12","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.MetaqChannel","configureName":"blink.source.aegis.proc_windows_proc_alert","type":"channel","version":"1.0","timeout":"30000","activtyTimeOut":"3000","tags":"T_MSG_PROC","syncTimeout":"5000","groupName":"blink.source.aegis.proc_windows_proc_alert","pullIntervalMs":"100","isBatc [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_windows_proc_alert_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_windows_proc_alert_script_10001","type":"script","value":"___!_10010=!((euid_name,==,'root'));\n___!_10011=!((gid_name,==,'root'));\n___!_10012=!((cwd,like,'/%'));\n___!_10013=!((file_path,like,'/%'));\n___lower_cmdline_10006=lower(cm [...]
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_windows_proc_alert_rule_10001&&&&{"expressionStr":"(((___!_10010&___!_10011)&___!_10012)&___!_10013)&(((((((((((((((((((((((((((((((cmdline,==,'C:\\Windows\\sysnative\\rundll32.exe')|(___lower_cmdline_10006,like,'%remove-etwtraceprovider%'))|(___lower_cmdline_10007,like,'%set-etwtraceprovide%'))|(___lower_cmdline_10008,like,'downloadfile'))|(___lower_cmdline_10009,like,'%python%'))|(___lower_cmdline_10010,like,'%-server%'))|(_ [...]
-dipper.private.blink.rules&&&&express&&&&10691&&&&{"aesFlag":1,"varName":"___lower_cmdline_10035","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"1DXNO09V22zYtPqZn1bbiaCW339/r91kikbTs8L1pg0="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10690&&&&{"aesFlag":1,"varName":"___lower_cmdline_10034","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"HcB8+G3bhDFwwGOEQCQ/iA5ugVmH16U4xqPOFtbd7EA="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10693&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10692\",\"___!_10012\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10692&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"___!_10010\",\"___!_10011\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10695&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10661\",\"10662\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10694&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10693\",\"___!_10013\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10697&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10696\",\"10664\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10696&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10695\",\"10663\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10699&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10698\",\"10666\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10698&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10697\",\"10665\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10668&&&&{"aesFlag":1,"varName":"___lower_cmdline_10012","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"nlLh9YdsiV2yjTK21R8oOMcLWrcKR7KDYi4tkGtBwgk="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10701&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10700\",\"10668\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10667&&&&{"aesFlag":1,"varName":"___lower_cmdline_10011","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"c+6mVJmqxvnL5T2vgZIAgMgnIPLs7yS9AKhnOaKvYsU="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10700&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10699\",\"10667\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10703&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10702\",\"10670\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10669&&&&{"aesFlag":1,"varName":"___lower_cmdline_10013","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"bts3C7Iv0knNtua4HgMBcA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10702&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10701\",\"10669\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10705&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10704\",\"10672\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10704&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10703\",\"10671\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10707&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10706\",\"10674\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10706&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10705\",\"10673\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10662&&&&{"aesFlag":1,"varName":"___lower_cmdline_10006","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"UJuXH91K5ufQdMSBw/9dO3oJ7mBcuOLj+G679L4ZgAg="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10661&&&&{"aesFlag":1,"varName":"cmdline","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"GiuFTdqkKrVRCETpgMXTSEhSw/bXgUe62P9eR7Tuf/14TDWHZ+KpkarcJYIAWS20"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10664&&&&{"aesFlag":1,"varName":"___lower_cmdline_10008","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"MHw1NnOw0fBv/9wOUu/TYw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10663&&&&{"aesFlag":1,"varName":"___lower_cmdline_10007","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"5Y3vR5kCpCB0CRfg5I05KVWwO4xV12x7LFX1auE1fco="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10666&&&&{"aesFlag":1,"varName":"___lower_cmdline_10010","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0wspT/mcGZ1o5eFV6M5utQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10665&&&&{"aesFlag":1,"varName":"___lower_cmdline_10009","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"FxMXP+57FWiYqHaAJrBpHA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10709&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10708\",\"10676\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10708&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10707\",\"10675\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10679&&&&{"aesFlag":1,"varName":"___lower_cmdline_10023","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"dSGJNPlFXFT53DW7MD+RnXjRLrLuRQw4c9oWKAiOQNA="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10712&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10711\",\"10679\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10678&&&&{"aesFlag":1,"varName":"___lower_cmdline_10022","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"lSx6dNF+UJTajp8+qo+4Z3hfKlDRh+NpNOOAmUeUHDI="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10711&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10710\",\"10678\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10714&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10713\",\"10681\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10713&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10712\",\"10680\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10716&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10715\",\"10683\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10715&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10714\",\"10682\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10718&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10717\",\"10685\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10717&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10716\",\"10684\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10671&&&&{"aesFlag":1,"varName":"___lower_cmdline_10015","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"9l1k2cuAJNWa587hRslM0w=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10670&&&&{"aesFlag":1,"varName":"___lower_cmdline_10014","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"mDV/avDfTki1AHskqwom4yKcLtJR6sshCj9PlyQ1VXM="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10673&&&&{"aesFlag":1,"varName":"___lower_cmdline_10017","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"511mpeNKv1n6cNnHXHq9YQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10672&&&&{"aesFlag":1,"varName":"___lower_cmdline_10016","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"O+q1fjwuh2/ArsMWF7bUJkXPjAtBuvER9GC7ERKdSrY="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10675&&&&{"aesFlag":1,"varName":"___lower_cmdline_10019","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"QJnS8KMYMO2TuFV8fF6hEA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10674&&&&{"aesFlag":1,"varName":"___lower_cmdline_10018","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"TtxFg2EFWi0FFLwAVRJ8+Q=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10677&&&&{"aesFlag":1,"varName":"___lower_cmdline_10021","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"iPdmBDpUrSMrJiq+OR4/puSrW0Tq5Pqa+VHC/wFPLJ8="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10710&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10709\",\"10677\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10676&&&&{"aesFlag":1,"varName":"___lower_cmdline_10020","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"dK6GdLIx7xnwcqUSBRYc6Q=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10680&&&&{"aesFlag":1,"varName":"___lower_cmdline_10024","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"eGwOGkBJt9/XpwUo+xiSj+wHE9/u4t4yeQgEuguZNJg="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10719&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10718\",\"10686\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10723&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10722\",\"10690\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10689&&&&{"aesFlag":1,"varName":"___lower_cmdline_10033","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"4JmhxgOXw39XdlLTx6IoItJOs7TvOGGw1NTS/SECYp4="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10722&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10721\",\"10689\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10725&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10694\",\"10724\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10724&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10723\",\"10691\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10682&&&&{"aesFlag":1,"varName":"___lower_cmdline_10026","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"+mmSz9m3qq6YHEPpm0Jw6nAegjoaujGC7wEcUtaFMKA="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10681&&&&{"aesFlag":1,"varName":"___lower_cmdline_10025","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"crIVGRIff/X1UwOyvqe3tbFWVfNhMF9xriQDcM2+x8s="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10684&&&&{"aesFlag":1,"varName":"___lower_cmdline_10028","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"P3uaGLf3R9thLbIDVVMSBQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10683&&&&{"aesFlag":1,"varName":"___lower_cmdline_10027","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"waeN/HHOGnVV2R9JOkgnZw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10686&&&&{"aesFlag":1,"varName":"___lower_cmdline_10030","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"K0QdvfhOJxt6iOFEpIHhkw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10685&&&&{"aesFlag":1,"varName":"___lower_cmdline_10029","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"Xjwz23RLOZZZkOuIjIyKw/mq16npd1kn10XLxnH2Fpg="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10688&&&&{"aesFlag":1,"varName":"___lower_cmdline_10032","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"3y2jL4IIVqZQo9/OaBbxSMXhLn2TlFXbu7ktXEoN1gc="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10721&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10720\",\"10688\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10687&&&&{"aesFlag":1,"varName":"___lower_cmdline_10031","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"NCiSf16sRFDAnXhnd5QoBQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10720&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10719\",\"10687\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_windows_proc_alert_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_windows_proc_alert_script_10002","type":"script","value":"retainField(k8spodname,file_path,dockerimageid,pid,k8snodename,cmdline,containermip,euid_name,parent_file_path,buySas,file_uid,containerhostname,dockerimagename,index,file_uid_ [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_windows_proc_alert_script_10003&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_windows_proc_alert_script_10003","type":"script","value":"___compare_10037=equals(cmdline,'C:\\Windows\\sysnative\\rundll32.exe');\n___lower_cmdline_10036=lower(cmdline);\n___lower_cmdline_10037=lower(cmdline);\n___regexp_10043=regex( [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_windows_proc_alert_script_10004&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_windows_proc_alert_script_10004","type":"script","value":"retainField(feature,k8spodname,file_path,dockerimageid,pid,k8snodename,cmdline,containermip,euid_name,parent_file_path,buySas,file_uid,containerhostname,dockerimagename,index,f [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_windows_proc_alert_script_10005&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_windows_proc_alert_script_10005","type":"script","value":"___!null_10017=!null(feature);\n___lower_cmdline_10072=lower(cmdline);\n___!_10020=!((___lower_cmdline_10072,regex,'\\\\(kingdee|u8soft)\\\\|软件|安装|kingdee\\.erp\\b'));\n___lowe [...]
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_windows_proc_alert_rule_10002&&&&{"expressionStr":"(___!null_10017&___!_10020)&___!_10021","scriptNames":"[]","expressionName":"10727","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_windows_proc_alert_rule_10002","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[]"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10727&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10726\",\"___!_10021\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10726&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"___!null_10017\",\"___!_10020\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_windows_proc_alert_script_10006&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_windows_proc_alert_script_10006","type":"script","value":"retainField(k8spodname,file_path,dockerimageid,pid,k8snodename,cmdline,containermip,euid_name,parent_file_path,buySas,file_uid,containerhostname,dockerimagename,index,file_uid_ [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink.source.aegis.proc_windows_proc_alert_union_10001&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.FilterChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"names\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"subpipline_blink.source.aegis.proc_windows_proc_alert_union_10001_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\" [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_blink.source.aegis.proc_windows_proc_alert_union_10001_rule_10001&&&&{"expressionStr":"feature,in,'\\'win_abnormal_cs_rundll32\\''","scriptNames":"[]","expressionName":"10728","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_windows_proc_alert_union_10001_rule_10001","ruleStatus":"0","type":"rule","version":"1.0","ex [...]
-dipper.private.blink.rules&&&&express&&&&10728&&&&{"aesFlag":1,"varName":"feature","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"FOflTKdMf+15emnUH/e77Gskgf5mE4YpmyIkXdl3FN4="}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_windows_proc_alert_union_10001_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_windows_proc_alert_union_10001_script_10001","type":"script","value":"uuid=host_uuid;\nali_uid=aliUid;\n___cast_10317=cast(null_10005,'string');\nclient_ip=___cast_10317;rm('___cast_10317');\nevent_ty [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_windows_proc_alert_union_10001_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_windows_proc_alert_union_10001_script_10002","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_windows_proc_alert_union_10001_script_10003&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_windows_proc_alert_union_10001_script_10003","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink.source.aegis.proc_windows_proc_alert_union_10002&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.FilterChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"names\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"subpipline_blink.source.aegis.proc_windows_proc_alert_union_10002_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\" [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_blink.source.aegis.proc_windows_proc_alert_union_10002_rule_10001&&&&{"expressionStr":"feature,in,'\\'win_easyproxy_command_hw\\''","scriptNames":"[]","expressionName":"10729","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_windows_proc_alert_union_10002_rule_10001","ruleStatus":"0","type":"rule","version":"1.0","ex [...]
-dipper.private.blink.rules&&&&express&&&&10729&&&&{"aesFlag":1,"varName":"feature","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"PSDdkQol428PFYGh7Dcfm/PbASwQqp3BKR4psJ6rZfw="}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_windows_proc_alert_union_10002_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_windows_proc_alert_union_10002_script_10001","type":"script","value":"uuid=host_uuid;\nali_uid=aliUid;\n___cast_10326=cast(null_10006,'string');\nclient_ip=___cast_10326;rm('___cast_10326');\nevent_ty [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_windows_proc_alert_union_10002_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_windows_proc_alert_union_10002_script_10002","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink.source.aegis.proc_windows_proc_alert_union_10003&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.FilterChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"names\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"subpipline_blink.source.aegis.proc_windows_proc_alert_union_10003_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\" [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_blink.source.aegis.proc_windows_proc_alert_union_10003_rule_10001&&&&{"expressionStr":"feature,in,'\\'win_zerologon_cve_exploit\\''","scriptNames":"[]","expressionName":"10730","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_windows_proc_alert_union_10003_rule_10001","ruleStatus":"0","type":"rule","version":"1.0","e [...]
-dipper.private.blink.rules&&&&express&&&&10730&&&&{"aesFlag":1,"varName":"feature","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"w9wWWEIjQxvyHyfREtUC1FflMm/xKCEDvO0aMitr0ks="}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_windows_proc_alert_union_10003_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_windows_proc_alert_union_10003_script_10001","type":"script","value":"uuid=host_uuid;\nali_uid=aliUid;\n___cast_10335=cast(null_10007,'string');\nclient_ip=___cast_10335;rm('___cast_10335');\nevent_ty [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_windows_proc_alert_union_10003_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_windows_proc_alert_union_10003_script_10002","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink.source.aegis.proc_windows_proc_alert_union_10004&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.FilterChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"names\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"subpipline_blink.source.aegis.proc_windows_proc_alert_union_10004_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\" [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_blink.source.aegis.proc_windows_proc_alert_union_10004_rule_10001&&&&{"expressionStr":"feature,in,'\\'win_payload_exec_mshta\\',\\'win_payload_exec_msiexec\\',\\'win_payload_exec_regsvr32\\',\\'win_defender_downloadfile\\',\\'win_etw_remove_provider\\',\\'win_etw_set_0x11\\',\\'win_payload_exec_msbuild\\',\\'win_payload_exec_compiler\\',\\'win_payload_exec_odbcconf\\',\\'win_payload_exec_odbcconf_1\\',\\'win_exec_payload_zipfldr_dll\\',\\' [...]
-dipper.private.blink.rules&&&&express&&&&10731&&&&{"aesFlag":1,"varName":"feature","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"QOMF2vJ0Y7lTl0jvnS+jFy6YC/1IcZNWIRz7BAJsQlF0Um3Lx25s0sg5+ZgyKt+ExllvEbHcp+evw1La9GuhLV2HZqz4OjoE0WPOJ4+HDFxo8uraxQMXA2JTp9bl6ED64M9crMNGx6bf7CU4ltbtePKaChzWo/GpxBseKHMDTDXV/ujaLmvZgBVqpEPUsXZlrhI9uvln1AJFjBhYs/g25CaC1WfAlv1C86j9zdSFZ [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_windows_proc_alert_union_10004_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_windows_proc_alert_union_10004_script_10001","type":"script","value":"uuid=host_uuid;\nali_uid=aliUid;\n___cast_10342=cast(null_10008,'string');\nclient_ip=___cast_10342;rm('___cast_10342');\nevent_ty [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_windows_proc_alert_union_10004_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_windows_proc_alert_union_10004_script_10002","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink.source.aegis.proc_windows_proc_alert_union_10005&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.FilterChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"names\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"subpipline_blink.source.aegis.proc_windows_proc_alert_union_10005_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\" [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_blink.source.aegis.proc_windows_proc_alert_union_10005_rule_10001&&&&{"expressionStr":"feature,in,'\\'win_payload_exec_msiexec_1\\',\\'win_payload_exec_installutil\\',\\'win_exec_payload_url_dll\\',\\'win_exec_payload_fsi\\',\\'win_exec_payload_msxsl\\',\\'win_exec_payload_slmgr_vbs\\''","scriptNames":"[]","expressionName":"10732","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.b [...]
-dipper.private.blink.rules&&&&express&&&&10732&&&&{"aesFlag":1,"varName":"feature","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"QOMF2vJ0Y7lTl0jvnS+jF7Kfoo4R5snvDayimjOhNU2hjulq20vnUX5/6wjAS54pe2FMLTgo8ZMTMutdahW0sxrBfa/ykiTZ/YJPSAAR0CgQpmUGXkmJxyfHNhk7BiBJq43uQWH3g8VwmyQcP8YMzygYJIdTwDAtd2PRwiKcUN7+09rJZYiiNbGB7KCdHr9rkHhm+iNRKoxMDPaKzVatbNyFyA+Ps9ot8RwR5CBCy [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_windows_proc_alert_union_10005_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_windows_proc_alert_union_10005_script_10001","type":"script","value":"uuid=host_uuid;\nali_uid=aliUid;\n___cast_10351=cast(null_10009,'string');\nclient_ip=___cast_10351;rm('___cast_10351');\nevent_ty [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_windows_proc_alert_union_10005_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_windows_proc_alert_union_10005_script_10002","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField [...]
-dipper.private.blink.rules&&&&channel&&&&blink.source.aegis.proc_windows_proc_alert_channel_10001&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink.source.aegis.proc_windows_proc_alert_channel_10001","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeInOrder":"true","maxFetchLogGroupSize":"1 [...]
-dipper.private.blink.rules&&&&pipline&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_reverse_shell&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"nextStageLables\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"blink.source.aegis.proc_sas_linux_alert_proc_cmd_reverse_shell_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"cancelAfterConfigurableRefreshListerner\\\\\\\":\\\\\\\"false\\\\\\\" [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_reverse_shell;b64_auto_10004&&&&{"fullClassName":"com.lyra.xs.udf.ext.b64_auto","initMethodName":"open","functionName":"b64_auto","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_reverse_shell;b64_auto_10004","type":"script","version":"1. [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_reverse_shell;json_concat_10008&&&&{"fullClassName":"com.aliyun.sec.lyra.udf.ext.JsonConcat","initMethodName":"open","functionName":"json_concat","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_reverse_shell;json_concat_10008","type":"sc [...]
-dipper.private.blink.rules&&&&channel&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_reverse_shell&&&&{"offset":"0","isJsonData":"true","maxThread":"12","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.MetaqChannel","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_reverse_shell","type":"channel","version":"1.0","timeout":"30000","activtyTimeOut":"3000","syncTimeout":"5000","groupName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_rever [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_reverse_shell_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_reverse_shell_script_10001","type":"script","value":"___lower_cmdline_10074=lower(cmdline);\n___lower_cmdline_10075=lower(cmdline);\n___lower_cmdline_10076=lower(cmdline);\n___lower_cmdline [...]
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_reverse_shell_rule_10001&&&&{"expressionStr":"((((((((((((((((___lower_cmdline_10074,like,'%mkfifo%')|(___lower_cmdline_10075,like,'%mknod%'))|(___lower_cmdline_10076,like,'%ruby%'))|(___lower_cmdline_10077,like,'%php%'))|(___lower_cmdline_10078,like,'%python%'))|(___lower_cmdline_10079,like,'%perl%'))|(___lower_cmdline_10080,like,'%lua%'))|(___lower_cmdline_10081,like,'%/dev/tcp%'))|(___lower_cmdline_ [...]
-dipper.private.blink.rules&&&&express&&&&10734&&&&{"aesFlag":1,"varName":"___lower_cmdline_10075","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"76Z3DGJQG4I8UX4yQ0zKMA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10756&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10755\",\"10740\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10733&&&&{"aesFlag":1,"varName":"___lower_cmdline_10074","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"xuBsPOhvC+eJ+3psYSQkeQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10755&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10754\",\"10739\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10736&&&&{"aesFlag":1,"varName":"___lower_cmdline_10077","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"ceKMAB0lbTSftqkArbAv4A=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10758&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10757\",\"10742\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10735&&&&{"aesFlag":1,"varName":"___lower_cmdline_10076","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"CCSvCz48rm4jh396b6cayQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10757&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10756\",\"10741\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10738&&&&{"aesFlag":1,"varName":"___lower_cmdline_10079","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"4D0BUhBf+jwANRCDCn+WAQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10737&&&&{"aesFlag":1,"varName":"___lower_cmdline_10078","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"FxMXP+57FWiYqHaAJrBpHA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10759&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10758\",\"10743\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10739&&&&{"aesFlag":1,"varName":"___lower_cmdline_10080","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"nAs/fHOR7BFfSJbADmiLBg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10750&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10733\",\"10734\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10752&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10751\",\"10736\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10751&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10750\",\"10735\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10754&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10753\",\"10738\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10753&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10752\",\"10737\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10745&&&&{"aesFlag":1,"varName":"___lower_cmdline_10086","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"DrhqThY1WiHymPq4y4rHDQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10744&&&&{"aesFlag":1,"varName":"___lower_cmdline_10085","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"GPf/PJUXzl7O8JEk0vEK4g=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10747&&&&{"aesFlag":1,"varName":"___lower_cmdline_10088","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"NEZsDO4VmZINRfxAxhdUFw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10746&&&&{"aesFlag":1,"varName":"___lower_cmdline_10087","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"wTFYxBwBPt0Xry61A5WAlw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10749&&&&{"aesFlag":1,"varName":"___lower_cmdline_10090","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"EBOsKA8HJr0TeID5FpnEGjfpDHBSFupZNEu1Ylq6iBg="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10748&&&&{"aesFlag":1,"varName":"___lower_cmdline_10089","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"+VXudFH5qdlcHBoTnnOF6Q=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10761&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10760\",\"10745\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10760&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10759\",\"10744\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10741&&&&{"aesFlag":1,"varName":"___lower_cmdline_10082","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"fma6b4Spm2KiAVOvs7FGgA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10763&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10762\",\"10747\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10740&&&&{"aesFlag":1,"varName":"___lower_cmdline_10081","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0XtgBTM2g7m5T8Z+yW4W/g=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10762&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10761\",\"10746\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10743&&&&{"aesFlag":1,"varName":"___lower_cmdline_10084","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"tR2RJ8N0B0TommIJUF6mtQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10765&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10764\",\"10749\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10742&&&&{"aesFlag":1,"varName":"___lower_cmdline_10083","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"GazzQ4W76X8ul6UhR3fkiw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10764&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10763\",\"10748\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_reverse_shell_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_reverse_shell_script_10002","type":"script","value":"___lower_cmdline_10091=lower(cmdline);\n___REGEXP_REPLACE_10020=REGEXP_REPLACE(___lower_cmdline_10091,'\\s+',' ');\nstd_cmdline=___REGEX [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_reverse_shell_script_10003&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_reverse_shell_script_10003","type":"script","value":"retainField(b64_cmdline,std_cmdline,k8spodname,file_path,dockerimageid,gid,perm,pid,k8snodeid,host_uuid,file_gid_name,sid,uid,file_gid,k [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_reverse_shell_script_10004&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_reverse_shell_script_10004","type":"script","value":"___compare_10039=equals(b64_cmdline,'');\n___null_10041=null(b64_cmdline);\n___B64_AUTO_10003=B64_AUTO(b64_cmdline);\nif((___compare_100 [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_reverse_shell_script_10005&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_reverse_shell_script_10005","type":"script","value":"retainField(k8spodname,file_path,dockerimageid,gid,perm,pid,k8snodeid,host_uuid,file_gid_name,sid,uid,file_gid,k8snodename,cmdline,euid_ [...]
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_reverse_shell_rule_10002&&&&{"expressionStr":"(((((((((((((((((((((((((((((((((((((((std_cmdline,regex,'(mkfifo|mknod).*?&&\\s*(nc|telnet).*?<.*?\\|.*?ash')|(std_cmdline,regex,'(mkfifo|mknod)\\s+.*?(nc|telnet)\\s+\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\s+\\d+'))|(std_cmdline,regex,'(mkfifo|mknod).*?&&.*?\\s+(nc|telnet).*?<.*?\\|.*?ash'))|(std_cmdline,regex,'(mkfifo|mknod).*?cat.*?\\|.*?/sh\\s+-i.*?\ [...]
-dipper.private.blink.rules&&&&express&&&&10808&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10807\",\"10769\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10807&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10806\",\"10768\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10809&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10808\",\"10770\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10778&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"LBKmL6KqZ002Z8LPX2ZXKgs8LXP3px/nMlaPWul31KlA99EJmuPqx8xN1p/4MU86zmORdaJdZNbGUY9riocOPYSYlAMoc/+JMZTmZc4cIno="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10811&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10810\",\"10772\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10777&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"lHpjAZDDnHmcBr6mvOIo5/Ukm8at96usKGZaj9YxaO7p3FOjnlLJ6EGv0wmw/6nSW0YTOBYIrGusJdf4aDa/ATpXsXbzD09fqQI4TNYAG+/yYeZi1Z6S5g4s58YqlGi83kL/hb3rcihxdnBnrj8VWw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10810&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10809\",\"10771\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10813&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10812\",\"10774\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10779&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"NV7bVKYDDm8nb2NigCGmN1xpPVFB1oTf5Ys/TyaFOnAHb5lzJgf0g/BRpbvmdToRTTqSRsdL2+lPDkchdB9r/kPBJCB7eXD8HQPnqrewfC0="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10812&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10811\",\"10773\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10815&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10814\",\"10776\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10814&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10813\",\"10775\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10817&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10816\",\"10778\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10816&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10815\",\"10777\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10770&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"BSkRZytZ0ZpioJ9N79IarCuS0vN5CWanNxAcOH+MV+NL9zjcLpOCrCkedIJ8uvou"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10772&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"CrxhPkc0RHm9lXpo7SsrSUn8iaXwMMn3Xkh0YiMRk1GX57WY6Z31M+5zW72Z+XDn"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10771&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"BSkRZytZ0ZpioJ9N79IarCuS0vN5CWanNxAcOH+MV+Och4O2jYw4lY6meG12L2QW"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10774&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"LBKmL6KqZ002Z8LPX2ZXKn4RNcTFDxJ9ZnjE5wwsbZkQJg03gcZQxN7azcTpUH92Gceyab2d79NWCFDS4Ybd2A=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10773&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"LBKmL6KqZ002Z8LPX2ZXKgs8LXP3px/nMlaPWul31KlPju2FD1rSzWDM83X8J51bdSKErCi3ziUUk6epR1Oia/Dq68Ed/irl/tSau7jWCPg="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10776&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"CXjr80lckkCQoRmGUctHXXNahm3tYU2TR6eNXFlHK8v4p55wK/41TMoXz8QSOep+x5+NkUo7biKiqiMSGjVxlA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10775&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"LBKmL6KqZ002Z8LPX2ZXKhduy30Dn26KBpyISm4ovJM="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10819&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10818\",\"10780\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10818&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10817\",\"10779\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10789&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"wY2TXDp84Np+fxCQw153qSOBRfZ5HzQhU8et9MwgBsM="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10822&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10821\",\"10783\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10788&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"ECKiR+VSLN0UJKMNnLRBzhCRKUJ1lwglvQPeOT3o6HZ1w1lsZlOOAWGqvcy19vuPUD2qnyEV5dcJYaWPL7dnuw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10821&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10820\",\"10782\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10824&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10823\",\"10785\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10823&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10822\",\"10784\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10826&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10825\",\"10787\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10825&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10824\",\"10786\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10828&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10827\",\"10789\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10827&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10826\",\"10788\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10781&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"08eIrXRH4/zBzclSu/pGXTKNLNe4aDaGLzNp6CZLZHqIScKtHVjW+rNnDlQNM5Ul"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10780&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"q446atAYdfv/juzzkg2av0YC+IdO/V4Q7ZO660c7xMoX4DzsQKcRd1HYZWKBSWI/hu/f5lLfCrhVpG0Hkx7Dfg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10783&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"X7LL46uxt6ySvKv7Mdn0I0Q8YoR2tVNQoK1NrfBE5hQCa3mihhfi4yRujZLJc6TIWqnQ9RA9VYkKXDll+CTTdNd2VZWpFAoEqj0fNfmqHew="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10782&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"Mj0zkksT1HdenpDhT9WFOrm0tENxg8Kldpy7gHrJ8qe6wUQs3LP0HTi35tI7/UMHWQGeXe97XMEauxOrwZ2DKInetFnPJdXRApTHPvjiB9UedO6AuYUeC7f9NNfG8VwQ90pH09+qU1yrErVIR9CUoOmCbDPc5YoyoBRziz1Y7B8="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10785&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"EBOsKA8HJr0TeID5FpnEGv1DYfkDqtN7zTZ9BqruY3GKaxnBlsl84RDlXhtxLN/VhPF7pfBk1150eJDAaSvD3EZDddj34f3nf7XJxONVt8o="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10784&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"NfOPvvUA2C02nnvJmlFVPIb1AQXWnScVSFTS0j3O1NPUlMwe38KWRFHo43FlwB+bI7Ls06CYxVy8xW4FGawtoQfHg7fPcVMJm4aGtUsThf0="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10787&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"EBOsKA8HJr0TeID5FpnEGk8OfW5gge289PC+T42dZMPfzvz38Yo1A+ktYxd8gOdRbTkQ6daB0x0EWcyD88KknA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10820&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10819\",\"10781\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10786&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"EBOsKA8HJr0TeID5FpnEGhsz6jEjv19XPg1ML+aMKIi3VPwmHsnAX/AjthoMDVrxmmW3OL4PYANEf/gEOw3q9983zvSxEaeEZnnJHSA0vOr5et9uXOwEWhVBISH7pcah3E3j8pR9+z33jDX5I8XAFw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10790&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"EltdZFqbkkAES4K1pPltxkDPXxUciI3AL/903+0qeBtp7p/QOb1x7LSgxP/GQ1GW6tr48aEmZrzeGa+ihc7JuA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10829&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10828\",\"10790\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10833&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10832\",\"10794\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10799&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"rbNiWCFJ1VOTUaTYNWWFjOks2W49Kh8DLdIZtQ2EzLkO3m8OuP3OtjgtdOk34JMKOCRHpblLKu0kWKtpbcHWGd6BdDk61fAXwRgrnAd915SLoifdkhuIEXICOM+x2iV3hzg9oPHQPavoAbS8d4dy8lJIXhBrj0Q+OoG+y+s3HLkBoQrLcQA1Hv/NhNWn8Wp7"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10832&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10831\",\"10793\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10835&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10834\",\"10796\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10834&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10833\",\"10795\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10837&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10836\",\"10798\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10836&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10835\",\"10797\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10839&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10838\",\"10800\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10838&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10837\",\"10799\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10792&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"WOspaXLAGxMn68UjGzgPMtwakB96I2wWn8tkcxUQ+tg6f1w3UvEhq/9YTeG7aUxpjSRn8vaLVPWw9v4R3pjTNQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10791&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"fKKxPOBqvGDBL8gf1En+jXSeH09f2NiEAxOCTMHPPVyyDXOfZVpRhzCifKHBQ/GhlUIK5Tys3qhw+gGDA0mkYYXESYwkz3eqWtkvgNfFRNisXNcGFispFD6fzctEQX4prer11PAbdiGkpKUs7Ppf5Aug1zMNiDHQ5zUyuXArtGk="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10794&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"HDGdCUToGqETm9cV+Jw1DsDelVY01kGxLRoXbmDebmq0ptZlxf4r1YBlKFA+DgIYCIa6biOy7/UnU5VuvhpfoVaHGznnJEXLCs/fkUI3MN8="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10793&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"mPvs4d/P/GyxoYG1nO5HKdG8cMpJ6tz8ID7jPha5bnSqCeJrUCkGBjKl18tBIg+QCUV0N4n7ecI6VRZC40pLoQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10796&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"5ZNqmWb9CISLIDGXbtTJTy8m42FzmkUCi4gyHgOtkeTKv2s/vYmBcMKOvaE5obvRp3ccPXiszeVJn658WBz0awsM1TMCFl7GUnPi2+gs2eAcKDT4upRc42a3Za4yhGAh"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10795&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"CXjr80lckkCQoRmGUctHXan4nAjLiopKSgP/26U9t4yuK6Bl8q0n/biqPgoe9SohDIqwDc9EOxlTF2d6HT/ydqFlHAq4xThHqrmrmzQuNME="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10798&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"EBOsKA8HJr0TeID5FpnEGk8OfW5gge289PC+T42dZMOqMmQwUl25j4sV82bdZQzF/V0UaVObgu2k9W0KBlU62DcNpYQX+HXJ/NOfAiltx3uJUafisLp3TjbNz8RW2t31PrxOin2ExPZ/iFDtjtJ7KhEc+xSD3WstU/hAkG7NUVFai5glBbssLSfmpGv3kpK92izpiyUS+lwlnhImJlzGGA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10831&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10830\",\"10792\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10797&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"EBOsKA8HJr0TeID5FpnEGk8OfW5gge289PC+T42dZMNMisrib+zJeoIn9G5WoIasLow+VyKYs3BOJvFkqaP26LIsE+5OHgL2M2NQOIpDEs4LZ2WgKDZR/FQSNb2IcClCRtp7cXJook9ARB5Oa15rJ+IgQOtzOJCpZmQ/Rsox/0bPMQtaY6NZwFe073gtexrpwpLmeNy+zV/4cjaI1OgX2A=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10830&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10829\",\"10791\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10767&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0yWv4wJ+Lh91yiZrWoWwA0QiYld/6Ag9ohQZOM8G0t+P6eS4ias9aG9XikdKtcAEOkSjBa5XjgVVzcEIy2GZZYXIX+Z5+7mUXqjc86ueoec="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10800&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0IGi0JB4pKA1Wt/wiGkw5MzLsWGbN1r3tBDGoqrEzx8QSLV+qaUX8OUFTyjlPN+c8BUDJ3iKalLC0XMulL+y1g=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10844&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10843\",\"10805\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10766&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"1aiRaiqvOFibkrM/JSSjCN3h9r9DoiD1IbnovraXqlwyRhSP5ggEQfYnFfl56+p2smz0AS1sJ88HqT9U6zr2aA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10843&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10842\",\"10804\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10769&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"1aiRaiqvOFibkrM/JSSjCCff+VqjUQK1P+2rBOqpjPlMSmR7R6NZMGtHOYlOv6U7ndgdp46lb0IXh8tZc3TW2g=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10802&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"hxdJkcCWlOLR0Z08v9P8nBuhHVB3C0BGnBHaWtJlSoBwAGmOVvMzJCR/hBaDbCnB27OvIRbjU3+4qErWPKZ9Xw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10768&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"1aiRaiqvOFibkrM/JSSjCJzvufibcOPV/BweRqu9NE1TTKOMTfxX/7meQm0mhGkzPsZpA01E7TD/sOUgpuVPHw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10801&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"BauhN0Vt5lLlzj2I0vWRTySU6uqEoeYBkilbTqDIl3nH9IlsrUVDmu4oFMQcrwtt"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10804&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"o3EBIPr0c0ZcyGjjhLGtU5S156e8TkmQE/omx4zSWpxN0jUy3o/vPrIz91zv/pZhLJPRxNK0HZPVzjbnMZmnPA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10803&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"xex36LJ6hs3NrKtc0smZnWbxKiHzkiCmnj1Lo3E4heuCCew6PLmc/yoE92e5dJzXMlJUjVZMn9/07Qt4Zz3t5w=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10806&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10766\",\"10767\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10805&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"Vs2VKxNtKiA0PJ0bP4XcHxxELX2RNqfaCC0F087m48+6IRw2wSZX0qa3CawbGZEvXc+rXK7S2iURJDOlHDRagA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10840&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10839\",\"10801\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10842&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10841\",\"10803\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10841&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10840\",\"10802\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_reverse_shell_script_10006&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_reverse_shell_script_10006","type":"script","value":"retainField(k8spodname,file_path,dockerimageid,gid,perm,pid,k8snodeid,host_uuid,file_gid_name,sid,uid,file_gid,k8snodename,cmdline,euid_ [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_reverse_shell_script_10007&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_reverse_shell_script_10007","type":"script","value":"___regexp_10057=regex(std_cmdline,'(mkfifo|mknod).*?&&\\s*(nc|telnet).*?<.*?\\|.*?ash');\n___regexp_10058=regex(std_cmdline,'(mkfifo|mkn [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_reverse_shell_script_10008&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_reverse_shell_script_10008","type":"script","value":"retainField(hit_result,k8spodname,file_path,dockerimageid,gid,perm,pid,k8snodeid,host_uuid,file_gid_name,sid,uid,file_gid,k8snodename,cm [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_reverse_shell_script_10009&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_reverse_shell_script_10009","type":"script","value":"___in_10024=contain(hit_result,'rs_12','rs_32','rs_33')\n;___lower_b64_cmdline_10001=lower(b64_cmdline);\n___regexp_10101=regex(___lower [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_reverse_shell_script_10010&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_reverse_shell_script_10010","type":"script","value":"retainField(is_white,k8spodname,file_path,dockerimageid,gid,perm,pid,k8snodeid,host_uuid,file_gid_name,sid,uid,file_gid,k8snodename,cmdl [...]
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_reverse_shell_rule_10003&&&&{"expressionStr":"is_white,==,double,0","scriptNames":"[]","expressionName":"10845","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_reverse_shell_rule_10003","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames" [...]
-dipper.private.blink.rules&&&&express&&&&10845&&&&{"varName":"is_white","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0.0"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_reverse_shell_script_10011&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_reverse_shell_script_10011","type":"script","value":"uuid=host_uuid;\nali_uid='';\nclient_ip='';\nevent_type='进程异常行为';\nevent_name='反弹shell命令';\nlevel='high';\n___cast_10360=cast(scan_time, [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_reverse_shell_script_10012&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_reverse_shell_script_10012","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField":"[]" [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_reverse_shell_script_10013&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_reverse_shell_script_10013","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField":"[]" [...]
-dipper.private.blink.rules&&&&channel&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_reverse_shell_channel_10001&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_reverse_shell_channel_10001","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeI [...]
-dipper.private.blink.rules&&&&pipline&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"nextStageLables\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"cancelAfterConfigurableRefreshListerner\\\\\\\":\\\\\\\"false [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq;blackrule_10002&&&&{"setCollectorMethodName":"setCollector","fullClassName":"com.sas.zing.blink.udf.BlackRule","initMethodName":"open","functionName":"blackrule","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udtf.BlinkUDTFScript","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq;blackrule_10002","type":"script","version":"1. [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq;BlackSeq_10002&&&&{"setCollectorMethodName":"setCollector","fullClassName":"com.aliyun.sec.lyra.hsh.udf.ext.BlackSeq","initMethodName":"open","functionName":"BlackSeq","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udtf.BlinkUDTFScript","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq;BlackSeq_10002","type":"script","version [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq;json_concat_10009&&&&{"fullClassName":"com.aliyun.sec.lyra.udf.ext.JsonConcat","initMethodName":"open","functionName":"json_concat","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq;json_concat_10009","t [...]
-dipper.private.blink.rules&&&&channel&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeInOrder":"true","maxF [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10001","type":"script","value":"___lower_cmdline_10092=lower(cmdline);\n___REGEXP_REPLACE_10022=REGEXP_REPLACE(___lower_cmdline_10092,'\\s+',' ');\n___!null_100 [...]
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_rule_10001&&&&{"expressionStr":"(___REGEXP_REPLACE_10022,regex,'^(((/?([a-zA-Z0-9_\\.\\-]+/){1,20})bin/)|/bin/|/|-)?(bash|sh|dash|ash|tcsh|csh|ksh|zsh)(\\s+-{1,2}[a-z0-9\\-]{1,20}){0,5}$')|((((((((cmdline,regex,'((\\W+cron)|(^|\\W+)(scp|ssh|crontab|wget|curl|base64|ifconfig|whoami|traceroute|touch|last|history|uname|arp|netstat|useradd|adduser|nslookup|ping|chmod)(\\W+|$)|/rc\\.d|auth [...]
-dipper.private.blink.rules&&&&express&&&&10855&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10854\",\"___!null_10018\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10854&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10853\",\"10850\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10846&&&&{"aesFlag":1,"varName":"___REGEXP_REPLACE_10022","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"tC+6BVeaz7RbgiE7GI5nvI+3SNAfCktVEXCRJ2PiAZV6l5KuC1EsplbweRXMpO7VdK7uu993qm1aedLTtbg2Yklh5ogU1JJ5eSOwZIQRjAEtV80MPXXh06kZgtoPkjTg5e8d6OAQg+aXWquDpF0AqofIJYMSV3i99rSh+VpdkFQ="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10857&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10856\",\"___!null_10020\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10856&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10855\",\"___!null_10019\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10848&&&&{"aesFlag":1,"varName":"file_name","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"I6Xra4uKOQ4clEQ9mBojYkMaUdfAOQ2dXE7d81U0x9csfvPcc+8lxKZRKrbebd5/JDwUfA8t8ih5S30Yht6bgDIBWdhDUO6wP2N9wP9MQUqZQzEPE6nK6kQ04Z8sBwgl22VW2p6Hyb9cLNWFF3LxQiaUsbXRDNi4fr5RngBrBigzMjSfEvKYay3j7m9XZbdJlMiNRUHyksM4G1utPL73Z+z1YPWkI08BGbnvMv0 [...]
-dipper.private.blink.rules&&&&express&&&&10859&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10846\",\"10858\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10847&&&&{"aesFlag":1,"varName":"cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"CwLJQmDR+14h1C68i/ZdDKjlUoeRATpmIGhjlfwX/gvorTB3SGEMkuWKqRHucHZ4g6mLjxqfHKJkH16NBvcUIW9YMuV9x8l1TkETAnaqguwnFY/YJzB3nPQbuRs62XKadorJP+GrKNgFzTw4UesTatNHMHOYprQDT9nWMuUOGSHguXGtZdFFbmCQT9smj8IKCu5Lga17fb93nxud8UPm+SsYrtgGPjx/MQfGqk [...]
-dipper.private.blink.rules&&&&express&&&&10858&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10857\",\"10851\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10849&&&&{"aesFlag":1,"varName":"cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"wuFzc6RuWAdEfPGMyg1TecBnjYtObODlDycixQ7BhZ2+HRmKZZL+SHqnSqZm3R78udkxlc7BUuH7BjmlFauMMiZ3RAuZ0Fhni4mz4y3caG8="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10851&&&&{"aesFlag":1,"varName":"___REGEXP_REPLACE_10023","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"tC+6BVeaz7RbgiE7GI5nvI+3SNAfCktVEXCRJ2PiAZV6l5KuC1EsplbweRXMpO7VdK7uu993qm1aedLTtbg2Yklh5ogU1JJ5eSOwZIQRjAEtV80MPXXh06kZgtoPkjTg5e8d6OAQg+aXWquDpF0AqofIJYMSV3i99rSh+VpdkFQ="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10850&&&&{"aesFlag":1,"varName":"cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"twnNM4S7+LShivhp2iyZVb4Xm4ErQA0BvqUDN741JdA="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10853&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10852\",\"10849\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10852&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10847\",\"10848\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10002","type":"script","value":"retainField(k8spodname,file_path,dockerimageid,gid,perm,pid,k8snodeid,host_uuid,file_gid_name,sid,uid,file_gid,k8snodename,cmdli [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10003&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10003","type":"script","value":"___lower_cmdline_10093=lower(cmdline);\n___REGEXP_REPLACE_10024=REGEXP_REPLACE(___lower_cmdline_10093,'\\s+',' ');\n","version": [...]
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_rule_10002&&&&{"expressionStr":"___REGEXP_REPLACE_10024,regex,'^(((/?([a-zA-Z0-9_\\.\\-]+/){1,20})bin/)|/bin/|/|-)?(bash|sh|dash|ash|tcsh|csh|ksh|zsh)(\\s+-{1,2}[a-z0-9\\-]{1,20}){0,5}$'","scriptNames":"[]","expressionName":"10860","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aeg [...]
-dipper.private.blink.rules&&&&express&&&&10860&&&&{"aesFlag":1,"varName":"___REGEXP_REPLACE_10024","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"tC+6BVeaz7RbgiE7GI5nvI+3SNAfCktVEXCRJ2PiAZV6l5KuC1EsplbweRXMpO7VdK7uu993qm1aedLTtbg2Yklh5ogU1JJ5eSOwZIQRjAEtV80MPXXh06kZgtoPkjTg5e8d6OAQg+aXWquDpF0AqofIJYMSV3i99rSh+VpdkFQ="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10004&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10004","type":"script","value":"___unixtime_10011=unixtime(scan_time);\n___division_10008=division(___unixtime_10011,3600);\n___cast_10404=cast(___division_1000 [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10005&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10005","type":"script","value":"retainField(key_time,k8spodname,file_path,dockerimageid,gid,perm,pid,k8snodeid,host_uuid,file_gid_name,sid,uid,file_gid,k8snoden [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10006&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10006","type":"script","value":"___!null_10021=!null(host_uuid);\n___!null_10022=!null(pid);\n___!null_10023=!null(parent_cmd_line);\n___lower_parent_cmd_line_1 [...]
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_rule_10003&&&&{"expressionStr":"(((((((cmdline,regex,'((\\W+cron)|(^|\\W+)(scp|ssh|crontab|wget|curl|base64|ifconfig|whoami|traceroute|touch|last|history|uname|arp|netstat|useradd|adduser|nslookup|ping|chmod)(\\W+|$)|/rc\\.d|authorized_keys|/etc/passwd|ld\\.so\\.preload|spool/cron|bash_history)')|(file_name,in,'\\'wget\\',\\'ifconfig\\',\\'whoami\\',\\'id\\',\\'curl\\',\\'base64\\',\\ [...]
-dipper.private.blink.rules&&&&express&&&&10866&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10861\",\"10862\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10865&&&&{"aesFlag":1,"varName":"___REGEXP_REPLACE_10025","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"tC+6BVeaz7RbgiE7GI5nvI+3SNAfCktVEXCRJ2PiAZV6l5KuC1EsplbweRXMpO7VdK7uu993qm1aedLTtbg2Yklh5ogU1JJ5eSOwZIQRjAEtV80MPXXh06kZgtoPkjTg5e8d6OAQg+aXWquDpF0AqofIJYMSV3i99rSh+VpdkFQ="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10868&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10867\",\"10864\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10867&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10866\",\"10863\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10869&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10868\",\"___!null_10021\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10871&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10870\",\"___!null_10023\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10870&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10869\",\"___!null_10022\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10862&&&&{"aesFlag":1,"varName":"file_name","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"I6Xra4uKOQ4clEQ9mBojYkMaUdfAOQ2dXE7d81U0x9csfvPcc+8lxKZRKrbebd5/JDwUfA8t8ih5S30Yht6bgDIBWdhDUO6wP2N9wP9MQUqZQzEPE6nK6kQ04Z8sBwgl22VW2p6Hyb9cLNWFF3LxQiaUsbXRDNi4fr5RngBrBigzMjSfEvKYay3j7m9XZbdJlMiNRUHyksM4G1utPL73Z+z1YPWkI08BGbnvMv0 [...]
-dipper.private.blink.rules&&&&express&&&&10861&&&&{"aesFlag":1,"varName":"cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"CwLJQmDR+14h1C68i/ZdDKjlUoeRATpmIGhjlfwX/gvorTB3SGEMkuWKqRHucHZ4g6mLjxqfHKJkH16NBvcUIW9YMuV9x8l1TkETAnaqguwnFY/YJzB3nPQbuRs62XKadorJP+GrKNgFzTw4UesTatNHMHOYprQDT9nWMuUOGSHguXGtZdFFbmCQT9smj8IKCu5Lga17fb93nxud8UPm+SsYrtgGPjx/MQfGqk [...]
-dipper.private.blink.rules&&&&express&&&&10872&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10871\",\"10865\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10864&&&&{"aesFlag":1,"varName":"cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"twnNM4S7+LShivhp2iyZVb4Xm4ErQA0BvqUDN741JdA="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10863&&&&{"aesFlag":1,"varName":"cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"wuFzc6RuWAdEfPGMyg1TecBnjYtObODlDycixQ7BhZ2+HRmKZZL+SHqnSqZm3R78udkxlc7BUuH7BjmlFauMMiZ3RAuZ0Fhni4mz4y3caG8="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10007&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10007","type":"script","value":"___unixtime_10012=unixtime(scan_time);\n___division_10009=division(___unixtime_10012,3600);\n___cast_10405=cast(___division_1000 [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10008&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10008","type":"script","value":"retainField(key_time,k8spodname,file_path,dockerimageid,gid,perm,pid,k8snodeid,host_uuid,file_gid_name,sid,uid,file_gid,k8snoden [...]
-dipper.private.blink.rules&&&&window&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_join_window_10001&&&&{"waterMarkMinute":"0","className":"com.aliyun.yundun.dipper.window.model.JoinWindow","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_join_window_10001","type":"window","retainWindowCount":"6","windowType":"tumble","timeout":"30000","sizeInterval":"8","isAutoFlush":"false","nameSpace":"dipper.private.blink.rules","havingMap":" [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_join_left_10001&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"entityName\\\\\\\":\\\\\\\"script\\\\\\\",\\\\\\\"scriptName\\\\\\\":\\\\\\\"subpipline_bl [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_join_left_10001_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_join_left_10001_script_10001","type":"script","value":"retainField(k8spodname,file_path,dockerimageid,gid,perm,pid,k8snodeid,host [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_join_right_10001&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"entityName\\\\\\\":\\\\\\\"script\\\\\\\",\\\\\\\"scriptName\\\\\\\":\\\\\\\"subpipline_b [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_join_right_10001_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_join_right_10001_script_10001","type":"script","value":"___!null_10024=!null(host_uuid);\n___!null_10025=!null(pid);\n___!null_1 [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_join_right_10001_rule_10001&&&&{"expressionStr":"((___!null_10024&___!null_10025)&___!null_10026)&___!null_10027","scriptNames":"[]","expressionName":"10875","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_j [...]
-dipper.private.blink.rules&&&&express&&&&10873&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"___!null_10024\",\"___!null_10025\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10875&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10874\",\"___!null_10027\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10874&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10873\",\"___!null_10026\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_join_right_10001_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_join_right_10001_script_10002","type":"script","value":"retainField(k8spodname,file_path,dockerimageid,gid,perm,pid,k8snodeid,ho [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10009&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10009","type":"script","value":"___null_10042=null(b.cmdline);\nif(___null_10042){___case_10032='';}else{___case_10032=b.cmdline;};\nsub_cmdline=___case_10032;r [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10010&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10010","type":"script","value":"docker_file_path=b.docker_file_path;\nfile_uid=b.file_uid;\nparent_cmd_line=b.parent_cmd_line;\ndockerimagename=b.dockerimagenam [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10011&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10011","type":"script","value":"___BLACKRULE_10002=BLACKRULE(sub_file_name,sub_cmdline);T.hit_result=udtf.0;T.score=udtf.1;","version":"1.0","extendField":"[]"} [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10012&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10012","type":"script","value":"___unixtime_10013=unixtime(scan_time,'yyyy-MM-dd HH:mm:ss');\n___division_10010=division(___unixtime_10013,3600);\n___floor_1000 [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10013&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10013","type":"script","value":"retainField(score,hit_result,time_win,k8spodname,file_path,dockerimageid,sub_perm,pid,k8snodename,cmdline,euid_name,parent_file_ [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10014&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10014","type":"script","value":"___lower_sub_cmdline_10001=lower(sub_cmdline);\n__compare_value_10003=!((___lower_sub_cmdline_10001,regex,'(http(s)?://100\\.100 [...]
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_rule_10004&&&&{"expressionStr":"((__compare_value_10003&___!_10022)&___!_10023)&___!_10024","scriptNames":"[]","expressionName":"10878","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_rule_10004","ruleStatus":"0","type":"rule","v [...]
-dipper.private.blink.rules&&&&express&&&&10877&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10876\",\"___!_10023\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10876&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"__compare_value_10003\",\"___!_10022\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10878&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10877\",\"___!_10024\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10015&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10015","type":"script","value":"retainField(k8spodname,file_path,dockerimageid,sub_perm,pid,score,k8snodename,cmdline,euid_name,parent_file_path,sub_scan_time,f [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10016&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10016","type":"script","value":"___!null_10028=!null(hit_result);\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_rule_10005&&&&{"expressionStr":"___!null_10028&(hit_result,<>,'')","scriptNames":"[]","expressionName":"10880","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_rule_10005","ruleStatus":"0","type":"rule","version":"1.0","extendFiel [...]
-dipper.private.blink.rules&&&&express&&&&10879&&&&{"aesFlag":1,"varName":"hit_result","functionName":"<>","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"wc8fi0p7qwBHnY1x2c6Qsw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10880&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"___!null_10028\",\"10879\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&window&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_window_10001&&&&{"groupByFieldName":"host_uuid;pid;cmdline;key_time;hit_result","waterMarkMinute":"0","className":"com.aliyun.yundun.dipper.window.model.OverWindow","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_window_10001","type":"window","version":"1.0","fireDelaySecond":"30","timeout":"30000","activtyTimeOut":"3000","sizeInterval":"60","grou [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10017&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10017","type":"script","value":"___concat_10013=concat(sub_scan_time,' ',sub_pid,' ',sub_cmdline);\ninfo=___concat_10013;rm('___concat_10013');\nrnk=over_ [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10018&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10018","type":"script","value":"retainField(score,key_time,hit_result,time_win,pid,host_uuid,sub_scan_time,rnk,ppid,info);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_rule_10006&&&&{"expressionStr":"rnk,==,double,1","scriptNames":"[]","expressionName":"10881","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_rule_10006","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNam [...]
-dipper.private.blink.rules&&&&express&&&&10881&&&&{"varName":"rnk","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"1.0"}&&&&null
-dipper.private.blink.rules&&&&window&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_window_10002&&&&{"groupByFieldName":"host_uuid;pid;ppid;key_time;time_win","waterMarkMinute":"0","sizeAdjust":"0","className":"com.aliyun.yundun.dipper.window.model.WindowMessageProcessor","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_window_10002","type":"window","version":"1.0","windowType":"hop","fireDelaySecond":"30","timeout":"30000","activ [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10019&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10019","type":"script","value":"retainField(score,key_time,hit_result,cmd_seq_detail_markdown,min_sub_scan_time,time_win,pid,max_sub_scan_time,host_uuid,ppid);" [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10020&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10020","type":"script","value":"___unixtime_10014=unixtime(max_sub_scan_time);\n___unixtime_10015=unixtime(min_sub_scan_time);\n___subtraction_10007=subtraction [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10021&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10021","type":"script","value":"retainField(score,key_time,hit_result,cmd_seq_detail_markdown,time_win,pid,sub_time_delta,host_uuid,ppid);","version":"1.0","ext [...]
-dipper.private.blink.rules&&&&window&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_join_window_10002&&&&{"waterMarkMinute":"0","className":"com.aliyun.yundun.dipper.window.model.JoinWindow","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_join_window_10002","type":"window","retainWindowCount":"6","windowType":"tumble","timeout":"30000","sizeInterval":"8","isAutoFlush":"false","nameSpace":"dipper.private.blink.rules","havingMap":" [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_join_left_10002&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"entityName\\\\\\\":\\\\\\\"script\\\\\\\",\\\\\\\"scriptName\\\\\\\":\\\\\\\"subpipline_bl [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_join_left_10002_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_join_left_10002_script_10001","type":"script","value":"___concat_10014=concat(host_uuid,pid,ppid,key_time,time_win);\njoin_key=__ [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_join_left_10002_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_join_left_10002_script_10002","type":"script","value":"retainField(join_key,k8spodname,file_path,dockerimageid,sub_perm,pid,score [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_join_right_10002&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"entityName\\\\\\\":\\\\\\\"script\\\\\\\",\\\\\\\"scriptName\\\\\\\":\\\\\\\"subpipline_b [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_join_right_10002_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_join_right_10002_script_10001","type":"script","value":"___concat_10015=concat(host_uuid,pid,ppid,key_time,time_win);\njoin_key= [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_join_right_10002_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_join_right_10002_script_10002","type":"script","value":"retainField(join_key,score,key_time,hit_result,cmd_seq_detail_markdown,t [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10022&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10022","type":"script","value":"hit_result=b.hit_result;\ncmd_seq_detail_markdown=b.cmd_seq_detail_markdown;\nscore=b.score;\nsub_time_delta=b.sub_time_delta;\n [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10023&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10023","type":"script","value":"___!null_10029=!null(hit_result);\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_rule_10007&&&&{"expressionStr":"___!null_10029&(hit_result,!in,'\\'\\',\\'N/A\\'')","scriptNames":"[]","expressionName":"10883","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_rule_10007","ruleStatus":"0","type":"rule","version": [...]
-dipper.private.blink.rules&&&&express&&&&10882&&&&{"aesFlag":1,"varName":"hit_result","functionName":"!in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"TVEW40h+v56CrjMCEeE0Kg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10883&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"___!null_10029\",\"10882\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10024&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10024","type":"script","value":"retainField(k8spodname,file_path,dockerimageid,sub_perm,pid,score,k8snodename,cmdline,euid_name,parent_file_path,sub_time_delta, [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10025&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10025","type":"script","value":"___BLACKSEQ_10002=BLACKSEQ(hit_result);T.seq_hit_result=udtf.0;T.seq_score=udtf.1;","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10026&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10026","type":"script","value":"seq_hit_result=T.seq_hit_result;\nseq_score=T.seq_score;\nretainField(seq_hit_result,seq_score,k8spodname,file_path,dockerimagei [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10027&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10027","type":"script","value":"___regexp_10104=regex(parent_file_path,'/(script|ttyrec|rootsh|sniffy|ttyrpld|ttysnoop)$');\n___regexp_10105=regex(parent_cmd_li [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10028&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10028","type":"script","value":"retainField(k8spodname,file_path,dockerimageid,sub_perm,pid,score,k8snodename,cmdline,euid_name,parent_file_path,sub_time_delta, [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_union_10001&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"entityName\\\\\\\":\\\\\\\"script\\\\\\\",\\\\\\\"scriptName\\\\\\\":\\\\\\\"subpipline_blink. [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_union_10001_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_union_10001_script_10001","type":"script","value":"___cast_10407=cast(is_white,'long');\n___cast_10408=cast(score,'long');\n","versio [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_union_10001_rule_10001&&&&{"expressionStr":"((seq_type,in,'\\'para3interactive\\',\\'para3devinteractive\\'')&(___cast_10407,==,double,0))&(___cast_10408,>=,double,6)","scriptNames":"[]","expressionName":"10888","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.a [...]
-dipper.private.blink.rules&&&&express&&&&10888&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10887\",\"10886\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10887&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10884\",\"10885\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10884&&&&{"aesFlag":1,"varName":"seq_type","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"Mk7qAUhKoGyVze118b2tIDIiRDM1FG1aAVpXHYEnN/6GbgbJXd4eFfrsmIBbB+0w"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10886&&&&{"varName":"___cast_10408","functionName":">=","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"6.0"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10885&&&&{"varName":"___cast_10407","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0.0"}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_union_10001_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_union_10001_script_10002","type":"script","value":"retainField(k8spodname,file_path,dockerimageid,sub_perm,pid,score,k8snodename,cmdl [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_union_10002&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"entityName\\\\\\\":\\\\\\\"script\\\\\\\",\\\\\\\"scriptName\\\\\\\":\\\\\\\"subpipline_blink. [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_union_10002_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_union_10002_script_10001","type":"script","value":"___cast_10409=cast(is_white,'long');\n___cast_10410=cast(seq_score,'long');\n___ca [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_union_10002_rule_10001&&&&{"expressionStr":"(((seq_type,in,'\\'para3black_parent\\',\\'para3other\\'')&(___cast_10409,==,double,0))&(___cast_10410,>,double,6))&(___cast_10411,>=,double,20)","scriptNames":"[]","expressionName":"10895","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"sub [...]
-dipper.private.blink.rules&&&&express&&&&10889&&&&{"aesFlag":1,"varName":"seq_type","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"EBiRJniz05EDVoauyOnrLJ8ZBGOjpUWyHjuu75JfU+IFto/JjA7Cfnv+PqW5AZxD"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10891&&&&{"varName":"___cast_10410","functionName":">","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"6.0"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10890&&&&{"varName":"___cast_10409","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0.0"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10893&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10889\",\"10890\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10892&&&&{"varName":"___cast_10411","functionName":">=","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"20.0"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10895&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10894\",\"10892\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10894&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10893\",\"10891\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_union_10002_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_union_10002_script_10002","type":"script","value":"retainField(k8spodname,file_path,dockerimageid,sub_perm,pid,score,k8snodename,cmdl [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_union_10002_script_10003&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_union_10002_script_10003","type":"script","value":"retainField(k8spodname,file_path,dockerimageid,sub_perm,pid,score,k8snodename,cmdl [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_union_10002_script_10004&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_union_10002_script_10004","type":"script","value":"retainField(k8spodname,file_path,dockerimageid,sub_perm,pid,score,k8snodename,cmdl [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10029&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10029","type":"script","value":"retainField(k8spodname,file_path,dockerimageid,sub_perm,pid,score,k8snodename,cmdline,euid_name,parent_file_path,sub_time_delta, [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10030&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10030","type":"script","value":"___REGEXP_REPLACE_10028=REGEXP_REPLACE(cmdline,'([^\\\\\\:\\>\\-\\&\\@\\=\\%\\s~~~~~\\/\\.\\(\\)\\[\\]]{1})','A');\n___REGEXP_RE [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10031&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10031","type":"script","value":"retainField(k8spodname,file_path,dockerimageid,sub_perm,pid,score,k8snodename,cmdline,euid_name,parent_file_path,sub_time_delta, [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10032&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10032","type":"script","value":"___md5_10008=md5(abk_raw);\nabk=___md5_10008;rm('___md5_10008');\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10033&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10033","type":"script","value":"retainField(k8spodname,file_path,dockerimageid,sub_perm,pid,score,k8snodename,cmdline,euid_name,parent_file_path,sub_time_delta, [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10034&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10034","type":"script","value":"ali_uid='';\nclient_ip='';\nevent_type='进程异常行为';\nevent_name='linux可疑命令序列';\nlevel='high';\n___cast_10412=cast(host_uuid,'string [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10035&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10035","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,host_uuid);","version":"1.0","exte [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10036&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_script_10036","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,host_uuid);","version":"1.0","exte [...]
-dipper.private.blink.rules&&&&channel&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_channel_10001&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_sus_shell_cmd_seq_channel_10001","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false"," [...]
-dipper.private.blink.rules&&&&pipline&&&&blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"nextStageLables\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"cancelAfterConfigurableRefreshListerner\\\\\\\":\\\\\\\"false\\\\\\\",\ [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert;json_concat_10010&&&&{"fullClassName":"com.aliyun.sec.lyra.udf.ext.JsonConcat","initMethodName":"open","functionName":"json_concat","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert;json_concat_10010","type":"scri [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert;extract_download_source_10001&&&&{"setCollectorMethodName":"setCollector","fullClassName":"com.self.extract_download_source","initMethodName":"open","functionName":"extract_download_source","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udtf.BlinkUDTFScript","configureName":"blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert;extract_download_source_1000 [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert;salt_hash_10002&&&&{"fullClassName":"com.aliyun.sec.lyra.hsh.udf.ext.SaltHash","initMethodName":"open","functionName":"salt_hash","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert;salt_hash_10002","type":"script", [...]
-dipper.private.blink.rules&&&&channel&&&&blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert&&&&{"offset":"0","isJsonData":"true","maxThread":"12","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.MetaqChannel","configureName":"blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert","type":"channel","version":"1.0","timeout":"30000","activtyTimeOut":"3000","tags":"T_MSG_PROC","syncTimeout":"5000","groupName":"blink.source.aegis.proc_adl_sas_apsa [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_script_10001","type":"script","value":"___len_10005=len(cmdline);\n___lower_cmdline_10095=lower(cmdline);\n___lower_file_path_10002=lower(file_path);\n___lower_file_path_10003=l [...]
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_rule_10001&&&&{"expressionStr":"((___len_10005,>,double,20)&(cmdline,like,'% %'))&(((((((___lower_cmdline_10095,regex,'(wget|curl|-fssl|cmd.exe /c|open|sh -c |python|powershell|download|reg|mshta|script|bitsadmin|certutil|msiexec|\\\\./\\\\S+)')|(file_path,==,'/usr/bin/wget'))|(file_path,==,'/usr/bin/curl'))|(___lower_file_path_10002,like,'%/powershell.exe'))|(___lower_file_path_10003,like [...]
-dipper.private.blink.rules&&&&express&&&&10907&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10906\",\"10900\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10906&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10898\",\"10899\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10909&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10908\",\"10902\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10908&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10907\",\"10901\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10899&&&&{"aesFlag":1,"varName":"file_path","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"deLRgYjbwrFSyX//riHo4w=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10910&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10909\",\"10903\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10898&&&&{"aesFlag":1,"varName":"___lower_cmdline_10095","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0uvJBH6+fDDtwCWrHQ+dPZeLoQwm14fqsWar0apbTvZqPS68JcCV/GFB/m6qt+Dc3mWMu/97JHHRHRXxcpCGyDEbl6BnhXb0u27hjkxY6BBjY6SmYIg1G1g1War6IoDV8iH8k9macCQiSissb5V9WkLZc3MDjEKPt+dS9kN6vRY="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10901&&&&{"aesFlag":1,"varName":"___lower_file_path_10002","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"NivVtcGftc/lRYpOj0WVRRSAei8gV+LJX4QrURhgtNg="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10912&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10905\",\"10911\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10900&&&&{"aesFlag":1,"varName":"file_path","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"e1sQFEbJ2l0YWqhd8z+LJQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10911&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10910\",\"10904\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10903&&&&{"aesFlag":1,"varName":"parent_file_path","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"l13p3ml25iJF9CbtbZzOYg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10902&&&&{"aesFlag":1,"varName":"___lower_file_path_10003","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"EQvgfH+iOJ2sUlmt4GPiHg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10905&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10896\",\"10897\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10904&&&&{"aesFlag":1,"varName":"parent_file_path","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"kXriqf/MFpHp8VdhL41WYw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10897&&&&{"aesFlag":1,"varName":"cmdline","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"8mcwLkiCoiJXKb4UZc1uNQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10896&&&&{"varName":"___len_10005","functionName":">","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"20.0"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_script_10002","type":"script","value":"uuid=host_uuid;\npfile_path=parent_file_path;\npcmdline=parent_cmd_line;\nuser_name=euid_name;\nlogTime=scan_time;\ncontainerid=dockercont [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_script_10003&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_script_10003","type":"script","value":"retainField(file_path,pfile_path,cmdline,user_name,pid,k8snodeid,k8sclusterid,containerid,uuid,pcmdline,ppid,logTime);","version":"1.0","e [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_script_10004&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_script_10004","type":"script","value":"___EXTRACT_DOWNLOAD_SOURCE_10001=EXTRACT_DOWNLOAD_SOURCE(cmdline);T.cmdline=udtf.0;T.url=udtf.1;","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_script_10005&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_script_10005","type":"script","value":"cmdline=cmdline;\nurl=T.url;\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_script_10006&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_script_10006","type":"script","value":"retainField(file_path,user_name,pid,k8snodeid,k8sclusterid,uuid,url,ppid,logTime,pfile_path,cmdline,containerid,pcmdline);","version":"1.0 [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_script_10007&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_script_10007","type":"script","value":"___lower_url_10001=lower(url);\n___!_10025=!((___lower_url_10001,regex,'(zabbix|notice\\\\.json|action\\\\.json|\\\\.shtml|\\\\.m3u8|51xia [...]
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_rule_10002&&&&{"expressionStr":"(___!_10025)","scriptNames":"[]","expressionName":"10914","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_rule_10002","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[]"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10914&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10913\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10913&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"___!_10025\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_script_10008&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_script_10008","type":"script","value":"retainField(file_path,user_name,pid,k8snodeid,k8sclusterid,uuid,url,ppid,logTime,pfile_path,cmdline,containerid,pcmdline);","version":"1.0 [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_script_10009&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_script_10009","type":"script","value":"___lower_url_10002=lower(url);\n___trim_10004=trim(___lower_url_10002);\nurl=___trim_10004;rm('___trim_10004');\n","version":"1.0","extend [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_script_10010&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_script_10010","type":"script","value":"retainField(file_path,user_name,pid,k8snodeid,k8sclusterid,uuid,url,ppid,logTime,pfile_path,cmdline,containerid,pcmdline);","version":"1.0 [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_script_10011&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_script_10011","type":"script","value":"___!null_10030=!null(url);\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_rule_10003&&&&{"expressionStr":"(___!null_10030)","scriptNames":"[]","expressionName":"10916","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_rule_10003","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[]"} [...]
-dipper.private.blink.rules&&&&express&&&&10916&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10915\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10915&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"___!null_10030\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_script_10012&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_script_10012","type":"script","value":"retainField(file_path,pfile_path,cmdline,user_name,pid,uuid,pcmdline,url,ppid,logTime);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_script_10013&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_script_10013","type":"script","value":"___SALT_HASH_10002=SALT_HASH(url);;","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&dataSource&&&&blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_dataSource_10001&&&&{"className":"com.aliyun.yundun.dipper.configurable.http.resource.JDBCDataSource","configureName":"blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_dataSource_10001","userName":"intelligence.rds.jdbc.username","type":"dataSource","version":"1.0","url":"intelligence.rds.jdbc.url","timeout":"30000","activtyTimeOut":"3000","password":"intelligence.rds.jd [...]
-dipper.private.blink.rules&&&&intelligence&&&&blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_intelligence_10001&&&&{"className":"com.aliyun.filter.intelligence.URLIntelligenceCache","pollingTimeMintue":"30","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_intelligence_10001","idFieldName":"id","batchSize":"3000","datasourceName":"blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_dataSource_1 [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_script_10014&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_script_10014","type":"script","value":"intelligence('dipper.private.blink.rules','blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_intelligence_10001',___SALT_HASH_ [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_script_10015&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_script_10015","type":"script","value":"___!null_10031=!null(url);\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_rule_10004&&&&{"expressionStr":"___!null_10031&(p.is_malicious_source,in,'\\'1\\'')","scriptNames":"[]","expressionName":"10918","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_rule_10004","ruleStatus":"0","type":"rule","version":"1.0","ex [...]
-dipper.private.blink.rules&&&&express&&&&10918&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"___!null_10031\",\"10917\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10917&&&&{"aesFlag":1,"varName":"p.is_malicious_source","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"jUdckUKzQ+H+FRkCrZ4LEA=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_script_10016&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_script_10016","type":"script","value":"gmt_create=logTime;\ngmt_modified=logTime;\nurl=url;\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_script_10017&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_script_10017","type":"script","value":"retainField(gmt_create,file_path,pfile_path,cmdline,user_name,pid,gmt_modified,uuid,pcmdline,url,ppid);","version":"1.0","extendField":"[] [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_script_10018&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_script_10018","type":"script","value":"ali_uid='-';\nclient_ip='-';\nevent_type='威胁情报';\nevent_name='访问恶意下载源(2)';\nlevel='high';\n___cast_10456=cast(pfile_path,'string');\n___ca [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_script_10019&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_script_10019","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&channel&&&&blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_channel_10001&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink.source.aegis.proc_adl_sas_apsara_intelligence_url_alert_channel_10001","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeInO [...]
-dipper.private.blink.rules&&&&pipline&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"nextStageLables\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10002\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"cancelAfterConfigurableRefreshListerner\\\\\\\":\\\\\\\"fal [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2;json_concat_10011&&&&{"fullClassName":"com.aliyun.sec.lyra.udf.ext.JsonConcat","initMethodName":"open","functionName":"json_concat","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2;json_concat_10011","t [...]
-dipper.private.blink.rules&&&&channel&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeInOrder":"true","maxF [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10001","type":"script","value":"logtime=scan_time;\nuuid=host_uuid;\nproc_name=file_name;\ncmd=cmdline;\npproc_name=parent_file_name;\npcmd=parent_cmd_line;\npe [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10002","type":"script","value":"retainField(pexe,pid,cmd,aliuid,proc_name,ppexe,logtime,uuid,pproc_name,pcmd,ppid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10003&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10003","type":"script","value":"___lower_proc_name_10004=lower(proc_name);\n___lower_pproc_name_10005=lower(pproc_name);\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_rule_10001&&&&{"expressionStr":"(((((___lower_proc_name_10004,like,'%.exe')|(___lower_pproc_name_10005,like,'%.exe'))|(pexe,like,'_:/%'))|(ppexe,like,'_:/%'))|(pexe,like,'//%'))|(ppexe,like,'//%')","scriptNames":"[]","expressionName":"10929","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink. [...]
-dipper.private.blink.rules&&&&express&&&&10921&&&&{"aesFlag":1,"varName":"pexe","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"iEGyh/evZuz6gUfAf8QayQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10920&&&&{"aesFlag":1,"varName":"___lower_pproc_name_10005","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"+hpefcjI0jxX3V74t6E7Ow=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10923&&&&{"aesFlag":1,"varName":"pexe","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"dZof4o+5XqLmguDMF556hA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10922&&&&{"aesFlag":1,"varName":"ppexe","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"iEGyh/evZuz6gUfAf8QayQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10925&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10919\",\"10920\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10924&&&&{"aesFlag":1,"varName":"ppexe","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"dZof4o+5XqLmguDMF556hA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10927&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10926\",\"10922\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10926&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10925\",\"10921\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10929&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10928\",\"10924\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10928&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10927\",\"10923\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10919&&&&{"aesFlag":1,"varName":"___lower_proc_name_10004","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"+hpefcjI0jxX3V74t6E7Ow=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10004&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10004","type":"script","value":"data_type='online';\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10005&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10005","type":"script","value":"retainField(pexe,data_type,pid,cmd,aliuid,proc_name,ppexe,logtime,uuid,pproc_name,pcmd,ppid);","version":"1.0","extendField":"[] [...]
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_rule_10002&&&&{"expressionStr":"data_type,==,'online'","scriptNames":"[]","expressionName":"10930","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_rule_10002","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","act [...]
-dipper.private.blink.rules&&&&express&&&&10930&&&&{"aesFlag":1,"varName":"data_type","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"MhriRxEga5GjFlRBwhEN4Q=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10006&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10006","type":"script","value":"___lower_proc_name_10005=lower(proc_name);\nclean_proc=___lower_proc_name_10005;rm('___lower_proc_name_10005');\n___lower_pproc_ [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10007&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10007","type":"script","value":"retainField(clean_proc,clean_pproc,clean_cmd,pexe,data_type,pid,cmd,aliuid,proc_name,ppexe,logtime,uuid,pproc_name,pcmd,ppid);", [...]
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_rule_10003&&&&{"expressionStr":"clean_pproc,<>,'cmd.exe'","scriptNames":"[]","expressionName":"10931","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_rule_10003","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]"," [...]
-dipper.private.blink.rules&&&&express&&&&10931&&&&{"aesFlag":1,"varName":"clean_pproc","functionName":"<>","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"/Ds+jB1sW7F7xphJLmyPRw=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10008&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10008","type":"script","value":"___compare_10041=equals(clean_proc,'cmd.exe');\n___compare_10042=equals(clean_proc,'powershell.exe');\nif(((___compare_10041&(cl [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10009&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10009","type":"script","value":"retainField(cmd_c,clean_cmd,pid,ppexe,uuid,pproc_name,ppid,clean_proc,pexe,clean_pproc,data_type,cmd,aliuid,proc_name,logtime,pc [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10010&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10010","type":"script","value":"___regexp_10122=regex(clean_cmd,'(\\s|&|~~~~~)(qprocess|tasklist)(\\.exe)?(\\s|&|~~~~~|$)');\n___compare_10043=equals(clean_proc [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10011&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10011","type":"script","value":"retainField(clean_cmd,pid,ppexe,uuid,pproc_name,ppid,clean_proc,pexe,clean_pproc,cmd_c,data_type,cmd,aliuid,proc_name,logtime,pc [...]
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_rule_10004&&&&{"expressionStr":"sub_proc,<>,''","scriptNames":"[]","expressionName":"10932","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_rule_10004","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionName [...]
-dipper.private.blink.rules&&&&express&&&&10932&&&&{"aesFlag":1,"varName":"sub_proc","functionName":"<>","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"wc8fi0p7qwBHnY1x2c6Qsw=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10012&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10012","type":"script","value":"retainField(clean_cmd,pid,ppexe,uuid,pproc_name,ppid,clean_proc,pexe,clean_pproc,cmd_c,data_type,cmd,aliuid,proc_name,logtime,pc [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10013&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10013","type":"script","value":"retainField(clean_cmd,pid,ppexe,uuid,pproc_name,ppid,clean_proc,pexe,clean_pproc,cmd_c,data_type,cmd,aliuid,proc_name,logtime,pc [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10014&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10014","type":"script","value":"___unixtime_10016=unixtime(logtime);\n___cast_10465=cast(___unixtime_10016,'long');\n___multiplication_10011=multiplication(60,6 [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10015&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10015","type":"script","value":"retainField(time_part,clean_cmd,pid,ppexe,uuid,pproc_name,ppid,clean_proc,pexe,clean_pproc,cmd_c,data_type,cmd,aliuid,proc_name, [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10016&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10016","type":"script","value":"___concat_10017=concat(uuid,' - ',time_part,' - ',pcmd,' - ',ppid);\n___md5_10009=md5(___concat_10017);\n___concat_10018=concat( [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10017&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10017","type":"script","value":"retainField(session_key,time_part,clean_cmd,pid,ppexe,uuid,pproc_name,ppid,clean_proc,pexe,clean_pproc,cmd_c,data_type,cmd,aliui [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10018&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10018","type":"script","value":"retainField(time_part,clean_cmd,pid,ppexe,uuid,pproc_name,ppid,clean_proc,pexe,clean_pproc,session_key,cmd_c,data_type,cmd,aliui [...]
-dipper.private.blink.rules&&&&window&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_window_10001&&&&{"groupByFieldName":"session_key;sub_proc","waterMarkMinute":"0","className":"com.aliyun.yundun.dipper.window.model.OverWindow","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_window_10001","type":"window","version":"1.0","fireDelaySecond":"30","timeout":"30000","activtyTimeOut":"3000","sizeInterval":"60","groupMap":"[]","slideInte [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10019&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10019","type":"script","value":"tmp_asdu1yd12=over_parition_10004;\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10020&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10020","type":"script","value":"retainField(tmp_asdu1yd12,time_part,clean_cmd,pid,ppexe,uuid,pproc_name,ppid,clean_proc,pexe,clean_pproc,session_key,cmd_c,data_ [...]
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_rule_10005&&&&{"expressionStr":"tmp_asdu1yd12,<=,double,5","scriptNames":"[]","expressionName":"10933","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_rule_10005","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]", [...]
-dipper.private.blink.rules&&&&express&&&&10933&&&&{"varName":"tmp_asdu1yd12","functionName":"<=","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"5.0"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10021&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10021","type":"script","value":"retainField(time_part,clean_cmd,pid,ppexe,uuid,pproc_name,ppid,clean_proc,tmp_asdu1yd12,pexe,clean_pproc,session_key,cmd_c,data_ [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10022&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10022","type":"script","value":"___unixtime_10017=unixtime(logtime);\n___cast_10467=cast(___unixtime_10017,'long');\n___multiplication_10013=multiplication(60,6 [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10023&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10023","type":"script","value":"retainField(time_part,clean_cmd,pid,ppexe,uuid,pproc_name,ppid,clean_proc,pexe,clean_pproc,cmd_c,data_type,cmd,aliuid,proc_name, [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10024&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10024","type":"script","value":"___concat_10019=concat(uuid,' - ',time_part,' - ',pcmd,' - ',ppid);\n___md5_10010=md5(___concat_10019);\n___concat_10020=concat( [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10025&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10025","type":"script","value":"retainField(session_key,time_part,clean_cmd,pid,ppexe,uuid,pproc_name,ppid,clean_proc,pexe,clean_pproc,cmd_c,data_type,cmd,aliui [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10026&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10026","type":"script","value":"retainField(time_part,clean_cmd,pid,ppexe,uuid,pproc_name,ppid,clean_proc,pexe,clean_pproc,session_key,cmd_c,data_type,cmd,aliui [...]
-dipper.private.blink.rules&&&&window&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_window_10002&&&&{"groupByFieldName":"session_key;sub_proc","waterMarkMinute":"0","className":"com.aliyun.yundun.dipper.window.model.OverWindow","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_window_10002","type":"window","version":"1.0","fireDelaySecond":"30","timeout":"30000","activtyTimeOut":"3000","sizeInterval":"60","groupMap":"[]","slideInte [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10027&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10027","type":"script","value":"tmp_asdu1yd12=over_parition_10005;\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10028&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10028","type":"script","value":"retainField(tmp_asdu1yd12,time_part,clean_cmd,pid,ppexe,uuid,pproc_name,ppid,clean_proc,pexe,clean_pproc,session_key,cmd_c,data_ [...]
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_rule_10006&&&&{"expressionStr":"tmp_asdu1yd12,<=,double,5","scriptNames":"[]","expressionName":"10934","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_rule_10006","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]", [...]
-dipper.private.blink.rules&&&&express&&&&10934&&&&{"varName":"tmp_asdu1yd12","functionName":"<=","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"5.0"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10029&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10029","type":"script","value":"retainField(time_part,clean_cmd,pid,ppexe,uuid,pproc_name,ppid,clean_proc,tmp_asdu1yd12,pexe,clean_pproc,session_key,cmd_c,data_ [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_union_10001&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"entityName\\\\\\\":\\\\\\\"script\\\\\\\",\\\\\\\"scriptName\\\\\\\":\\\\\\\"subpipline_blink. [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_union_10001_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_union_10001_script_10001","type":"script","value":"retainField(pid,ppexe,uuid,pproc_name,ppid,pexe,data_type,session_key,cmd,proc_nam [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_union_10002&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"entityName\\\\\\\":\\\\\\\"script\\\\\\\",\\\\\\\"scriptName\\\\\\\":\\\\\\\"subpipline_blink. [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_union_10002_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_union_10002_script_10001","type":"script","value":"retainField(pid,ppexe,uuid,pproc_name,ppid,pexe,data_type,session_key,cmd,proc_nam [...]
-dipper.private.blink.rules&&&&window&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_window_10003&&&&{"groupByFieldName":"session_key","waterMarkMinute":"0","sizeAdjust":"0","className":"com.aliyun.yundun.dipper.window.model.WindowMessageProcessor","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_window_10003","type":"window","version":"1.0","windowType":"hop","fireDelaySecond":"30","timeout":"30000","activtyTimeOut":"3000","sizeIn [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10030&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10030","type":"script","value":"retainField(cmd_cnt,sub_proc_detail,sub_proc_cnt,session_key,max_time,ppexe,uuid,pproc_name,min_time,pcmd,sub_proc_all,ppid);"," [...]
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_rule_10007&&&&{"expressionStr":"sub_proc_cnt,>=,double,2","scriptNames":"[]","expressionName":"10935","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_rule_10007","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]"," [...]
-dipper.private.blink.rules&&&&express&&&&10935&&&&{"varName":"sub_proc_cnt","functionName":">=","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"2.0"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10031&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10031","type":"script","value":"retainField(cmd_cnt,sub_proc_detail,sub_proc_cnt,session_key,max_time,ppexe,uuid,pproc_name,min_time,pcmd,sub_proc_all,ppid);"," [...]
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_rule_10008&&&&{"expressionStr":"sub_proc_cnt,>=,double,2","scriptNames":"[]","expressionName":"10936","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_rule_10008","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]"," [...]
-dipper.private.blink.rules&&&&express&&&&10936&&&&{"varName":"sub_proc_cnt","functionName":">=","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"2.0"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10032&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10032","type":"script","value":"___lower_pproc_name_10007=lower(pproc_name);\nclean_pproc=___lower_pproc_name_10007;rm('___lower_pproc_name_10007');\n","version [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10033&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10033","type":"script","value":"retainField(clean_pproc,cmd_cnt,sub_proc_detail,sub_proc_cnt,session_key,max_time,ppexe,uuid,pproc_name,min_time,pcmd,sub_proc_a [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10034&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10034","type":"script","value":"___regexp_10135=regex(clean_pproc,'^(java|tomcat|w3wp|oracle|redis|tomcat|php\\-cgi|httpd|jboss|mysqld|sqlsgent|node|openfired|n [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10035&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10035","type":"script","value":"retainField(sub_proc_detail,ppexe,uuid,pproc_name,min_time,ppid,cmd_cnt,sub_proc_cnt,clean_pproc,session_key,max_time,pcmd,sub_p [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10036&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10036","type":"script","value":"___regexp_10136=regex(sub_proc_all,'\\b(quser)\\b');\n___regexp_10137=regex(sub_proc_all,'\\b(tasklist|netstat|systeminfo|ipconf [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10037&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10037","type":"script","value":"retainField(is_intrude,sub_proc_detail,ppexe,is_vuln_pproc,uuid,pproc_name,min_time,ppid,cmd_cnt,sub_proc_cnt,clean_pproc,sessio [...]
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_rule_10009&&&&{"expressionStr":"is_intrude,==,'v3'","scriptNames":"[]","expressionName":"10937","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_rule_10009","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","action [...]
-dipper.private.blink.rules&&&&express&&&&10937&&&&{"aesFlag":1,"varName":"is_intrude","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"5vloiBx+f0j7E1ntp8Fn0g=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10038&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10038","type":"script","value":"ali_uid='';\nclient_ip='';\nevent_type='进程异常行为';\nevent_name='可疑CMD命令序列(2)';\nlevel='medium';\n___cast_10469=cast(session_key,'s [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10039&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_script_10039","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendFie [...]
-dipper.private.blink.rules&&&&channel&&&&blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_channel_10001&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_cmd_proc_shell_alert_v2_channel_10001","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false"," [...]
-dipper.private.blink.rules&&&&pipline&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"nextStageLables\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10002\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"cancelAfterConfigurableRefreshListerner\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"ownerSq [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert;sas_black_rule_v3_10002&&&&{"fullClassName":"com.lyra.xs.udf.ext.sas_black_rule_v3","initMethodName":"open","functionName":"sas_black_rule_v3","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert;sas_black_rule_v3_10002","type":"scr [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert;b64_auto_10005&&&&{"fullClassName":"com.lyra.xs.udf.ext.b64_auto","initMethodName":"open","functionName":"b64_auto","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert;b64_auto_10005","type":"script","version":"1.0","closeMethodNam [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert;instr_10003&&&&{"fullClassName":"com.lyra.xs.udf.ext.instr","initMethodName":"open","functionName":"instr","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert;instr_10003","type":"script","version":"1.0","closeMethodName":"close"," [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert;json_concat_10012&&&&{"fullClassName":"com.aliyun.sec.lyra.udf.ext.JsonConcat","initMethodName":"open","functionName":"json_concat","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert;json_concat_10012","type":"script","version":"1 [...]
-dipper.private.blink.rules&&&&channel&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeInOrder":"true","maxFetchLogGroupSize":"100","s [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10001","type":"script","value":"logtime=scan_time;\nuuid=host_uuid;\nproc_name=file_name;\ncmd=cmdline;\npproc_name=parent_file_name;\npcmd=parent_cmd_line;\npexe=file_path;\nppexe=paren [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10002","type":"script","value":"retainField(pexe,pid,cmd,aliuid,proc_name,ppexe,logtime,uuid,pproc_name,pcmd,ppid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10003&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10003","type":"script","value":"___lower_proc_name_10006=lower(proc_name);\n___lower_pproc_name_10008=lower(pproc_name);\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_rule_10001&&&&{"expressionStr":"(((((___lower_proc_name_10006,like,'%.exe')|(___lower_pproc_name_10008,like,'%.exe'))|(pexe,like,'_:/%'))|(ppexe,like,'_:/%'))|(pexe,like,'//%'))|(ppexe,like,'//%')","scriptNames":"[]","expressionName":"10948","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis. [...]
-dipper.private.blink.rules&&&&express&&&&10943&&&&{"aesFlag":1,"varName":"ppexe","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"dZof4o+5XqLmguDMF556hA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10942&&&&{"aesFlag":1,"varName":"pexe","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"dZof4o+5XqLmguDMF556hA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10945&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10944\",\"10940\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10944&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10938\",\"10939\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10947&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10946\",\"10942\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10946&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10945\",\"10941\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10938&&&&{"aesFlag":1,"varName":"___lower_proc_name_10006","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"+hpefcjI0jxX3V74t6E7Ow=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10948&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10947\",\"10943\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10939&&&&{"aesFlag":1,"varName":"___lower_pproc_name_10008","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"+hpefcjI0jxX3V74t6E7Ow=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10941&&&&{"aesFlag":1,"varName":"ppexe","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"iEGyh/evZuz6gUfAf8QayQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10940&&&&{"aesFlag":1,"varName":"pexe","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"iEGyh/evZuz6gUfAf8QayQ=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10004&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10004","type":"script","value":"data_type='online';\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10005&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10005","type":"script","value":"retainField(pexe,data_type,pid,cmd,aliuid,proc_name,ppexe,logtime,uuid,pproc_name,pcmd,ppid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_rule_10002&&&&{"expressionStr":"data_type,==,'online'","scriptNames":"[]","expressionName":"10949","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_rule_10002","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[]"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10949&&&&{"aesFlag":1,"varName":"data_type","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"MhriRxEga5GjFlRBwhEN4Q=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10006&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10006","type":"script","value":"retainField(pexe,data_type,pid,cmd,aliuid,proc_name,ppexe,logtime,uuid,pproc_name,pcmd,ppid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10007&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10007","type":"script","value":"___unixtime_10018=unixtime(logtime);\nunixtime=___unixtime_10018;rm('___unixtime_10018');\n___lower_proc_name_10007=lower(proc_name);\nclean_proc=___lower [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10008&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10008","type":"script","value":"retainField(unixtime,clean_proc,clean_cmd_noquote,clean_pproc,pexe,data_type,pid,cmd,aliuid,proc_name,ppexe,logtime,uuid,pproc_name,pcmd,ppid);","version" [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10009&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10009","type":"script","value":"___multiplication_10017=multiplication(60,60);\n___multiplication_10018=multiplication(___multiplication_10017,24);\n___division_10013=division(unixtime,_ [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10010&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10010","type":"script","value":"retainField(shot_pcmd,timepart_1hour,log_uid,timepart_1day,unixtime,pid,ppexe,uuid,pproc_name,ppid,clean_proc,clean_cmd_noquote,pexe,clean_pproc,data_type [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10001&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.FilterChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"names\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10001_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\" [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10001_rule_10001&&&&{"expressionStr":"data_type,<>,'online'","scriptNames":"[]","expressionName":"10950","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10001_rule_10001","ruleStatus":"0","type":"rule","version":"1.0","ex [...]
-dipper.private.blink.rules&&&&express&&&&10950&&&&{"aesFlag":1,"varName":"data_type","functionName":"<>","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"MhriRxEga5GjFlRBwhEN4Q=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10001_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10001_script_10001","type":"script","value":"p2_logtime='';\np2_proc_name='';\np2_cmd='';\np2_pproc_name='';\np2_pcmd='';\np2_pexe='';\np2_ppexe='';\np2_ [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10001_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10001_script_10002","type":"script","value":"retainField(p2_pid,p2_ppid,p2_pexe,pid,p3_ppexe,ppexe,uuid,p3_pid,pexe,aliuid,p2_proc_name,p2_pcmd,p2_ppexe, [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10002&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"entityName\\\\\\\":\\\\\\\"script\\\\\\\",\\\\\\\"scriptName\\\\\\\":\\\\\\\"subpipline_blink.source.aegis. [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10002_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10002_script_10001","type":"script","value":"___null_10044=null(p2_logtime);\nif(___null_10044){___if_v3_10019=''}else{___if_v3_10019=p2_logtime};\np2_lo [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10002_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10002_script_10002","type":"script","value":"retainField(p2_pid,p2_ppid,p2_pexe,pid,p3_ppexe,ppexe,uuid,p3_pid,pexe,aliuid,p2_proc_name,p2_pcmd,p2_ppexe, [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10003&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.FilterChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"names\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10003_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\" [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10003_rule_10001&&&&{"expressionStr":"step_2_continue,==,boolean,false","scriptNames":"[]","expressionName":"10951","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10003_rule_10001","ruleStatus":"0","type":"rule","version [...]
-dipper.private.blink.rules&&&&express&&&&10951&&&&{"varName":"step_2_continue","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Boolean\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"false"}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10003_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10003_script_10001","type":"script","value":"p2_logtime='';\np2_proc_name='';\np2_cmd='';\np2_pproc_name='';\np2_pcmd='';\np2_pexe='';\np2_ppexe='';\np2_ [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10003_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10003_script_10002","type":"script","value":"retainField(p2_pid,p2_ppid,p2_pexe,pid,p3_ppexe,ppexe,uuid,p3_pid,pexe,aliuid,p2_proc_name,p2_pcmd,p2_ppexe, [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10004&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.FilterChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"names\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10004_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\" [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10004_rule_10001&&&&{"expressionStr":"step_3_continue,==,boolean,false","scriptNames":"[]","expressionName":"10952","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10004_rule_10001","ruleStatus":"0","type":"rule","version [...]
-dipper.private.blink.rules&&&&express&&&&10952&&&&{"varName":"step_3_continue","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Boolean\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"false"}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10004_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10004_script_10001","type":"script","value":"___null_10062=null(p2_logtime);\nif(___null_10062){___if_v_tmp_ayd871y7dy12_10010=''}else{___if_v_tmp_ayd871 [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10004_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10004_script_10002","type":"script","value":"retainField(p2_pid,p2_ppid,p2_pexe,pid,p3_ppexe,ppexe,uuid,p3_pid,pexe,aliuid,p2_proc_name,p2_pcmd,p2_ppexe, [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10005&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.FilterChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"names\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10005_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\" [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10005_rule_10001&&&&{"expressionStr":"chain_level,in,'\\'p2\\',\\'p3\\''","scriptNames":"[]","expressionName":"10953","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10005_rule_10001","ruleStatus":"0","type":"rule","versi [...]
-dipper.private.blink.rules&&&&express&&&&10953&&&&{"aesFlag":1,"varName":"chain_level","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"H2zd1kq1OIhVTUDIiE4pYw=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10005_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10005_script_10001","type":"script","value":"retainField(p2_pid,p2_ppid,p2_pexe,pid,p3_ppexe,ppexe,uuid,chain_pcmd,p3_pid,pexe,aliuid,p2_proc_name,p2_pcm [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10006&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"entityName\\\\\\\":\\\\\\\"script\\\\\\\",\\\\\\\"scriptName\\\\\\\":\\\\\\\"subpipline_blink.source.aegis. [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10006_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10006_script_10001","type":"script","value":"p2_logtime='';\np2_proc_name='';\np2_cmd='';\np2_pproc_name='';\np2_pcmd='';\np2_pexe='';\np2_ppexe='';\np2_ [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10006_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10006_script_10002","type":"script","value":"retainField(p2_pid,p2_ppid,p2_pexe,pid,p3_ppexe,ppexe,uuid,chain_pcmd,p3_pid,pexe,aliuid,p2_proc_name,p2_pcm [...]
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_rule_10003&&&&{"expressionStr":"clean_proc,in,'\\'cmd.exe\\',\\'powershell.exe\\''","scriptNames":"[]","expressionName":"10954","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_rule_10003","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]"," [...]
-dipper.private.blink.rules&&&&express&&&&10954&&&&{"aesFlag":1,"varName":"clean_proc","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"YGeHkwJkc4xJFjK1BtvAndrjJb4sJfqN+JRKfoTDfew="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10011&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10011","type":"script","value":"retainField(unixtime,shot_pcmd,timepart_1day,pid,ppexe,uuid,pproc_name,ppid,clean_proc,timepart_1hour,clean_cmd_noquote,log_uid,pexe,clean_pproc,data_type [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10012&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10012","type":"script","value":"___lower_pproc_name_10010=lower(pproc_name);\n___in_10033=contain(___lower_pproc_name_10010,'cmd.exe','powershell.exe')\n;if(___in_10033){___if_v0_10002=t [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10013&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10013","type":"script","value":"retainField(step_2_continue,unixtime,shot_pcmd,timepart_1day,pid,ppexe,uuid,pproc_name,ppid,clean_proc,timepart_1hour,clean_cmd_noquote,log_uid,pexe,clean [...]
-dipper.private.blink.rules&&&&window&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_join_window_10001&&&&{"waterMarkMinute":"0","className":"com.aliyun.yundun.dipper.window.model.JoinWindow","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_join_window_10001","type":"window","retainWindowCount":"6","windowType":"tumble","timeout":"30000","sizeInterval":"8","isAutoFlush":"false","nameSpace":"dipper.private.blink.rules","havingMap":"[]","extendField":"[]","gr [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_join_left_10001&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.FilterChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"names\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_join_left_10001_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\", [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_join_left_10001_rule_10001&&&&{"expressionStr":"step_2_continue,==,boolean,true","scriptNames":"[]","expressionName":"10955","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_join_left_10001_rule_10001","ruleStatus":"0","type":"rule"," [...]
-dipper.private.blink.rules&&&&express&&&&10955&&&&{"varName":"step_2_continue","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Boolean\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"true"}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_join_left_10001_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_join_left_10001_script_10001","type":"script","value":"___uuid_10004=uuid();\n___concat_10022=concat(uuid,___uuid_10004);\nthe_uuid_dayd71y28y17=___concat_ [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_join_left_10001_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_join_left_10001_script_10002","type":"script","value":"retainField(the_uuid_dayd71y28y17,unixtime,step_2_continue,shot_pcmd,timepart_1day,pid,ppexe,uuid,pp [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_join_right_10001&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.WindowChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"entityName\\\\\\\":\\\\\\\"window\\\\\\\",\\\\\\\"lable\\\\\\\":\\\\\\\"blink.source.aegis.proc_adl_sa [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10014&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10014","type":"script","value":"p2_logtime=b.logtime;\np2_unixtime=b.unixtime;\np2_uuid=b.uuid;\np2_proc_name=b.proc_name;\np2_cmd=b.cmd;\np2_pproc_name=b.pproc_name;\np2_pcmd=b.pcmd;\np [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10015&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10015","type":"script","value":"aliuid=b.aliuid;\nclean_cmd_noquote=b.clean_cmd_noquote;\ntimepart_1day=b.timepart_1day;\ntimepart_1hour=b.timepart_1hour;\ndata_type=b.data_type;\nretain [...]
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_rule_10004&&&&{"expressionStr":"r_Dau8d192yd712yd7,==,double,1","scriptNames":"[]","expressionName":"10956","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_rule_10004","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[]"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10956&&&&{"varName":"r_Dau8d192yd712yd7","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"1.0"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10016&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10016","type":"script","value":"retainField(unixtime,p2_pid,shot_pcmd,p2_ppid,p2_pexe,p2_clean_proc,timepart_1day,pid,ppexe,uuid,p2_unixtime,timepart_1hour,pexe,aliuid,p2_proc_name,p2_pc [...]
-dipper.private.blink.rules&&&&window&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_join_window_10002&&&&{"waterMarkMinute":"0","className":"com.aliyun.yundun.dipper.window.model.JoinWindow","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_join_window_10002","type":"window","retainWindowCount":"6","windowType":"tumble","timeout":"30000","sizeInterval":"8","isAutoFlush":"false","nameSpace":"dipper.private.blink.rules","havingMap":"[]","extendField":"[]","gr [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_join_left_10002&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.FilterChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"names\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_join_left_10002_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\", [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_join_left_10002_rule_10001&&&&{"expressionStr":"step_3_continue,==,boolean,true","scriptNames":"[]","expressionName":"10957","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_join_left_10002_rule_10001","ruleStatus":"0","type":"rule"," [...]
-dipper.private.blink.rules&&&&express&&&&10957&&&&{"varName":"step_3_continue","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Boolean\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"true"}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_join_left_10002_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_join_left_10002_script_10001","type":"script","value":"retainField(unixtime,p2_pid,shot_pcmd,p2_ppid,p2_pexe,timepart_1day,p2_clean_proc,pid,ppexe,uuid,p2_ [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_join_right_10002&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.WindowChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"entityName\\\\\\\":\\\\\\\"window\\\\\\\",\\\\\\\"lable\\\\\\\":\\\\\\\"blink.source.aegis.proc_adl_sa [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10017&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10017","type":"script","value":"p3_logtime=b.logtime;\np3_unixtime=b.unixtime;\np3_uuid=b.uuid;\np3_proc_name=b.proc_name;\np3_cmd=b.cmd;\np3_pproc_name=b.pproc_name;\np3_pcmd=b.pcmd;\np [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10018&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10018","type":"script","value":"clean_proc=b.clean_proc;\nclean_cmd_noquote=b.clean_cmd_noquote;\ntimepart_1day=b.timepart_1day;\ndata_type=b.data_type;\nlog_uid=b.log_uid;\naliuid=b.ali [...]
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_rule_10005&&&&{"expressionStr":"r_Dadu1y2871yd2821tg28,==,double,1","scriptNames":"[]","expressionName":"10958","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_rule_10005","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[] [...]
-dipper.private.blink.rules&&&&express&&&&10958&&&&{"varName":"r_Dadu1y2871yd2821tg28","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"1.0"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10019&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10019","type":"script","value":"retainField(unixtime,p2_pid,shot_pcmd,p2_ppid,p2_pexe,p3_uuid,p3_shot_pcmd,pid,uuid,p2_unixtime,timepart_1hour,p3_pid,step_3_continue,pexe,p2_proc_name,r_ [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10020&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10020","type":"script","value":"___lower_p2_pproc_name_10002=lower(p2_pproc_name);\n___in_10034=contain(___lower_p2_pproc_name_10002,'cmd.exe','powershell.exe')\n;if(___in_10034){___if_v [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10021&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10021","type":"script","value":"retainField(step_3_continue,unixtime,p2_pid,shot_pcmd,p2_ppid,p2_pexe,timepart_1day,p2_clean_proc,pid,ppexe,uuid,p2_unixtime,timepart_1hour,pexe,p2_proc_n [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10007&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"entityName\\\\\\\":\\\\\\\"script\\\\\\\",\\\\\\\"scriptName\\\\\\\":\\\\\\\"subpipline_blink.source.aegis. [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10007_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10007_script_10001","type":"script","value":"chain_level='p1';\nchain_pproc_name=pproc_name;\nchain_pcmd=pcmd;\nchain_ppexe=ppexe;\nchain_ppid=ppid;\n"," [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10007_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10007_script_10002","type":"script","value":"retainField(chain_pcmd,chain_ppid,chain_pproc_name,p2_pid,p2_ppid,p2_pexe,pid,p3_ppexe,uuid,p3_pid,pexe,aliu [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10008&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.FilterChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"names\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10008_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\" [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10008_rule_10001&&&&{"expressionStr":"p2_pproc_name,<>,''","scriptNames":"[]","expressionName":"10959","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10008_rule_10001","ruleStatus":"0","type":"rule","version":"1.0","exte [...]
-dipper.private.blink.rules&&&&express&&&&10959&&&&{"aesFlag":1,"varName":"p2_pproc_name","functionName":"<>","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"wc8fi0p7qwBHnY1x2c6Qsw=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10008_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10008_script_10001","type":"script","value":"chain_level='p2';\nchain_pproc_name=p2_pproc_name;\nchain_pcmd=p2_pcmd;\nchain_ppexe=p2_ppexe;\nchain_ppid=p [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10008_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10008_script_10002","type":"script","value":"retainField(chain_pcmd,chain_ppid,chain_pproc_name,p2_pid,p2_pexe,pid,p3_ppexe,ppexe,uuid,p3_pid,pexe,aliuid [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10009&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.FilterChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"names\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10009_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\" [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10009_rule_10001&&&&{"expressionStr":"p3_pproc_name,<>,''","scriptNames":"[]","expressionName":"10960","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10009_rule_10001","ruleStatus":"0","type":"rule","version":"1.0","exte [...]
-dipper.private.blink.rules&&&&express&&&&10960&&&&{"aesFlag":1,"varName":"p3_pproc_name","functionName":"<>","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"wc8fi0p7qwBHnY1x2c6Qsw=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10009_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10009_script_10001","type":"script","value":"chain_level='p3';\nchain_pproc_name=p3_pproc_name;\nchain_pcmd=p3_pcmd;\nchain_ppexe=p3_ppexe;\nchain_ppid=p [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10009_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_union_10009_script_10002","type":"script","value":"retainField(chain_pcmd,chain_ppid,chain_pproc_name,p2_pid,p2_ppid,p2_pexe,pid,ppexe,uuid,p3_pid,pexe,aliuid, [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10022&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10022","type":"script","value":"retainField(chain_ppid,p2_pid,p2_ppid,p2_pexe,pid,p3_ppexe,ppexe,uuid,chain_pcmd,p3_pid,pexe,aliuid,p2_proc_name,p2_pcmd,p2_ppexe,p3_cmd,p3_ppid,p3_proc_n [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10023&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10023","type":"script","value":"___REGEXP_EXTRACT_10005=REGEXP_EXTRACT(cmd,'([a-zA-Z0-9\\/+=]{80,})',1);\ncmd_b64_raw=___REGEXP_EXTRACT_10005;rm('___REGEXP_EXTRACT_10005');\n___lower_pro [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10024&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10024","type":"script","value":"retainField(clean_proc,log_uuid,clean_pexe,clean_chain_pproc,clean_chain_pcmd,cmd_b64_raw,clean_pproc,clean_cmd,clean_chain_ppexe,p2_pid,p2_ppid,p2_pexe,p [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10025&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10025","type":"script","value":"___null_10071=null(cmd_b64_raw);\nif(___null_10071){___if_proc_log_union_10004=''}else{___if_proc_log_union_10004=cmd_b64_raw};\n___B64_AUTO_10004=B64_AUT [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10026&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10026","type":"script","value":"retainField(b64_decode_raw,p2_pid,p2_ppid,p2_pexe,clean_cmd,pid,p3_ppexe,ppexe,uuid,chain_pcmd,log_uuid,clean_chain_pproc,clean_chain_pcmd,p3_pid,pexe,cmd [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10027&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10027","type":"script","value":"___lower_b64_decode_raw_10002=lower(b64_decode_raw);\nb64_decode=___lower_b64_decode_raw_10002;rm('___lower_b64_decode_raw_10002');\n___!null_10032=!null( [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10028&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10028","type":"script","value":"retainField(b64_decode,p2_pid,p2_ppid,p2_pexe,clean_cmd,pid,p3_ppexe,script_path,ppexe,uuid,chain_pcmd,log_uuid,clean_chain_pproc,clean_chain_pcmd,p3_pid, [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10029&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10029","type":"script","value":"___compare_10056=equals(b64_decode,'');\n___concat_10023=concat(clean_cmd,' / ',b64_decode);\nif(___compare_10056){___if_proc_log_union_10006=clean_cmd}el [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10030&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10030","type":"script","value":"retainField(p2_pid,p2_ppid,p2_pexe,pid,script_path,uuid,chain_pcmd,p3_pid,pexe,clean_chain_ppexe,p2_proc_name,b64_decode_raw,p2_ppexe,clean_pexe,p3_cmd,p3 [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10031&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10031","type":"script","value":"_sep1='|';\nblack_feature_number=234;\ntmp_1='1';\ntmp_2='2';\ntmp_3='3';\ntmp_4='4';\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10032&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10032","type":"script","value":"retainField(_sep1,black_feature_number,tmp_4,p2_pid,p2_ppid,p2_pexe,pid,script_path,uuid,chain_pcmd,p3_pid,pexe,clean_chain_ppexe,p2_proc_name,b64_decode_ [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10033&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10033","type":"script","value":"___cast_10481=cast(black_feature_number,'string');\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10034&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10034","type":"script","value":"retainField(p2_pid,_sep1,p2_ppid,p2_pexe,pid,script_path,uuid,chain_pcmd,p3_pid,pexe,clean_chain_ppexe,tmp_4,p2_proc_name,tmp_1,tmp_3,tmp_2,b64_decode_raw [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10035&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10035","type":"script","value":"___compare_10057=equals(clean_proc,'powershell.exe');\n___compare_10058=equals(clean_proc,'cmd.exe');\n___regexp_10162=regex(clean_cmd,'/c.+\\b(powershell [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10036&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10036","type":"script","value":"retainField(cmd_msbuild,p2_pid,_sep1,p2_ppid,p2_pexe,pid,script_path,uuid,chain_pcmd,p3_pid,pexe,clean_chain_ppexe,tmp_4,p2_proc_name,tmp_1,tmp_3,tmp_2,b6 [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10037&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10037","type":"script","value":"___regexp_10174=regex(clean_chain_pproc,'^javaw?\\.exe$');\n___compare_10083=equals(clean_chain_pproc,'jbossservice.exe');\n___regexp_10175=regex(clean_ch [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10038&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10038","type":"script","value":"retainField(ext_0,p2_pid,_sep1,p2_ppid,p2_pexe,pid,script_path,uuid,chain_pcmd,cmd_wscript,p3_pid,pexe,clean_chain_ppexe,tmp_4,cmd_cscript,p2_proc_name,tm [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10039&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10039","type":"script","value":"___!null_10033=!null(ext_0);\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_rule_10006&&&&{"expressionStr":"___!null_10033&(ext_0,<>,'')","scriptNames":"[]","expressionName":"10962","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_rule_10006","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[]"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10961&&&&{"aesFlag":1,"varName":"ext_0","functionName":"<>","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"wc8fi0p7qwBHnY1x2c6Qsw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10962&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"___!null_10033\",\"10961\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10040&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10040","type":"script","value":"retainField(p2_pid,_sep1,p2_ppid,p2_pexe,pid,script_path,uuid,chain_pcmd,cmd_wscript,p3_pid,pexe,clean_chain_ppexe,tmp_4,cmd_cscript,p2_proc_name,tmp_1,tm [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10041&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10041","type":"script","value":"___cast_10482=cast(uuid,'string');\nuuid=___cast_10482;rm('___cast_10482');\n___cast_10483=cast(logtime,'string');\nlogtime=___cast_10483;rm('___cast_1048 [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10042&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10042","type":"script","value":"retainField(p2_pid,_sep1,p2_ppid,p2_pexe,pid,script_path,uuid,chain_pcmd,cmd_wscript,p3_pid,pexe,tmp_4,clean_chain_ppexe,cmd_cscript,p2_proc_name,tmp_1,tm [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10043&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10043","type":"script","value":"___concat_10026=concat(clean_cmd,p2_cmd,p2_pexe,p2_ppexe,p3_cmd,p3_pcmd);\n___lower____concat_10026_10001=lower(___concat_10026);\n___!_10044=!((((((((((( [...]
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_rule_10007&&&&{"expressionStr":"(___!_10044)","scriptNames":"[]","expressionName":"10964","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_rule_10007","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[]"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10964&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10963\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10963&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"___!_10044\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10044&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10044","type":"script","value":"_tmp_qweyhkuyiyiuy='@';\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10045&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10045","type":"script","value":"retainField(_tmp_qweyhkuyiyiuy,p2_pid,_sep1,p2_ppid,p2_pexe,pid,script_path,uuid,chain_pcmd,cmd_wscript,p3_pid,pexe,tmp_4,clean_chain_ppexe,cmd_cscript,p2 [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10046&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10046","type":"script","value":"if((ext_0,like,'%|cmd_cmd_obfs|%')){___case_10082='命令混淆';}else{___case_10082='';};\nif((ext_0,like,'%|tool_cmd_pingswap|%')){___case_10083='no_body';}else [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10047&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10047","type":"script","value":"retainField(p2_pid,_sep1,p2_ppid,p2_pexe,pid,script_path,uuid,chain_pcmd,cmd_wscript,p3_pid,pexe,tmp_4,clean_chain_ppexe,cmd_cscript,p2_proc_name,tmp_1,tm [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10048&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10048","type":"script","value":"___STRING_SPLIT_10002=STRING_SPLIT(threat_type_all,_tmp_qweyhkuyiyiuy);T.v=udtf.0;","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10049&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10049","type":"script","value":"___trim_10005=trim(T.v);\nthreat_type=___trim_10005;rm('___trim_10005');\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10050&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10050","type":"script","value":"retainField(threat_type,p2_pid,_sep1,p2_ppid,p2_pexe,pid,script_path,uuid,chain_pcmd,cmd_wscript,p3_pid,pexe,tmp_4,clean_chain_ppexe,cmd_cscript,p2_proc_n [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10051&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10051","type":"script","value":"___!null_10034=!null(threat_type);\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_rule_10008&&&&{"expressionStr":"___!null_10034&(threat_type,regex,'\\S+')","scriptNames":"[]","expressionName":"10966","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_rule_10008","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNam [...]
-dipper.private.blink.rules&&&&express&&&&10965&&&&{"aesFlag":1,"varName":"threat_type","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"BhPVyKanBl/NbolFClSKRQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10966&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"___!null_10034\",\"10965\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10052&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10052","type":"script","value":"retainField(p2_pid,_sep1,p2_ppid,p2_pexe,pid,script_path,uuid,chain_pcmd,cmd_wscript,p3_pid,pexe,tmp_4,clean_chain_ppexe,cmd_cscript,p2_proc_name,tmp_1,tm [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10053&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10053","type":"script","value":"the_version='20210120';\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10054&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10054","type":"script","value":"retainField(p2_pid,p2_ppid,p2_pexe,pid,p3_ppexe,script_path,ppexe,uuid,chain_pcmd,p3_pid,ext_0,pexe,threat_type,aliuid,p2_proc_name,p2_pcmd,threat_type_al [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10055&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10055","type":"script","value":"___!null_10035=!null(threat_type);\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_rule_10009&&&&{"expressionStr":"___!null_10035&(threat_type,<>,'')","scriptNames":"[]","expressionName":"10968","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_rule_10009","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[] [...]
-dipper.private.blink.rules&&&&express&&&&10967&&&&{"aesFlag":1,"varName":"threat_type","functionName":"<>","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"wc8fi0p7qwBHnY1x2c6Qsw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10968&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"___!null_10035\",\"10967\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10056&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10056","type":"script","value":"___compare_10218=equals(p2_pproc_name,'');\n___compare_10219=equals(p3_pproc_name,'');\n___concat_10027=concat('[',p2_pproc_name,'] -> [',pproc_name,'] -> [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10057&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10057","type":"script","value":"retainField(p2_pid,p2_ppid,p2_pexe,pid,p3_ppexe,script_path,ppexe,uuid,chain_pcmd,p3_pid,ext_0,pexe,threat_type,aliuid,p2_proc_name,p2_pcmd,threat_type_al [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10058&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10058","type":"script","value":"___lower_pcmd_10005=lower(pcmd);\n___lower_cmd_10020=lower(cmd);\n___lower_cmd_10021=lower(cmd);\n___!_10070=!(((aliuid,==,'1142026142908714')&(((___lower [...]
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_rule_10010&&&&{"expressionStr":"(___!_10070)","scriptNames":"[]","expressionName":"10970","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_rule_10010","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[]"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10969&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"___!_10070\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10970&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10969\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10059&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10059","type":"script","value":"___in_10065=contain(threat_type,'uninstall_av','修改defender配置')\n;___in_10066=contain(threat_type,'可疑的注册表操作','可疑的注册表操作_2')\n;___in_10067=contain(threat_typ [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10060&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10060","type":"script","value":"retainField(event_name,p2_pid,p2_ppid,p2_pexe,pid,p3_ppexe,script_path,ppexe,uuid,chain_pcmd,p3_pid,ext_0,pexe,threat_type,aliuid,p2_proc_name,p2_pcmd,thr [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10061&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10061","type":"script","value":"___md5_10011=md5(abk_raw);\nabk=___md5_10011;rm('___md5_10011');\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10062&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10062","type":"script","value":"retainField(p2_pid,p2_ppid,p2_pexe,pid,p3_ppexe,script_path,ppexe,uuid,chain_pcmd,p3_pid,ext_0,pexe,threat_type,aliuid,p2_proc_name,abk_raw,p2_pcmd,threat [...]
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_rule_10011&&&&{"expressionStr":"event_name,<>,''","scriptNames":"[]","expressionName":"10971","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_rule_10011","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[]"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10971&&&&{"aesFlag":1,"varName":"event_name","functionName":"<>","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"wc8fi0p7qwBHnY1x2c6Qsw=="}&&&&null
-dipper.private.blink.rules&&&&window&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_window_10001&&&&{"groupByFieldName":"uuid;event_name;abk;___cast_10559","waterMarkMinute":"0","className":"com.aliyun.yundun.dipper.window.model.OverWindow","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_window_10001","type":"window","version":"1.0","fireDelaySecond":"30","timeout":"30000","activtyTimeOut":"3000","sizeInterval":"60","groupMap":"[]","slideInterval":"60","i [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10063&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10063","type":"script","value":"___unixtime_10019=unixtime(logtime);\n___multiplication_10023=multiplication(30,1);\n___division_10015=division(___unixtime_10019,___multiplication_10023) [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10064&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10064","type":"script","value":"retainField(r_dasud891ud912,p2_pid,p2_ppid,p2_pexe,pid,p3_ppexe,script_path,ppexe,uuid,chain_pcmd,p3_pid,ext_0,pexe,threat_type,aliuid,p2_proc_name,abk_ra [...]
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_rule_10012&&&&{"expressionStr":"r_dasud891ud912,==,double,1","scriptNames":"[]","expressionName":"10972","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_rule_10012","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[]"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10972&&&&{"varName":"r_dasud891ud912","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"1.0"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10065&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10065","type":"script","value":"retainField(p2_pid,p2_ppid,p2_pexe,pid,p3_ppexe,script_path,ppexe,uuid,chain_pcmd,p3_pid,ext_0,pexe,threat_type,aliuid,p2_proc_name,abk_raw,p2_pcmd,threat [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10066&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10066","type":"script","value":"ali_uid='';\nclient_ip='';\nevent_type='进程异常行为';\nlevel='high';\n___cast_10560=cast(uuid,'string');\n___cast_10561=cast(logtime,'string');\n___cast_10562= [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10067&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_script_10067","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&channel&&&&blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_channel_10001&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink.source.aegis.proc_adl_sas_apsara_win_proc_alert_channel_10001","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeInOrder":"true","ma [...]
-dipper.private.blink.rules&&&&pipline&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"nextStageLables\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"blink.source.aegis.proc_sas_linux_alert_proc_cmd_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"cancelAfterConfigurableRefreshListerner\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"ownerSqlNodeTableNa [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd;json_concat_10013&&&&{"fullClassName":"com.aliyun.sec.lyra.udf.ext.JsonConcat","initMethodName":"open","functionName":"json_concat","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd;json_concat_10013","type":"script","version":"1.0","close [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd;b64_auto_10006&&&&{"fullClassName":"com.lyra.xs.udf.ext.b64_auto","initMethodName":"open","functionName":"b64_auto","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd;b64_auto_10006","type":"script","version":"1.0","closeMethodName":"close" [...]
-dipper.private.blink.rules&&&&channel&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd&&&&{"offset":"0","isJsonData":"true","maxThread":"12","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.MetaqChannel","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd","type":"channel","version":"1.0","timeout":"30000","activtyTimeOut":"3000","syncTimeout":"5000","groupName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd","pullIntervalMs":"100","isBatchM [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_script_10001","type":"script","value":"___lower_cwd_10001=lower(cwd);\n__compare_value_10007=!((___lower_cwd_10001,regex,'aliyun-assist|securecheck|aegis|aliyun|beaver'))\n;","version":"1.0","extendField [...]
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_rule_10001&&&&{"expressionStr":"(__compare_value_10007)","scriptNames":"[]","expressionName":"10974","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_rule_10001","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[]"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10974&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10973\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10973&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"__compare_value_10007\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_script_10002","type":"script","value":"uuid=host_uuid;\nfilename=file_name;\npfilename=parent_file_name;\npcmdline=parent_cmd_line;\nfilepath=file_path;\npfilepath=parent_file_path;\n___REGEXP_EXTRACT_10 [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_script_10003&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_script_10003","type":"script","value":"retainField(k8spodname,file_path,dockerimageid,gid,perm,pid,k8snodeid,uuid,file_gid_name,sid,uid,file_gid,k8snodename,cmdline,pfilename,filepath,euid_name,egroup_na [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_script_10004&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_script_10004","type":"script","value":"___lower_cmdline_10096=lower(cmdline);\n___REGEXP_REPLACE_10042=REGEXP_REPLACE(___lower_cmdline_10096,'\\s+',' ');\nstd_cmdline=___REGEXP_REPLACE_10042;rm('___REGEX [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_script_10005&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_script_10005","type":"script","value":"retainField(k8spodname,file_path,dockerimageid,gid,perm,std_filepath,pid,k8snodeid,uuid,std_pfilename,file_gid_name,sid,uid,file_gid,k8snodename,cmdline,pfilename,f [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_script_10006&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_script_10006","type":"script","value":"___lower_std_cmdline_10001=lower(std_cmdline);\n___concat_10030=concat(std_cmdline,std_pcmdline);\n___lower_cmdline_10097=lower(cmdline);\n","version":"1.0","extend [...]
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_rule_10002&&&&{"expressionStr":"((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((___lower_std_cmdline_10001,like,'%base%')|(std_pfilepath,like,'%bin/cron%'))|(std_pfilepath,like,'%bin/atd%'))|(std_pfilepath,like,'%bin/anacron%'))|((std_pfilename,like,'python%')&(((((std_cmdline,regex,'(/dev/shm/\\.)|(/tmp/\\.)')|(std_cmdline,regex,'((\\s+|/|^)history.*?-c)|((echo.*?>|rm\\s+).*?(/log/wtmp|\\. [...]
-dipper.private.blink.rules&&&&express&&&&11061&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11060\",\"10994\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11060&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11059\",\"10993\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11063&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10995\",\"11062\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11062&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10996\",\"10997\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11065&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11064\",\"10998\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11064&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11061\",\"11063\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11108&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11107\",\"11041\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11107&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11106\",\"11040\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11109&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11108\",\"11042\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11067&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11066\",\"11000\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11100&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11099\",\"11033\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11066&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11065\",\"10999\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11069&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11068\",\"11002\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11102&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11101\",\"11035\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11068&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11067\",\"11001\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11101&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11100\",\"11034\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11104&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11103\",\"11037\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11103&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11102\",\"11036\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11106&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11105\",\"11039\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11105&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11104\",\"11038\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11070&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11069\",\"11003\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11072&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11071\",\"11005\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11071&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11070\",\"11004\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11074&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11073\",\"11007\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11073&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11072\",\"11006\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11076&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11075\",\"11009\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11075&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11074\",\"11008\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11078&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11077\",\"11011\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11111&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11110\",\"11044\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11077&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11076\",\"11010\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11110&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11109\",\"11043\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11113&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11112\",\"11046\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11079&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11078\",\"11012\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11112&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11111\",\"11045\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11041&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"TVFYVr4IjZhHXMXYFZKw3A=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11040&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"HNlsfmzcq3GV3t2PVg3axqOxuRCR23HjZ7X2OMuHAds="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11043&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"hfP7cxvb2y9TYGol7rxbVw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11042&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"3aVwh7FsRE5E32EX8ohSFA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10998&&&&{"aesFlag":1,"varName":"std_pfilename","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"LMx96lEt0ofMQatz3hW9JA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10997&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"uaVPWG3+kMJsrVfkKA4gKr+Rb+eJo7anWrwk31gUv2ShqivMTT/ZmlNjhqNK65NqtzUhByv4FmURtuPBusRxdEUgdT68SVaFrDvvwrMWyEQfjQGScRxCEZ0nN8Lijeb7"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10999&&&&{"aesFlag":1,"varName":"pcmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"ufdKNE+MoBQmTicJrrpOmBCUKI+jGUIBKuBHTLBU9ng="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10990&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"+Avnh0zoiCv4arP4n2otmhJIJuuvTbHuyL1LzGm+HOQrPIuSCsoz5yzSJmxvGHd1BcIRWb8BT4MU9vaD6vId13SS+AFiPGQ0kjUpabMGGla0ns2l0sv/vaU1M9+ZiAM2tOPkViMXzB0klCfF1zgKL3iSiC+fpbA8wCPkSNVbhNbsCfI3rk620OtCBTxDt6Dioh/fyZQCcHspwbCjB76JwvQpJzdLECWvub [...]
-dipper.private.blink.rules&&&&express&&&&11045&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"Fh76UkPOBYxrDlIGKhLy3g=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11044&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"GPf/PJUXzl7O8JEk0vEK4g=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10992&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"vcUI9S5P7PIch7ciybbDkg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11047&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10975\",\"10976\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10991&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"HJRJmcYwRPmV4nEtVMMUJQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11046&&&&{"aesFlag":1,"varName":"___lower_cmdline_10097","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"o++L7/E2+XPNgEU7s7lIxnUlN3vGoLG/pi30BCLejvTGqEgTfQLC+ZPi4j4vQk4Q"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10994&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"MDel7t10CZtS8skhcN0HUw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11049&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11048\",\"10978\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10993&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"h9XkyXTTRMlmWnvVoZ8Txg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11048&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11047\",\"10977\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10996&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"dj6zdMkh+IFsV1dra3+B5kQhRDrC7IjEFE4xZ21BSerROIrdrpATclVeqXVxEPm2y/aKFVUAwobUtc3JbRG7MNv/bxqE3fox9RTWbs8fCgDIKfeTqJEQFvAmnfoRBtmS"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10995&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"BH5m6/QqWV5ojuMXWOcdAA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11050&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10980\",\"10981\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11052&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11051\",\"10983\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11051&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11050\",\"10982\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11054&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"10979\",\"11053\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11053&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11052\",\"10984\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11056&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11055\",\"10989\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11055&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11049\",\"11054\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11058&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11057\",\"10991\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11057&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11056\",\"10990\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11059&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11058\",\"10992\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11021&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"bMLK6lJHuLR2KQlAMGB4pg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11020&&&&{"aesFlag":1,"varName":"___concat_10030","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"WI7OXqpILJMlBJa9txI8ke/M4tL2bm5DrYE7ERsWzPGEZs34h7BANHJutJKBUW4wE98d2NjZglq2AzGj7fzU+QHUvKHuBAbM3VZo3c0VcVU="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10976&&&&{"aesFlag":1,"varName":"std_pfilepath","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"bSzz+k//aB+n/eoa8WLr9g=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10975&&&&{"aesFlag":1,"varName":"___lower_std_cmdline_10001","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"bu0RJCYz/VtkFu5QJ45CLg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10978&&&&{"aesFlag":1,"varName":"std_pfilepath","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"BFK7XklNgAGkRWWL/JMKow=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10977&&&&{"aesFlag":1,"varName":"std_pfilepath","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"zCTZGfH1iVMGCEBammSdBg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10979&&&&{"aesFlag":1,"varName":"std_pfilename","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"8YDipxdTx/8MQtmb2pwsWw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11023&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"ptbEPNACThGbQkGr7c3rRA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11022&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"lWDiKLo2Q5XAqZFh1JY/Gw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11025&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"cs5iZ30xVZIaEg8QmZjtsw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11024&&&&{"aesFlag":1,"varName":"cmdline","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"IzssAnC1BMB2v9nA+q1aLg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11027&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"x3eOJ3c0H8bBTOFUKxUwvg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11026&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"a8J7a+jUDzyIBIJHLH+u0w=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11029&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"Kwsaze27qwsXPvQuRTYfEu1gKmFzli2phmlaqBBlHNWmlxKVXWYj/gDEMO5KS3ab"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11028&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"VAiHxpc6w/lOnT/goAtsDTvfaXX+C8kcLx8q5CRvJyGaPYYeZymPv7YkF714R3tU/eKAPDFvoijIGto0dsvR1D6rb/b6jMi7TwWA65tvw7CTBoDGBES/FkdmEN1Mu16DZFTuO63W/3VIV7PV/4nmDAZD4GP+Km861y47b3hq/0SvMpCAlRYvyoO0FiX1TulB"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11030&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"bXnk9N2uamEoUq5sap8AIXyj8kJ9uBvkPoXwL1Cu0cP9v0u93QmppOp33gXvFof5QGewMH4SgR+eXDKDc3SNbSgaOZ220e3K/nOGZWK3aX4oDRM1mOnCg8wvKhxBbHsCuApPSFGbX9xwyETNtdEcu7GwWXChO0oNM2uEGvN/WnYrrC6ArlXfkVM6l8kIcHNN"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11032&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"rYmripZ9v8codAeoVXJQcVwaGLu+C6VdwfhFcoaiUq52UP3mDMG/cDoTrYFQH98Ak8nRpfzj7ru1AOC7K7HYm1flQf0KIDYGVDF0ClfhtuQ="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11031&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"fILg2Kj3NyWuwT9VD2DAFiZ9n0Y6S6nBEKd7FHWM0FIaujiDRWF9ehTvDIqBPA2uy4500rRtM/18U7Mu4Ui5/Q=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10987&&&&{"aesFlag":1,"varName":"std_pcmdline","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"l9TYEZNEgx5Smq8tuUUy9g=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10986&&&&{"aesFlag":1,"varName":"std_pcmdline","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"nJBjprx929aA/lZbLSUXqw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10989&&&&{"aesFlag":1,"varName":"std_pfilename","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"wjmOwzecnEg3F05hLnwG0A=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10988&&&&{"aesFlag":1,"varName":"std_pcmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"UqIEA0mgyxPygn9s2M18j6t59zr/GOjLa11h/KtCj5o="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11034&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"Y88XpK8iIbsDDSWzTVe7Jw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11033&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"1KmXvvnf4K31Gqbgz2NV7uc/hg4HlDe4eTJKNvbV0GF3kAhCNo6LKuf00QtKVIrQctutB+xGlGMw4I+ZdQIW7PiBFADSFxA+50YY+jOXtVzTlxNLbEQwUl5gGHUT0iSm"}&&&&null
-dipper.private.blink.rules&&&&express&&&&10981&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"2wGev09h1x35RIPn5hr0zOfGMvLhLCkyFXeXXUme7ornvsWCoR31uKyTaGbsEZTRra4DoUzioTPaKzh1dIhrc4KiOpEgKeiLhvtNMYZLZSmhzdycxECOXqOysb0myhnOfFGfSopx3hnMReSulU2tCURmJldZpWZowJhxdIuKsMflvGDBdavYBRrzELzc/ZkfGAvXV4Hfkqya9osNpKpsSw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11036&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"Z3evgG2VXwjmywr8m/at5Q=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10980&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"RgaZ+qq6ia8cXW/5TQ/hxbFKmLbxCrBqRkQcBiMzsa0="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11035&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"6K4xHNULGM0I19x+AyX8lmf9pIHBqC2U5C4NWKeaE5aNL8xPtICJ2X7/DyGonsruMRiYRzwSVAVZmEkSUU1WQw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10983&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"bXnk9N2uamEoUq5sap8AId8I0BKuewpjnMMhC8cCcKkNjSCLj2tJFtU/uZU41VGCSJp/wF/uQlMMMl0FQxiQB6lc/acMHJ/1VaWHQ0hFKeBzweX5xlJYcKtXOMpLF4H0L8emqzfxAs6vJKAvLAGR0BvZVji32jUeqEPooutYIy8="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11038&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"FaqlXF+NjGPqjsQY1Fs1NQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10982&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"v46lbaznhI5nGyi690SeDifWsx7dIadFxNKGEth1SW5VromxcyKwl1hUdWFUBH24iZ71uhFfavbHknZuPUG64KJheSt/mtX3sOZ8qf7m8UhRLyuRj3kz03goi0yy+FMCBBY7fEKQv5w5cUqKGbzzqg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11037&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"vQXfgUWX9KYmBuuiyI0Md0qBuWpbNLgIN4YLyICABmo="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10985&&&&{"aesFlag":1,"varName":"std_pcmdline","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"k4ZCQo0F8fT0hS75Hd5sMg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&10984&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"vSCxbCf398GrT+LaONPqXqD73IocQLKKBiv0FlAUO4yJ1Abh4Hyb17cGyKxmCxS9IAh6MGwdpJcWE/r5CLHFfsvWZD+VgHrlNwtjheaFLzoGY6meQHhRap7Ei6BHhRjPKrP5MP6gdC7W6EfC2E/E3n9ubhmq8zPRMMigkRqCMI+ZstifizoxAiEEUu4vCzIz"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11039&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"vQXfgUWX9KYmBuuiyI0Md0qBuWpbNLgIN4YLyICABmo="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11081&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11080\",\"11014\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11080&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11079\",\"11013\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11083&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11082\",\"11016\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11082&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11081\",\"11015\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11085&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11084\",\"11018\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11084&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11083\",\"11017\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11087&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11086\",\"11020\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11086&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11085\",\"11019\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11009&&&&{"aesFlag":1,"varName":"cmdline","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"ceQQ+T4hhIWo+KzjYIEwbw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11008&&&&{"aesFlag":1,"varName":"pcmdline","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"5XhiYhbSK5u+DDqvTDrrFw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11001&&&&{"aesFlag":1,"varName":"filepath","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"agzS9KWiGz89vU+v1X8JfQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11089&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11088\",\"11022\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11000&&&&{"aesFlag":1,"varName":"cwd","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"agzS9KWiGz89vU+v1X8JfQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11088&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11087\",\"11021\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11003&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"QknOB+ql9NKXvVsEoLQUqg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11002&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"9vmtEK+LW6oqgtoNGXri/B/7PMPU8QTLCVLoqwyIFYk="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11005&&&&{"aesFlag":1,"varName":"std_pcmdline","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"QknOB+ql9NKXvVsEoLQUqg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11004&&&&{"aesFlag":1,"varName":"std_pcmdline","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"9vmtEK+LW6oqgtoNGXri/B/7PMPU8QTLCVLoqwyIFYk="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11007&&&&{"aesFlag":1,"varName":"cmdline","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"KSGb4IrIYtMpRcbj2FK08g=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11006&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"HpGAXMP8jdcBm0ggnkqXH11+JNPLI9FePdCx63hvvNpPD7eFb0h8x4G9D9T8mZhWKt8PUIb2QwQwqjx7BVxYhIb+TrGkjVzAC//OfX4kEkUV0dwiTK1dr+KSO35OeRkvxbk0NoIhV4lDXKwHfuoEcNU5zdbO8pUrju6y6P62EJCfgRDF9hN2TD2loaj9BDVl"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11092&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11091\",\"11025\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11091&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11090\",\"11024\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11094&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11093\",\"11027\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11093&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11092\",\"11026\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11096&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11095\",\"11029\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11095&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11094\",\"11028\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11010&&&&{"aesFlag":1,"varName":"cmdline","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"JCQ+ifFXeLkOYGxkDn1AYA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11098&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11097\",\"11031\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11097&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11096\",\"11030\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11090&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11089\",\"11023\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11019&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"KAk1x/4nGha+x4Yf00SdMwnbpEPiL9++eM4vQxiKAPg="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11012&&&&{"aesFlag":1,"varName":"pcmdline","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"6WxBwqGTqeFcYf1pipmKcfp81FzKpmmcpIHdilKDpPU="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11011&&&&{"aesFlag":1,"varName":"cmdline","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"OBxJlsIL2EcTonXa+/dNQg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11099&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11098\",\"11032\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11014&&&&{"aesFlag":1,"varName":"std_pfilename","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0Q/1PBEDqr60PMbIRqKfYBWb5Vpg724nza7Y8cu27MZA8m6LXl1qWUoDYfqjtJbkk3bodVs2fpbffZEOXKg2dA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11013&&&&{"aesFlag":1,"varName":"std_pfilename","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"cjWGDo2eA5+zrRhIdflopA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11016&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"QXEC0iMe0Y01pb3zIrNY3w=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11015&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"oGJJ1tOpdY/BzLXTcOKznjzexr9upC5+357GQpnbk4p8p+LnmhsdaPfCMlhF3WPqyaE4uxZ8VJBXYVxYKPLGZg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11018&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"UlheacR+WhpMm1PDPVxkCA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11017&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"BnbZj5gSeJIwCZz2/gPDog=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_script_10007&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_script_10007","type":"script","value":"retainField(k8spodname,dockerimageid,gid,perm,std_filepath,pid,k8snodeid,uuid,std_pfilename,file_gid_name,sid,uid,file_gid,k8snodename,pfilename,cmdline,filepath,st [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_union_10001&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"entityName\\\\\\\":\\\\\\\"script\\\\\\\",\\\\\\\"scriptName\\\\\\\":\\\\\\\"subpipline_blink.source.aegis.proc_ [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_union_10001_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_union_10001_script_10001","type":"script","value":"___regexp_10469=regex(cmdline,'virustotal\\.com|hostname');\n___compare_10225=equals(___regexp_10469,false);\n___regex [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_union_10001_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_union_10001_script_10002","type":"script","value":"retainField(k8spodname,dockerimageid,gid,perm,std_filepath,pid,k8snodeid,uuid,std_pfilename,file_gid_name,sid,uid,file [...]
-dipper.private.blink.rules&&&&pipline&&&&subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_union_10002&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"entityName\\\\\\\":\\\\\\\"script\\\\\\\",\\\\\\\"scriptName\\\\\\\":\\\\\\\"subpipline_blink.source.aegis.proc_ [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_union_10002_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_union_10002_script_10001","type":"script","value":"___regexp_10545=regex(std_pfilename,'^postgres');\n___regexp_10546=regex(std_cmdline,'(/dev/shm/\\.)|(/tmp/\\.)');\n__ [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_union_10002_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_blink.source.aegis.proc_sas_linux_alert_proc_cmd_union_10002_script_10002","type":"script","value":"retainField(k8spodname,dockerimageid,gid,perm,std_filepath,pid,k8snodeid,uuid,std_pfilename,file_gid_name,sid,uid,file [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_script_10008&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_script_10008","type":"script","value":"___unixtime_10020=unixtime(scan_time,'yyyy-MM-dd HH:mm:ss');\n___division_10016=division(___unixtime_10020,3600);\n___floor_10007=floor(___division_10016);\n___cast [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_script_10009&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_script_10009","type":"script","value":"retainField(k8spodname,dockerimageid,gid,perm,std_filepath,pid,k8snodeid,uuid,std_pfilename,file_gid_name,sid,uid,file_gid,k8snodename,pfilename,cmdline,filepath,st [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_script_10010&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_script_10010","type":"script","value":"___!null_10037=!null(event_name);\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_rule_10003&&&&{"expressionStr":"___!null_10037&(event_name,!in,'\\'\\',\\'N/A\\'')","scriptNames":"[]","expressionName":"11115","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_rule_10003","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionName [...]
-dipper.private.blink.rules&&&&express&&&&11115&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"___!null_10037\",\"11114\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11114&&&&{"aesFlag":1,"varName":"event_name","functionName":"!in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"TVEW40h+v56CrjMCEeE0Kg=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_script_10011&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_script_10011","type":"script","value":"___concat_10033=concat(std_cmdline,std_filename,std_pfilepath,std_pcmdline);\n___regexp_10591=regex(___concat_10033,'dracut-install|aliyun-assist|/gcc/|virustotal|a [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_script_10012&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_script_10012","type":"script","value":"retainField(is_white,k8spodname,dockerimageid,gid,perm,std_filepath,pid,k8snodeid,uuid,std_pfilename,file_gid_name,sid,uid,file_gid,k8snodename,pfilename,cmdline,fi [...]
-dipper.private.blink.rules&&&&rule&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_rule_10004&&&&{"expressionStr":"is_white,==,double,0","scriptNames":"[]","expressionName":"11116","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_rule_10004","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[]"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11116&&&&{"varName":"is_white","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0.0"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_script_10013&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_script_10013","type":"script","value":"ali_uid='';\nclient_ip='';\nevent_type='进程异常行为';\nevent_name=event_name;\nlevel='high';\n___cast_10581=cast(uuid,'string');\n___cast_10582=cast(scan_time,'string'); [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_script_10014&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_script_10014","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_script_10015&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_script_10015","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&channel&&&&blink.source.aegis.proc_sas_linux_alert_proc_cmd_channel_10001&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink.source.aegis.proc_sas_linux_alert_proc_cmd_channel_10001","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeInOrder":"true","maxFetchLogG [...]
-dipper.private.blink.rules&&&&pipline&&&&blink.source.aegis.proc&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"entityName\\\\\\\":\\\\\\\"script\\\\\\\",\\\\\\\"scriptName\\\\\\\":\\\\\\\"blink.source.aegis.proc_script_10001\\\\\\\",\\\\\\\"lable\\\\\\\":\\\\\\\"blink.so [...]
-dipper.private.blink.rules&&&&channel&&&&blink.source.aegis.proc;channel&&&&{"offset":"0","isJsonData":"true","maxThread":"12","concurrentCount":"12","className":"com.aliyun.yundun.dipper.channel.adapter.MetaqChannel","configureName":"blink.source.aegis.proc;channel","type":"channel","version":"1.0","timeout":"30000","activtyTimeOut":"3000","tags":"blink.source.aegis.proc.tags","syncTimeout":"5000","groupName":"blink.source.aegis.proc.group","isBatchMessage":"true","isAutoFlush":"false", [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.proc_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.proc_script_10001","type":"script","value":"aliUid=JSON_VALUE (meta_conf, '$.aliUid');logContent=data;logTime=now();inner_message();splitArray('logContent');scan_time=coalesce(time,'N/A');uid=coalesce(uid,'N/A');uid_name=coalesce(username,'N/A');euid=coalesc [...]
-dipper.private.blink.rules&&&&pipline&&&&windows_reg_alert&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"nextStageLables\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"windows_reg_alert_script_10002\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"cancelAfterConfigurableRefreshListerner\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"ownerSqlNodeTableName\\\\\\\":\\\\\\\"dwd_yunsec_host_aegis_window_regedit_stre [...]
-dipper.private.blink.rules&&&&script&&&&windows_reg_alert;json_concat_10014&&&&{"fullClassName":"com.aliyun.sec.lyra.udf.ext.JsonConcat","initMethodName":"open","functionName":"json_concat","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"windows_reg_alert;json_concat_10014","type":"script","version":"1.0","closeMethodName":"close","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&windows_reg_alert;json_array_parse_10001&&&&{"setCollectorMethodName":"setCollector","fullClassName":"com.aliyun.isec.seraph.udtf.JsonArrayParser","initMethodName":"open","functionName":"json_array_parse","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udtf.BlinkUDTFScript","configureName":"windows_reg_alert;json_array_parse_10001","type":"script","version":"1.0","isURL":"false","nameSpace":"dipper.private.blink.rules","close [...]
-dipper.private.blink.rules&&&&channel&&&&windows_reg_alert&&&&{"offset":"0","isJsonData":"true","maxThread":"12","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.MetaqChannel","configureName":"windows_reg_alert","type":"channel","version":"1.0","timeout":"30000","activtyTimeOut":"3000","syncTimeout":"5000","groupName":"windows_reg_alert","isBatchMessage":"true","isAutoFlush":"false","maxFetchLogGroupSize":"100","syncCount":"1000","outputThreadCount":"-1","nameS [...]
-dipper.private.blink.rules&&&&script&&&&windows_reg_alert_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"windows_reg_alert_script_10001","type":"script","value":"___JSON_ARRAY_PARSE_10001=JSON_ARRAY_PARSE(data,'proc_cmd','proc_path','key_path','pid','time','ppid');m.proc_cmd=udtf.0;m.proc_path=udtf.1;m.key_path=udtf.2;m.pid=udtf.3;m.time=udtf.4;m.ppid=udtf.5;","version":"1.0","extendFi [...]
-dipper.private.blink.rules&&&&script&&&&windows_reg_alert_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"windows_reg_alert_script_10002","type":"script","value":"trace_id=traceId;\n___json_get_10095=json_get(meta_conf,'$.gcLevel');\ngc_level=___json_get_10095;rm('___json_get_10095');\n___json_get_10096=json_get(meta_conf,'$.aliUid');\nali_uid=___json_get_10096;rm('___json_get_10096');\ [...]
-dipper.private.blink.rules&&&&script&&&&windows_reg_alert_script_10003&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"windows_reg_alert_script_10003","type":"script","value":"ppid=m.ppid;\nkey_path=m.key_path;\ntime=m.time;\nproc_cmd=m.proc_cmd;\nproc_path=m.proc_path;\npid=m.pid;\nretainField(app,ppid,trace_id,buy_aegis,app_version,ip,safe_mode,key_path,type,uuid,version,time,ali_uid,buy_sas,proc_ [...]
-dipper.private.blink.rules&&&&script&&&&windows_reg_alert_script_10004&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"windows_reg_alert_script_10004","type":"script","value":"___lower_key_path_10001=lower(key_path);\n___regexp_10592=regex(key_path,'InprocServer32|LocalServer32|InprocServer|LocalServer');\nif((key_path,like,'%PowerShell\\ScriptBlockLogging%')){___case_10149='hkey_powershell_block';} [...]
-dipper.private.blink.rules&&&&script&&&&windows_reg_alert_script_10005&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"windows_reg_alert_script_10005","type":"script","value":"retainField(app,trace_id,proc_cmd,buy_aegis,app_version,ip,safe_mode,proc_path,pid,type,uuid,version,ppid,ali_uid,buy_sas,key_path,time,gc_level,seq,label);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&windows_reg_alert_script_10006&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"windows_reg_alert_script_10006","type":"script","value":"___!null_10038=!null(label);\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&windows_reg_alert_rule_10001&&&&{"expressionStr":"(___!null_10038)","scriptNames":"[]","expressionName":"11118","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"windows_reg_alert_rule_10001","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[]"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11118&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11117\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11117&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"___!null_10038\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&script&&&&windows_reg_alert_script_10007&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"windows_reg_alert_script_10007","type":"script","value":"retainField(app,trace_id,proc_cmd,buy_aegis,app_version,ip,safe_mode,proc_path,pid,label,type,uuid,version,ppid,ali_uid,buy_sas,key_path,time,gc_level,seq);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&windows_reg_alert_script_10008&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"windows_reg_alert_script_10008","type":"script","value":"___!_10071=!((proc_path,like,'%Trend Micro/%/TmListen.exe%'));\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&windows_reg_alert_rule_10002&&&&{"expressionStr":"(___!_10071)","scriptNames":"[]","expressionName":"11120","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"windows_reg_alert_rule_10002","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[]"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11119&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"___!_10071\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11120&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11119\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&script&&&&windows_reg_alert_script_10009&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"windows_reg_alert_script_10009","type":"script","value":"___in_10090=contain(label,'hkey_defender')\n;___concat_10035=concat(proc_cmd,proc_path);\n___lower____concat_10035_10001=lower(___concat_10035);\n___in_10091=contain(label,'hkey_defender')\n;___concat_10036=concat(proc_cmd,pro [...]
-dipper.private.blink.rules&&&&script&&&&windows_reg_alert_script_10010&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"windows_reg_alert_script_10010","type":"script","value":"retainField(app,trace_id,proc_cmd,buy_aegis,app_version,ip,safe_mode,proc_path,pid,label,type,uuid,version,ppid,ali_uid,buy_sas,key_path,time,gc_level,seq,alert_task_name);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&windows_reg_alert_script_10011&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"windows_reg_alert_script_10011","type":"script","value":"___!null_10041=!null(alert_task_name);\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&windows_reg_alert_rule_10003&&&&{"expressionStr":"(___!null_10041)","scriptNames":"[]","expressionName":"11122","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"windows_reg_alert_rule_10003","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[]"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11122&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11121\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11121&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"___!null_10041\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&script&&&&windows_reg_alert_script_10012&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"windows_reg_alert_script_10012","type":"script","value":"retainField(app,trace_id,proc_cmd,buy_aegis,app_version,ip,safe_mode,proc_path,pid,label,alert_task_name,type,uuid,version,ppid,ali_uid,buy_sas,key_path,time,gc_level,seq);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&windows_reg_alert_script_10013&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"windows_reg_alert_script_10013","type":"script","value":"retainField(app,trace_id,proc_cmd,buy_aegis,app_version,ip,safe_mode,proc_path,pid,label,alert_task_name,type,uuid,version,ppid,ali_uid,buy_sas,key_path,time,gc_level,seq);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&pipline&&&&subpipline_windows_reg_alert_union_10001&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.FilterChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"names\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"subpipline_windows_reg_alert_union_10001_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"entityName\\\\\\\": [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_windows_reg_alert_union_10001_rule_10001&&&&{"expressionStr":"alert_task_name,in,'\\'hkey_run_from_root\\''","scriptNames":"[]","expressionName":"11123","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_windows_reg_alert_union_10001_rule_10001","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[]"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11123&&&&{"aesFlag":1,"varName":"alert_task_name","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"fbQAtwZSw9S4GWSTkKdCFDBoPFzjQvWt3boQSF0SGec="}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_windows_reg_alert_union_10001_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_windows_reg_alert_union_10001_script_10001","type":"script","value":"___cast_10629=cast(null_10014,'string');\nclient_ip=___cast_10629;rm('___cast_10629');\nevent_type='进程异常行为';\nevent_name='修改Windows注册表自启动项';\nlevel='high';\n___cast_10630=cast(proc [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_windows_reg_alert_union_10001_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_windows_reg_alert_union_10001_script_10002","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&pipline&&&&subpipline_windows_reg_alert_union_10002&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.FilterChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"names\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"subpipline_windows_reg_alert_union_10002_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"entityName\\\\\\\": [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_windows_reg_alert_union_10002_rule_10001&&&&{"expressionStr":"alert_task_name,in,'\\'hkey_user_logon_trigger\\''","scriptNames":"[]","expressionName":"11124","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_windows_reg_alert_union_10002_rule_10001","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[]"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11124&&&&{"aesFlag":1,"varName":"alert_task_name","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"RbcVVSweeHoecAucQldJ0/tqocCsISkoaG+7JEDZcUM="}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_windows_reg_alert_union_10002_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_windows_reg_alert_union_10002_script_10001","type":"script","value":"___cast_10635=cast(null_10015,'string');\nclient_ip=___cast_10635;rm('___cast_10635');\nevent_type='进程异常行为';\nevent_name='通过Windows注册表修改用户登录配置';\nlevel='high';\n___cast_10636=cast( [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_windows_reg_alert_union_10002_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_windows_reg_alert_union_10002_script_10002","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&pipline&&&&subpipline_windows_reg_alert_union_10003&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.FilterChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"names\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"subpipline_windows_reg_alert_union_10003_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"entityName\\\\\\\": [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_windows_reg_alert_union_10003_rule_10001&&&&{"expressionStr":"alert_task_name,in,'\\'hkey_defender_ps\\''","scriptNames":"[]","expressionName":"11125","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_windows_reg_alert_union_10003_rule_10001","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[]"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11125&&&&{"aesFlag":1,"varName":"alert_task_name","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"IPYM38C/IOMAL3hzz89e5uJ8QqLWqADbdmLWhXvqOEc="}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_windows_reg_alert_union_10003_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_windows_reg_alert_union_10003_script_10001","type":"script","value":"___cast_10641=cast(null_10016,'string');\nclient_ip=___cast_10641;rm('___cast_10641');\nevent_type='进程异常行为';\nevent_name='Windows Defender配置修改_紧急';\nlevel='high';\n___cast_10642=ca [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_windows_reg_alert_union_10003_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_windows_reg_alert_union_10003_script_10002","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&pipline&&&&subpipline_windows_reg_alert_union_10004&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.FilterChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"names\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"subpipline_windows_reg_alert_union_10004_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"entityName\\\\\\\": [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_windows_reg_alert_union_10004_rule_10001&&&&{"expressionStr":"alert_task_name,in,'\\'hkey_defender_other\\''","scriptNames":"[]","expressionName":"11126","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_windows_reg_alert_union_10004_rule_10001","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[]"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11126&&&&{"aesFlag":1,"varName":"alert_task_name","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"/PAIOo0j1K5S+H5MuPeRMeJ4i3RL55iENQ6o0bwBUpI="}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_windows_reg_alert_union_10004_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_windows_reg_alert_union_10004_script_10001","type":"script","value":"___cast_10647=cast(null_10017,'string');\nclient_ip=___cast_10647;rm('___cast_10647');\nevent_type='进程异常行为';\nevent_name='Windows Defender配置修改_提醒';\nlevel='high';\n___cast_10648=ca [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_windows_reg_alert_union_10004_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_windows_reg_alert_union_10004_script_10002","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&pipline&&&&subpipline_windows_reg_alert_union_10005&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.FilterChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"names\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"subpipline_windows_reg_alert_union_10005_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"entityName\\\\\\\": [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_windows_reg_alert_union_10005_rule_10001&&&&{"expressionStr":"alert_task_name,in,'\\'hkey_control_wdigest_common\\''","scriptNames":"[]","expressionName":"11127","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_windows_reg_alert_union_10005_rule_10001","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[]"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11127&&&&{"aesFlag":1,"varName":"alert_task_name","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"gTbD0sj2xCNjBc8NAJYASWlzSx8U7VLTHKBVUnDBxlA="}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_windows_reg_alert_union_10005_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_windows_reg_alert_union_10005_script_10001","type":"script","value":"___cast_10653=cast(null_10018,'string');\nclient_ip=___cast_10653;rm('___cast_10653');\nevent_type='进程异常行为';\nevent_name='Windows登录凭证窃取';\nlevel='high';\n___cast_10654=cast(proc_cm [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_windows_reg_alert_union_10005_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_windows_reg_alert_union_10005_script_10002","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&pipline&&&&subpipline_windows_reg_alert_union_10006&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.FilterChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"names\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"subpipline_windows_reg_alert_union_10006_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"entityName\\\\\\\": [...]
-dipper.private.blink.rules&&&&rule&&&&subpipline_windows_reg_alert_union_10006_rule_10001&&&&{"expressionStr":"alert_task_name,in,'\\'hkey_image_hijack_debugger\\''","scriptNames":"[]","expressionName":"11128","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_windows_reg_alert_union_10006_rule_10001","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[]"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11128&&&&{"aesFlag":1,"varName":"alert_task_name","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"vxuFqmZe0cLC5qwj7UdwRWYRKwhvdoU0ezVD+a8NA0U="}&&&&null
-dipper.private.blink.rules&&&&script&&&&subpipline_windows_reg_alert_union_10006_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_windows_reg_alert_union_10006_script_10001","type":"script","value":"___cast_10659=cast(null_10019,'string');\nclient_ip=___cast_10659;rm('___cast_10659');\nevent_type='进程异常行为';\nevent_name='Windows映像劫持';\nlevel='high';\n___cast_10660=cast(proc_cmd, [...]
-dipper.private.blink.rules&&&&script&&&&subpipline_windows_reg_alert_union_10006_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"subpipline_windows_reg_alert_union_10006_script_10002","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&channel&&&&windows_reg_alert_channel_10001&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"windows_reg_alert_channel_10001","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeInOrder":"true","maxFetchLogGroupSize":"100","syncCount":"1000","nameSpace":"dipper.private [...]
-dipper.private.blink.rules&&&&pipline&&&&blink.direct.source.aegis_reg&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.SubPiplineChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"nextStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"filterMsgSwitch\\\\\\\":\\\\\\\"blink.direct.source.aegis_reg.filter.switch\\\\\\\",\\\\\\\"cancelAfterConfigurableRefreshListerner\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"className\\\\\\\":\\\\\\\"com.ali [...]
-dipper.private.blink.rules&&&&channel&&&&blink.direct.source.aegis_reg;channel&&&&{"offset":"0","isJsonData":"true","maxThread":"12","concurrentCount":"12","className":"com.aliyun.yundun.dipper.channel.adapter.MetaqChannel","configureName":"blink.direct.source.aegis_reg;channel","type":"channel","version":"1.0","timeout":"30000","activtyTimeOut":"3000","tags":"blink.direct.source.aegis_reg.tags","syncTimeout":"5000","groupName":"blink.direct.source.aegis_reg.group","isBatchMessage":"true [...]
-dipper.private.blink.rules&&&&pipline&&&&sas_linux_alert_client_reverse_shell&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"nextStageLables\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"sas_linux_alert_client_reverse_shell_script_10002\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"cancelAfterConfigurableRefreshListerner\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"ownerSqlNodeTableName\\\\\\\":\\\\\\\"mq_ [...]
-dipper.private.blink.rules&&&&script&&&&sas_linux_alert_client_reverse_shell;SplitString_10001&&&&{"setCollectorMethodName":"setCollector","fullClassName":"com.sas.zing.blink.udf.SplitString","initMethodName":"open","functionName":"SplitString","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udtf.BlinkUDTFScript","configureName":"sas_linux_alert_client_reverse_shell;SplitString_10001","type":"script","version":"1.0","isURL":"false","nameSpace":"dipper.private.blink [...]
-dipper.private.blink.rules&&&&script&&&&sas_linux_alert_client_reverse_shell;json_concat_10015&&&&{"fullClassName":"com.aliyun.sec.lyra.udf.ext.JsonConcat","initMethodName":"open","functionName":"json_concat","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"sas_linux_alert_client_reverse_shell;json_concat_10015","type":"script","version":"1.0","closeMethodName":"close","ext [...]
-dipper.private.blink.rules&&&&script&&&&sas_linux_alert_client_reverse_shell;ReverseShellFD_10001&&&&{"fullClassName":"com.sas.zing.blink.udf.ReverseShellFD","initMethodName":"open","functionName":"ReverseShellFD","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"sas_linux_alert_client_reverse_shell;ReverseShellFD_10001","type":"script","version":"1.0","closeMethodName":"clo [...]
-dipper.private.blink.rules&&&&channel&&&&sas_linux_alert_client_reverse_shell&&&&{"offset":"0","isJsonData":"true","maxThread":"12","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.MetaqChannel","configureName":"sas_linux_alert_client_reverse_shell","type":"channel","version":"1.0","timeout":"30000","activtyTimeOut":"3000","tags":"T_MSG_BASHSHELL","syncTimeout":"5000","groupName":"sas_linux_alert_client_reverse_shell","pullIntervalMs":"100","isBatchMessage":"tr [...]
-dipper.private.blink.rules&&&&script&&&&sas_linux_alert_client_reverse_shell_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"sas_linux_alert_client_reverse_shell_script_10001","type":"script","value":"log_content=data;\n___json_get_10099=json_get(meta_conf,'$.aliUid');\naliUid=___json_get_10099;rm('___json_get_10099');\nlog_protocol_type=type;\n___now_10002=now();\n___fromunixtime_10003 [...]
-dipper.private.blink.rules&&&&script&&&&sas_linux_alert_client_reverse_shell_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"sas_linux_alert_client_reverse_shell_script_10002","type":"script","value":"retainField(sls_time,app_version,log_protocol_type,aliUid,log_content,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&sas_linux_alert_client_reverse_shell_rule_10001&&&&{"expressionStr":"log_protocol_type,==,'T_MSG_BASHSHELL'","scriptNames":"[]","expressionName":"11129","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"sas_linux_alert_client_reverse_shell_rule_10001","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[]"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11129&&&&{"aesFlag":1,"varName":"log_protocol_type","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"t9ibOFmrc6H0cLcIbwDcZQ=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&sas_linux_alert_client_reverse_shell_script_10003&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"sas_linux_alert_client_reverse_shell_script_10003","type":"script","value":"___uuid_10005=uuid();\nunique_id=___uuid_10005;rm('___uuid_10005');\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&sas_linux_alert_client_reverse_shell_script_10004&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"sas_linux_alert_client_reverse_shell_script_10004","type":"script","value":"retainField(unique_id,sls_time,app_version,log_protocol_type,aliUid,log_content,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&sas_linux_alert_client_reverse_shell_script_10005&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"sas_linux_alert_client_reverse_shell_script_10005","type":"script","value":"retainField(unique_id,sls_time,app_version,log_protocol_type,aliUid,log_content,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&channel&&&&sas_linux_alert_client_reverse_shell_channel_10001&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"sas_linux_alert_client_reverse_shell_channel_10001","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeInOrder":"true","maxFetchLogGroupSize":"100","syncCou [...]
-dipper.private.blink.rules&&&&script&&&&sas_linux_alert_client_reverse_shell_script_10006&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"sas_linux_alert_client_reverse_shell_script_10006","type":"script","value":"___splitindex_10007=splitindex(app_version,'_',1);\n___splitindex_10008=splitindex(app_version,'_',2);\n___concat_10038=concat(___splitindex_10007,___splitindex_10008);\n___cast_10665=cast [...]
-dipper.private.blink.rules&&&&script&&&&sas_linux_alert_client_reverse_shell_script_10007&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"sas_linux_alert_client_reverse_shell_script_10007","type":"script","value":"retainField(unique_id,sls_time,app_version,aliUid,log_content,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&sas_linux_alert_client_reverse_shell_script_10008&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"sas_linux_alert_client_reverse_shell_script_10008","type":"script","value":"___compare_10257=greateeq(app_version,'1085');\n___REVERSESHELLFD_10001=REVERSESHELLFD(log_content,'new');\n___REVERSESHELLFD_10002=REVERSESHELLFD(log_content,'old');\nif(___compare_10257) [...]
-dipper.private.blink.rules&&&&script&&&&sas_linux_alert_client_reverse_shell_script_10009&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"sas_linux_alert_client_reverse_shell_script_10009","type":"script","value":"retainField(res,unique_id,sls_time,app_version,aliUid,log_content,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&sas_linux_alert_client_reverse_shell_rule_10002&&&&{"expressionStr":"res,<>,''","scriptNames":"[]","expressionName":"11130","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"sas_linux_alert_client_reverse_shell_rule_10002","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[]"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11130&&&&{"aesFlag":1,"varName":"res","functionName":"<>","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"wc8fi0p7qwBHnY1x2c6Qsw=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&sas_linux_alert_client_reverse_shell_script_10010&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"sas_linux_alert_client_reverse_shell_script_10010","type":"script","value":"___splitindex_10009=splitindex(res,'@@@',0);\ncontainer_name=___splitindex_10009;rm('___splitindex_10009');\n___splitindex_10010=splitindex(res,'@@@',1);\ncontainer_image_id=___splitindex_ [...]
-dipper.private.blink.rules&&&&script&&&&sas_linux_alert_client_reverse_shell_script_10011&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"sas_linux_alert_client_reverse_shell_script_10011","type":"script","value":"retainField(unique_id,container_image_id,k8s_cluster_id,sls_time,app_version,k8s_pod_name,uuid,dst_ip,layer,k8s_node_name,found_out,cmdline,container_name,p_info,dst_port,aliUid,log_conten [...]
-dipper.private.blink.rules&&&&rule&&&&sas_linux_alert_client_reverse_shell_rule_10003&&&&{"expressionStr":"found_out,==,'1'","scriptNames":"[]","expressionName":"11131","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"sas_linux_alert_client_reverse_shell_rule_10003","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[]"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11131&&&&{"aesFlag":1,"varName":"found_out","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"2vQJrhr8/lZ2HEw+NK0Glw=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&sas_linux_alert_client_reverse_shell_script_10012&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"sas_linux_alert_client_reverse_shell_script_10012","type":"script","value":"___concat_10039=concat(dst_ip,dst_port,cmdline);\nabk_raw=___concat_10039;rm('___concat_10039');\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&sas_linux_alert_client_reverse_shell_script_10013&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"sas_linux_alert_client_reverse_shell_script_10013","type":"script","value":"retainField(unique_id,container_image_id,k8s_cluster_id,sls_time,app_version,k8s_pod_name,uuid,dst_ip,layer,k8s_node_name,found_out,cmdline,container_name,p_info,dst_port,aliUid,log_conten [...]
-dipper.private.blink.rules&&&&script&&&&sas_linux_alert_client_reverse_shell_script_10014&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"sas_linux_alert_client_reverse_shell_script_10014","type":"script","value":"uuid=uuid;\nali_uid='';\nclient_ip='';\nevent_type='进程异常行为';\nevent_name='反弹shell_fd';\nlevel='high';\n___cast_10666=cast(dst_ip,'string');\n___cast_10667=cast(dst_port,'string');\n___cast [...]
-dipper.private.blink.rules&&&&script&&&&sas_linux_alert_client_reverse_shell_script_10015&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"sas_linux_alert_client_reverse_shell_script_10015","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&channel&&&&sas_linux_alert_client_reverse_shell_channel_10002&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"sas_linux_alert_client_reverse_shell_channel_10002","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeInOrder":"true","maxFetchLogGroupSize":"100","syncCou [...]
-dipper.private.blink.rules&&&&pipline&&&&blink.direct.source.aegis_client_origin&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.SubPiplineChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"nextStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"filterMsgSwitch\\\\\\\":\\\\\\\"blink.direct.source.aegis_client_origin.filter.switch\\\\\\\",\\\\\\\"cancelAfterConfigurableRefreshListerner\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"className\\\\ [...]
-dipper.private.blink.rules&&&&channel&&&&blink.direct.source.aegis_client_origin;channel&&&&{"offset":"0","isJsonData":"true","maxThread":"12","concurrentCount":"12","className":"com.aliyun.yundun.dipper.channel.adapter.MetaqChannel","configureName":"blink.direct.source.aegis_client_origin;channel","type":"channel","version":"1.0","timeout":"30000","activtyTimeOut":"3000","tags":"blink.direct.source.aegis_client_origin.tags","syncTimeout":"5000","groupName":"blink.direct.source.aegis_cli [...]
-dipper.private.blink.rules&&&&pipline&&&&windows_eventlog_alert&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"nextStageLables\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"windows_eventlog_alert_script_10002\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"cancelAfterConfigurableRefreshListerner\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"ownerSqlNodeTableName\\\\\\\":\\\\\\\"eventlog_input_patch_eventid\\\ [...]
-dipper.private.blink.rules&&&&script&&&&windows_eventlog_alert;json_concat_10016&&&&{"fullClassName":"com.aliyun.sec.lyra.udf.ext.JsonConcat","initMethodName":"open","functionName":"json_concat","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"windows_eventlog_alert;json_concat_10016","type":"script","version":"1.0","closeMethodName":"close","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&channel&&&&windows_eventlog_alert&&&&{"offset":"0","isJsonData":"true","maxThread":"12","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.MetaqChannel","configureName":"windows_eventlog_alert","type":"channel","version":"1.0","timeout":"30000","activtyTimeOut":"3000","syncTimeout":"5000","groupName":"windows_eventlog_alert","isBatchMessage":"true","isAutoFlush":"false","maxFetchLogGroupSize":"100","syncCount":"1000","outputThreadCou [...]
-dipper.private.blink.rules&&&&script&&&&windows_eventlog_alert_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"windows_eventlog_alert_script_10001","type":"script","value":"___json_get_10100=json_get(meta_conf,'$.aliUid');\naliUid=___json_get_10100;rm('___json_get_10100');\n___json_get_10101=json_get(data,'$.raw_xml');\nraw_xml=___json_get_10101;rm('___json_get_10101');\n___json_get_101 [...]
-dipper.private.blink.rules&&&&script&&&&windows_eventlog_alert_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"windows_eventlog_alert_script_10002","type":"script","value":"retainField(raw_xml,channel,app,traceId,meta_conf,app_version,data,ip,safe_mode,type,uuid,version,seq,aliUid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&windows_eventlog_alert_script_10003&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"windows_eventlog_alert_script_10003","type":"script","value":"___REGEXP_EXTRACT_10011=REGEXP_EXTRACT(raw_xml,'\\<EventID\\>(\\d+)\\<\\/EventID\\>',1);\nevent_id=___REGEXP_EXTRACT_10011;rm('___REGEXP_EXTRACT_10011');\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&windows_eventlog_alert_script_10004&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"windows_eventlog_alert_script_10004","type":"script","value":"retainField(event_id,app,traceId,meta_conf,app_version,data,raw_xml,ip,safe_mode,channel,type,uuid,version,aliUid,seq);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&windows_eventlog_alert_script_10005&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"windows_eventlog_alert_script_10005","type":"script","value":"___compare_10258=equals(event_id,'4720');\n___compare_10259=equals(event_id,'4722');\n___compare_10260=equals(event_id,'4726');\n___compare_10261=equals(event_id,'4732');\n___compare_10262=equals(event_id,'4733');\n_ [...]
-dipper.private.blink.rules&&&&script&&&&windows_eventlog_alert_script_10006&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"windows_eventlog_alert_script_10006","type":"script","value":"retainField(app,traceId,meta_conf,app_version,data,raw_xml,ip,safe_mode,channel,type,uuid,version,event_id,aliUid,seq,label);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&windows_eventlog_alert_script_10007&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"windows_eventlog_alert_script_10007","type":"script","value":"___!null_10042=!null(label);\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&windows_eventlog_alert_rule_10001&&&&{"expressionStr":"(___!null_10042)","scriptNames":"[]","expressionName":"11133","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"windows_eventlog_alert_rule_10001","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[]"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11133&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11132\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11132&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"___!null_10042\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&script&&&&windows_eventlog_alert_script_10008&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"windows_eventlog_alert_script_10008","type":"script","value":"retainField(app,traceId,meta_conf,app_version,data,raw_xml,ip,safe_mode,channel,label,type,uuid,version,event_id,aliUid,seq);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&windows_eventlog_alert_script_10009&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"windows_eventlog_alert_script_10009","type":"script","value":"___REGEXP_EXTRACT_10012=REGEXP_EXTRACT(raw_xml,'TargetUserName[\\s\\S]*?\\>\\s*?([^\\<]*?)\\s*?\\<',1);\n___lower____REGEXP_EXTRACT_10012_10001=lower(___REGEXP_EXTRACT_10012);\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&windows_eventlog_alert_rule_10002&&&&{"expressionStr":"(label,in,'\\'evt_user_enabled\\'')&(___lower____REGEXP_EXTRACT_10012_10001,like,'%guest%')","scriptNames":"[]","expressionName":"11136","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"windows_eventlog_alert_rule_10002","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[]"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11135&&&&{"aesFlag":1,"varName":"___lower____REGEXP_EXTRACT_10012_10001","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"6+/CEgfAE5XfqU7j6X2aVA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11134&&&&{"aesFlag":1,"varName":"label","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"HzqPYs1HxLiLowVI3oTE4+eZpFytiFGDP4d9xJwjx/s="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11136&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11134\",\"11135\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&script&&&&windows_eventlog_alert_script_10010&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"windows_eventlog_alert_script_10010","type":"script","value":"retainField(app,traceId,meta_conf,app_version,data,raw_xml,ip,safe_mode,channel,label,type,uuid,version,event_id,aliUid,seq);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&windows_eventlog_alert_script_10011&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"windows_eventlog_alert_script_10011","type":"script","value":"ali_uid=aliUid;\n___cast_10682=cast(null_10021,'string');\nclient_ip=___cast_10682;rm('___cast_10682');\nevent_type='进程异常行为';\nevent_name='Windows Guest账号激活';\nlevel='low';\n___REGEXP_EXTRACT_10013=REGEXP_EXTRACT(raw [...]
-dipper.private.blink.rules&&&&script&&&&windows_eventlog_alert_script_10012&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"windows_eventlog_alert_script_10012","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&channel&&&&windows_eventlog_alert_channel_10001&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"windows_eventlog_alert_channel_10001","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeInOrder":"true","maxFetchLogGroupSize":"100","syncCount":"1000","nameSpace":"dipp [...]
-dipper.private.blink.rules&&&&pipline&&&&blink.direct.source.aegis_eventlog&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.SubPiplineChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"nextStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"filterMsgSwitch\\\\\\\":\\\\\\\"blink.direct.source.aegis_eventlog.filter.switch\\\\\\\",\\\\\\\"cancelAfterConfigurableRefreshListerner\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"className\\\\\\\":\\\\\ [...]
-dipper.private.blink.rules&&&&channel&&&&blink.direct.source.aegis_eventlog;channel&&&&{"offset":"0","isJsonData":"true","maxThread":"12","concurrentCount":"12","className":"com.aliyun.yundun.dipper.channel.adapter.MetaqChannel","configureName":"blink.direct.source.aegis_eventlog;channel","type":"channel","version":"1.0","timeout":"30000","activtyTimeOut":"3000","tags":"blink.direct.source.aegis_eventlog.tags","syncTimeout":"5000","groupName":"blink.direct.source.aegis_eventlog.group","i [...]
-dipper.private.blink.rules&&&&pipline&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"nextStageLables\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_script_10002\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"cancelAfter [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4;json_concat_10017&&&&{"fullClassName":"com.aliyun.sec.lyra.udf.ext.JsonConcat","initMethodName":"open","functionName":"json_concat","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crac [...]
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4&&&&{"offset":"0","isJsonData":"true","maxThread":"12","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.MetaqChannel","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4","type":"channel","version":"1.0","timeout":"30000","activtyTimeOut":"3000","tags":"T_MSG_LOGIN_INFO","syncT [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_script_10001","type":"script","value":"retainField(src_ip,port,success,ip,count,pid,type,uuid,user,log_time,invalid_use [...]
-dipper.private.blink.rules&&&&nameList&&&&30.240.98.174;1616994986574;8&&&&{"indexs":"[\"account_name\"]","pollingTime":"60","className":"com.aliyun.filter.namelist.DBNameList","configureName":"30.240.98.174;1616994986574;8","userName":"blink.source.aegis.login.username","type":"nameList","version":"1.0","url":"blink.source.aegis.login.url","sql":"select account_name from brute_force_suspicious_account limit 1000000","cotainsIpFieldNames":"[]","password":"blink.source.aegis.login.passwor [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_script_10002","type":"script","value":"___dim_10008=left_join('dipper.private.blink.rules','30.240.98.174;1616994986574 [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_script_10003&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_script_10003","type":"script","value":"account_name=p.account_name;\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_script_10004&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_script_10004","type":"script","value":"retainField(account_name,src_ip,port,success,ip,count,pid,type,uuid,user,log_tim [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_script_10005&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_script_10005","type":"script","value":"rule_type='1';\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_script_10006&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_script_10006","type":"script","value":"retainField(rule_type,src_ip,port,success,account_name,ip,count,pid,type,uuid,us [...]
-dipper.private.blink.rules&&&&nameList&&&&30.240.98.174;1616994986575;9&&&&{"indexs":"[\"rule_type\"]","pollingTime":"60","className":"com.aliyun.filter.namelist.DBNameList","configureName":"30.240.98.174;1616994986575;9","userName":"blink.source.aegis.login.account.username","type":"nameList","version":"1.0","url":"blink.source.aegis.login.account.url","sql":"select rule_type,content from brute_force_rule limit 1000000","cotainsIpFieldNames":"[]","password":"blink.source.aegis.login.acc [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_script_10007&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_script_10007","type":"script","value":"___dim_10009=left_join('dipper.private.blink.rules','30.240.98.174;1616994986575 [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_script_10008&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_script_10008","type":"script","value":"___json_get_10103=json_get(p.content,'$.winsize');\n___cast_10683=cast(___json_g [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_script_10009&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_script_10009","type":"script","value":"retainField(fail_times,ip,count,pid,type,uuid,log_time,src_ip,rule_type,port,suc [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_rule_10001&&&&{"expressionStr":"uuid,!like,'inet-%'","scriptNames":"[]","expressionName":"11137","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_rule_10001","ruleStatus":"0","type": [...]
-dipper.private.blink.rules&&&&express&&&&11137&&&&{"aesFlag":1,"varName":"uuid","functionName":"!like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"KHAOzEfzUAKsbrVrfBuQcA=="}&&&&null
-dipper.private.blink.rules&&&&window&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_window_10001&&&&{"groupByFieldName":"type;src_ip;uuid;user;ip;port;winsize;fail_times","waterMarkMinute":"0","sizeAdjust":"1","sizeVariable":"winsize","className":"com.aliyun.yundun.dipper.window.model.WindowMessageProcessor","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_window_10001","type":"windo [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_script_10010&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_script_10010","type":"script","value":"retainField(succ_cns,ip,winsize,type,uuid,log_time,src_ip,winStart,winEnd,port,a [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_script_10011&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_script_10011","type":"script","value":"__compare_value_10008=!((src_ip,regex,'^127\\.'))\n;___!null_10043=!null(account [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_rule_10002&&&&{"expressionStr":"(__compare_value_10008&(succ_cns,>,double,0))&___!null_10043","scriptNames":"[]","expressionName":"11140","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_aler [...]
-dipper.private.blink.rules&&&&express&&&&11140&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11139\",\"___!null_10043\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11139&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"__compare_value_10008\",\"11138\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11138&&&&{"varName":"succ_cns","functionName":">","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0.0"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_script_10012&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_script_10012","type":"script","value":"country='';\nprovince='';\ncity='';\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_script_10013&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_script_10013","type":"script","value":"retainField(country,province,city,succ_cns,ip,winsize,type,uuid,log_time,src_ip, [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_script_10014&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_script_10014","type":"script","value":"retainField(country,succ_cns,city,ip,winsize,type,uuid,log_time,src_ip,winStart, [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_rule_10003&&&&{"expressionStr":"type,==,'SSH'","scriptNames":"[]","expressionName":"11141","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_rule_10003","ruleStatus":"0","type":"rule" [...]
-dipper.private.blink.rules&&&&express&&&&11141&&&&{"aesFlag":1,"varName":"type","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"yWD8YZYVXaFbtnsN24tuWg=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_script_10015&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_script_10015","type":"script","value":"ali_uid='';\nclient_ip=ip;\nevent_type='异常登录';\nevent_name='ECS被可疑账号登录成功(SSH)';\ [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_script_10016&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_script_10016","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_rule_10004&&&&{"expressionStr":"type,==,'RDP'","scriptNames":"[]","expressionName":"11142","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_rule_10004","ruleStatus":"0","type":"rule" [...]
-dipper.private.blink.rules&&&&express&&&&11142&&&&{"aesFlag":1,"varName":"type","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"XaH22EWkvLN7G+itcFtlvQ=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_script_10017&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_script_10017","type":"script","value":"ali_uid='';\nclient_ip=ip;\nevent_type='异常登录';\nevent_name='ECS被可疑账号登录成功(RDP)';\ [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_script_10018&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_script_10018","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_script_10019&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_script_10019","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content [...]
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_channel_10001&&&&{"offset":"0","isJsonData":"true","maxThread":"12","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.MetaqChannel","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_channel_10001","type":"channel","version":"1.0","timeout":"30000","activtyTimeOut":"3000","tag [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_script_10020&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_script_10020","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content [...]
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_channel_10002&&&&{"offset":"0","isJsonData":"true","maxThread":"12","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.MetaqChannel","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_account_alert_v4_channel_10002","type":"channel","version":"1.0","timeout":"30000","activtyTimeOut":"3000","tag [...]
-dipper.private.blink.rules&&&&pipline&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"nextStageLables\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_script_10002\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"cancelAfterConfig [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4;json_concat_10018&&&&{"fullClassName":"com.aliyun.sec.lyra.udf.ext.JsonConcat","initMethodName":"open","functionName":"json_concat","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_f [...]
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4&&&&{"offset":"0","isJsonData":"true","maxThread":"12","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.MetaqChannel","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4","type":"channel","version":"1.0","timeout":"30000","activtyTimeOut":"3000","tags":"T_MSG_LOGIN_INFO","syncTimeout [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_script_10001","type":"script","value":"retainField(src_ip,port,success,ip,count,pid,type,uuid,user,log_time,invalid_user);"," [...]
-dipper.private.blink.rules&&&&nameList&&&&30.240.98.174;1616994988318;10&&&&{"indexs":"[\"account_name\"]","pollingTime":"60","className":"com.aliyun.filter.namelist.DBNameList","configureName":"30.240.98.174;1616994988318;10","userName":"blink.source.aegis.login.username","type":"nameList","version":"1.0","url":"blink.source.aegis.login.url","sql":"select account_name from brute_force_suspicious_account limit 1000000","cotainsIpFieldNames":"[]","password":"blink.source.aegis.login.passw [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_script_10002","type":"script","value":"___dim_10010=left_join('dipper.private.blink.rules','30.240.98.174;1616994988318;10',' [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_script_10003&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_script_10003","type":"script","value":"account_name=p.account_name;\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_script_10004&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_script_10004","type":"script","value":"retainField(account_name,src_ip,port,success,ip,count,pid,type,uuid,user,log_time,inva [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_script_10005&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_script_10005","type":"script","value":"rule_type='1';\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_script_10006&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_script_10006","type":"script","value":"retainField(rule_type,src_ip,port,success,account_name,ip,count,pid,type,uuid,user,log [...]
-dipper.private.blink.rules&&&&nameList&&&&30.240.98.174;1616994988318;11&&&&{"indexs":"[\"rule_type\"]","pollingTime":"60","className":"com.aliyun.filter.namelist.DBNameList","configureName":"30.240.98.174;1616994988318;11","userName":"blink.source.aegis.login.account.username","type":"nameList","version":"1.0","url":"blink.source.aegis.login.account.url","sql":"select rule_type,content from brute_force_rule limit 1000000","cotainsIpFieldNames":"[]","password":"blink.source.aegis.login.a [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_script_10007&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_script_10007","type":"script","value":"___dim_10011=left_join('dipper.private.blink.rules','30.240.98.174;1616994988318;11',' [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_script_10008&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_script_10008","type":"script","value":"___json_get_10105=json_get(p.content,'$.winsize');\n___cast_10709=cast(___json_get_101 [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_script_10009&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_script_10009","type":"script","value":"retainField(fail_times,ip,count,pid,type,uuid,log_time,src_ip,rule_type,port,success,a [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_rule_10001&&&&{"expressionStr":"uuid,!like,'inet-%'","scriptNames":"[]","expressionName":"11143","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_rule_10001","ruleStatus":"0","type":"rule" [...]
-dipper.private.blink.rules&&&&express&&&&11143&&&&{"aesFlag":1,"varName":"uuid","functionName":"!like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"KHAOzEfzUAKsbrVrfBuQcA=="}&&&&null
-dipper.private.blink.rules&&&&window&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_window_10001&&&&{"groupByFieldName":"type;src_ip;uuid;user;ip;port;winsize;fail_times","waterMarkMinute":"0","sizeAdjust":"1","sizeVariable":"winsize","className":"com.aliyun.yundun.dipper.window.model.WindowMessageProcessor","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_window_10001","type":"window","ve [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_script_10010&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_script_10010","type":"script","value":"retainField(succ_cns,ip,winsize,type,uuid,log_time,src_ip,winStart,winEnd,port,account [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_script_10011&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_script_10011","type":"script","value":"__compare_value_10009=!((src_ip,regex,'^127\\.'))\n;___!null_10044=!null(account_name) [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_rule_10002&&&&{"expressionStr":"(__compare_value_10009&(succ_cns,==,double,0))&___!null_10044","scriptNames":"[]","expressionName":"11146","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_ [...]
-dipper.private.blink.rules&&&&express&&&&11144&&&&{"varName":"succ_cns","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0.0"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11146&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11145\",\"___!null_10044\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11145&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"__compare_value_10009\",\"11144\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_script_10012&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_script_10012","type":"script","value":"country='';\nprovince='';\ncity='';\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_script_10013&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_script_10013","type":"script","value":"retainField(country,province,city,succ_cns,ip,winsize,type,uuid,log_time,src_ip,winSta [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_script_10014&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_script_10014","type":"script","value":"retainField(country,succ_cns,city,ip,winsize,type,uuid,log_time,src_ip,winStart,provin [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_rule_10003&&&&{"expressionStr":"type,==,'SSH'","scriptNames":"[]","expressionName":"11147","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_rule_10003","ruleStatus":"0","type":"rule","vers [...]
-dipper.private.blink.rules&&&&express&&&&11147&&&&{"aesFlag":1,"varName":"type","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"yWD8YZYVXaFbtnsN24tuWg=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_script_10015&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_script_10015","type":"script","value":"ali_uid='';\nclient_ip=ip;\nevent_type='异常登录';\nevent_name='ECS被可疑账号登录失败(SSH)';\nlevel [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_script_10016&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_script_10016","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_m [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_rule_10004&&&&{"expressionStr":"type,==,'RDP'","scriptNames":"[]","expressionName":"11148","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_rule_10004","ruleStatus":"0","type":"rule","vers [...]
-dipper.private.blink.rules&&&&express&&&&11148&&&&{"aesFlag":1,"varName":"type","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"XaH22EWkvLN7G+itcFtlvQ=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_script_10017&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_script_10017","type":"script","value":"ali_uid='';\nclient_ip=ip;\nevent_type='异常登录';\nevent_name='ECS被可疑账号登录失败(RDP)';\nlevel [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_script_10018&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_script_10018","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_m [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_script_10019&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_script_10019","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_m [...]
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_channel_10001&&&&{"offset":"0","isJsonData":"true","maxThread":"12","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.MetaqChannel","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_channel_10001","type":"channel","version":"1.0","timeout":"30000","activtyTimeOut":"3000","tags":"T_ [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_script_10020&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_script_10020","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_m [...]
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_channel_10002&&&&{"offset":"0","isJsonData":"true","maxThread":"12","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.MetaqChannel","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_account_alert_v4_channel_10002","type":"channel","version":"1.0","timeout":"30000","activtyTimeOut":"3000","tags":"T_ [...]
-dipper.private.blink.rules&&&&pipline&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"nextStageLables\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10002\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"cancelAfterConfigurableRefreshLis [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4;json_concat_10019&&&&{"fullClassName":"com.aliyun.sec.lyra.udf.ext.JsonConcat","initMethodName":"open","functionName":"json_concat","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_aler [...]
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4&&&&{"offset":"0","isJsonData":"true","maxThread":"12","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.MetaqChannel","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4","type":"channel","version":"1.0","timeout":"30000","activtyTimeOut":"3000","tags":"T_MSG_LOGIN_INFO","syncTimeout":"5000","groupN [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10001","type":"script","value":"retainField(src_ip,port,success,ip,count,pid,type,uuid,user,log_time,invalid_user);","version":"1.0"," [...]
-dipper.private.blink.rules&&&&nameList&&&&30.240.98.174;1616994990145;12&&&&{"indexs":"[\"account_name\"]","pollingTime":"60","className":"com.aliyun.filter.namelist.DBNameList","configureName":"30.240.98.174;1616994990145;12","userName":"blink.source.aegis.login.username","type":"nameList","version":"1.0","url":"blink.source.aegis.login.url","sql":"select account_name from brute_force_suspicious_account limit 1000000","cotainsIpFieldNames":"[]","password":"blink.source.aegis.login.passw [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10002","type":"script","value":"___dim_10012=left_join('dipper.private.blink.rules','30.240.98.174;1616994990145;12','(user,==,account [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10003&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10003","type":"script","value":"account_name=p.account_name;\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10004&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10004","type":"script","value":"retainField(account_name,src_ip,port,success,ip,count,pid,type,uuid,user,log_time,invalid_user);","ver [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10005&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10005","type":"script","value":"rule_type='2';\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10006&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10006","type":"script","value":"retainField(rule_type,src_ip,port,success,ip,count,pid,type,uuid,user,log_time,invalid_user);","versio [...]
-dipper.private.blink.rules&&&&nameList&&&&30.240.98.174;1616994990145;13&&&&{"indexs":"[\"rule_type\"]","pollingTime":"60","className":"com.aliyun.filter.namelist.DBNameList","configureName":"30.240.98.174;1616994990145;13","userName":"blink.source.aegis.login.account.username","type":"nameList","version":"1.0","url":"blink.source.aegis.login.account.url","sql":"select rule_type,content from brute_force_rule limit 1000000","cotainsIpFieldNames":"[]","password":"blink.source.aegis.login.a [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10007&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10007","type":"script","value":"___dim_10013=left_join('dipper.private.blink.rules','30.240.98.174;1616994990145;13','(rule_type,==,ru [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10008&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10008","type":"script","value":"___json_get_10109=json_get(p.content,'$.winsize');\n___cast_10737=cast(___json_get_10109,'int');\nwins [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10009&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10009","type":"script","value":"retainField(src_ip,rule_type,port,success,ip,count,pid,type,uuid,user,log_time,invalid_user,winsize,se [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_rule_10001&&&&{"expressionStr":"uuid,!like,'inet-%'","scriptNames":"[]","expressionName":"11149","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_rule_10001","ruleStatus":"0","type":"rule","version":"1.0" [...]
-dipper.private.blink.rules&&&&express&&&&11149&&&&{"aesFlag":1,"varName":"uuid","functionName":"!like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"KHAOzEfzUAKsbrVrfBuQcA=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10010&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10010","type":"script","value":"___cast_10743=cast(invalid_user,'long');\n___compare_10281=equals(___cast_10743,1);\nif(___compare_102 [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10011&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10011","type":"script","value":"retainField(src_ip,port,success,ip,count,type,uuid,invalid_users,log_time,valid_users);","version":"1. [...]
-dipper.private.blink.rules&&&&window&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_window_10001&&&&{"groupByFieldName":"type;src_ip;uuid;ip;port","waterMarkMinute":"0","sizeAdjust":"0","className":"com.aliyun.yundun.dipper.window.model.WindowMessageProcessor","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_window_10001","type":"window","version":"1.0","windowType":"hop","fireDelaySecond":"30","timeout": [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10012&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10012","type":"script","value":"retainField(succ_cns,invalid_users_list,ip,invalid_users_cnt,type,uuid,log_time,src_ip,winStart,winEnd [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10013&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10013","type":"script","value":"rule_type='1';\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10014&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10014","type":"script","value":"retainField(rule_type,src_ip,port,success,account_name,ip,count,pid,type,uuid,user,log_time,invalid_us [...]
-dipper.private.blink.rules&&&&nameList&&&&30.240.98.174;1616994990145;14&&&&{"indexs":"[\"rule_type\"]","pollingTime":"60","className":"com.aliyun.filter.namelist.DBNameList","configureName":"30.240.98.174;1616994990145;14","userName":"blink.source.aegis.login.account.username","type":"nameList","version":"1.0","url":"blink.source.aegis.login.account.url","sql":"select rule_type,content from brute_force_rule limit 1000000","cotainsIpFieldNames":"[]","password":"blink.source.aegis.login.a [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10015&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10015","type":"script","value":"___dim_10014=left_join('dipper.private.blink.rules','30.240.98.174;1616994990145;14','(rule_type,==,ru [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10016&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10016","type":"script","value":"___json_get_10107=json_get(p.content,'$.winsize');\n___cast_10735=cast(___json_get_10107,'int');\nwins [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10017&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10017","type":"script","value":"retainField(fail_times,ip,count,pid,type,uuid,log_time,src_ip,rule_type,port,success,account_name,user [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_rule_10002&&&&{"expressionStr":"uuid,!like,'inet-%'","scriptNames":"[]","expressionName":"11150","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_rule_10002","ruleStatus":"0","type":"rule","version":"1.0" [...]
-dipper.private.blink.rules&&&&express&&&&11150&&&&{"aesFlag":1,"varName":"uuid","functionName":"!like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"KHAOzEfzUAKsbrVrfBuQcA=="}&&&&null
-dipper.private.blink.rules&&&&window&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_window_10002&&&&{"groupByFieldName":"type;src_ip;uuid;user;ip;port;winsize;fail_times","waterMarkMinute":"0","sizeAdjust":"1","sizeVariable":"winsize","className":"com.aliyun.yundun.dipper.window.model.WindowMessageProcessor","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_window_10002","type":"window","version":"1.0","wi [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10018&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10018","type":"script","value":"retainField(succ_cns,ip,winsize,type,uuid,log_time,src_ip,winStart,winEnd,port,account_name,fail_cns,f [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10019&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10019","type":"script","value":"__compare_value_10010=!((src_ip,regex,'^127\\.'))\n;___subtraction_10012=subtraction(fail_cns,fail_tim [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_rule_10003&&&&{"expressionStr":"(__compare_value_10010&(___subtraction_10012,>=,double,0))&(succ_cns,==,double,0)","scriptNames":"[]","expressionName":"11154","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert [...]
-dipper.private.blink.rules&&&&express&&&&11151&&&&{"varName":"___subtraction_10012","functionName":">=","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0.0"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11153&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"__compare_value_10010\",\"11151\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11152&&&&{"varName":"succ_cns","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0.0"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11154&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11153\",\"11152\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10020&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10020","type":"script","value":"country='';\nprovince='';\ncity='';\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10021&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10021","type":"script","value":"retainField(country,province,city,succ_cns,ip,winsize,type,uuid,log_time,src_ip,winStart,winEnd,port,a [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_rule_10004&&&&{"expressionStr":"type,==,'SSH'","scriptNames":"[]","expressionName":"11155","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_rule_10004","ruleStatus":"0","type":"rule","version":"1.0","exte [...]
-dipper.private.blink.rules&&&&express&&&&11155&&&&{"aesFlag":1,"varName":"type","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"yWD8YZYVXaFbtnsN24tuWg=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10022&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10022","type":"script","value":"ali_uid='';\nclient_ip=ip;\nevent_type='异常登录';\nevent_name='ECS被点对点暴力破解失败(SSH)';\nlevel='medium';\n___ [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10023&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10023","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);", [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_rule_10005&&&&{"expressionStr":"type,==,'RDP'","scriptNames":"[]","expressionName":"11156","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_rule_10005","ruleStatus":"0","type":"rule","version":"1.0","exte [...]
-dipper.private.blink.rules&&&&express&&&&11156&&&&{"aesFlag":1,"varName":"type","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"XaH22EWkvLN7G+itcFtlvQ=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10024&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10024","type":"script","value":"ali_uid='';\nclient_ip=ip;\nevent_type='异常登录';\nevent_name='ECS被点对点暴力破解失败(RDP)';\nlevel='medium';\n___ [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10025&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10025","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);", [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10026&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10026","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);", [...]
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_channel_10001&&&&{"offset":"0","isJsonData":"true","maxThread":"12","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.MetaqChannel","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_channel_10001","type":"channel","version":"1.0","timeout":"30000","activtyTimeOut":"3000","tags":"T_MSG_BRUTE_FORCE" [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10027&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10027","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);", [...]
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_channel_10002&&&&{"offset":"0","isJsonData":"true","maxThread":"12","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.MetaqChannel","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_channel_10002","type":"channel","version":"1.0","timeout":"30000","activtyTimeOut":"3000","tags":"T_MSG_BRUTE_FORCE" [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_rule_10006&&&&{"expressionStr":"uuid,!like,'inet-%'","scriptNames":"[]","expressionName":"11157","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_rule_10006","ruleStatus":"0","type":"rule","version":"1.0" [...]
-dipper.private.blink.rules&&&&express&&&&11157&&&&{"aesFlag":1,"varName":"uuid","functionName":"!like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"KHAOzEfzUAKsbrVrfBuQcA=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10028&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10028","type":"script","value":"___cast_10749=cast(invalid_user,'long');\n___compare_10285=equals(___cast_10749,1);\nif(___compare_102 [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10029&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10029","type":"script","value":"retainField(unsuccess_hosts,ip,count,winsize,type,uuid,success_hosts,log_time,valid_users,src_ip,port, [...]
-dipper.private.blink.rules&&&&window&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_window_10003&&&&{"groupByFieldName":"type;src_ip;port;winsize;server_count","waterMarkMinute":"0","sizeAdjust":"1","sizeVariable":"winsize","className":"com.aliyun.yundun.dipper.window.model.WindowMessageProcessor","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_window_10003","type":"window","version":"1.0","windowType":" [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10030&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10030","type":"script","value":"retainField(succ_cns,ip,invalid_users_list,invalid_users_cnt,winsize,type,uuid,log_time,unsuccess_host [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10031&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10031","type":"script","value":"__compare_value_10012=!((src_ip,regex,'^127\\.'))\n;___subtraction_10014=subtraction(unsuccess_hosts_c [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_rule_10007&&&&{"expressionStr":"(__compare_value_10012&(success_hosts_cnt,==,double,0))&(___subtraction_10014,>=,double,0)","scriptNames":"[]","expressionName":"11161","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_f [...]
-dipper.private.blink.rules&&&&express&&&&11160&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"__compare_value_10012\",\"11158\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11161&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11160\",\"11159\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11159&&&&{"varName":"___subtraction_10014","functionName":">=","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0.0"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11158&&&&{"varName":"success_hosts_cnt","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0.0"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10032&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10032","type":"script","value":"country='';\nprovince='';\ncity='';\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10033&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10033","type":"script","value":"retainField(country,province,city,succ_cns,ip,invalid_users_list,invalid_users_cnt,winsize,type,uuid,l [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_rule_10008&&&&{"expressionStr":"type,==,'SSH'","scriptNames":"[]","expressionName":"11162","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_rule_10008","ruleStatus":"0","type":"rule","version":"1.0","exte [...]
-dipper.private.blink.rules&&&&express&&&&11162&&&&{"aesFlag":1,"varName":"type","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"yWD8YZYVXaFbtnsN24tuWg=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10034&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10034","type":"script","value":"uuid=uuid;\nali_uid='';\nclient_ip='';\nevent_type='异常登录';\nevent_name='批量异常登录失败(SSH）';\nlevel='medium [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10035&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10035","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);", [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_rule_10009&&&&{"expressionStr":"type,==,'RDP'","scriptNames":"[]","expressionName":"11163","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_rule_10009","ruleStatus":"0","type":"rule","version":"1.0","exte [...]
-dipper.private.blink.rules&&&&express&&&&11163&&&&{"aesFlag":1,"varName":"type","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"XaH22EWkvLN7G+itcFtlvQ=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10036&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10036","type":"script","value":"uuid=uuid;\nali_uid='';\nclient_ip='';\nevent_type='异常登录';\nevent_name='批量异常登录失败(RDP）';\nlevel='medium [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10037&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10037","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);", [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10038&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10038","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);", [...]
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_channel_10003&&&&{"offset":"0","isJsonData":"true","maxThread":"12","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.MetaqChannel","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_channel_10003","type":"channel","version":"1.0","timeout":"30000","activtyTimeOut":"3000","tags":"T_MSG_BRUTE_FORCE" [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10039&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10039","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);", [...]
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_channel_10004&&&&{"offset":"0","isJsonData":"true","maxThread":"12","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.MetaqChannel","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_channel_10004","type":"channel","version":"1.0","timeout":"30000","activtyTimeOut":"3000","tags":"T_MSG_BRUTE_FORCE" [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10040&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10040","type":"script","value":"__compare_value_10011=!((src_ip,regex,'^127\\.'))\n;___subtraction_10013=subtraction(fail_cns,invalid_ [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_rule_10010&&&&{"expressionStr":"(((__compare_value_10011&(succ_cns,==,double,0))&(___subtraction_10013,>=,double,0))&(invalid_users_cnt,>,double,3))&(valid_users_cnt,>,double,0)","scriptNames":"[]","expressionName":"11171","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source [...]
-dipper.private.blink.rules&&&&express&&&&11171&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11170\",\"11167\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11170&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11169\",\"11166\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11164&&&&{"varName":"succ_cns","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0.0"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11166&&&&{"varName":"invalid_users_cnt","functionName":">","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"3.0"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11165&&&&{"varName":"___subtraction_10013","functionName":">=","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0.0"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11168&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"__compare_value_10011\",\"11164\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11167&&&&{"varName":"valid_users_cnt","functionName":">","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0.0"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11169&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11168\",\"11165\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10041&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10041","type":"script","value":"country='';\nprovince='';\ncity='';\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10042&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10042","type":"script","value":"retainField(country,province,city,succ_cns,invalid_users_list,ip,invalid_users_cnt,type,uuid,log_time, [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_rule_10011&&&&{"expressionStr":"type,==,'SSH'","scriptNames":"[]","expressionName":"11172","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_rule_10011","ruleStatus":"0","type":"rule","version":"1.0","exte [...]
-dipper.private.blink.rules&&&&express&&&&11172&&&&{"aesFlag":1,"varName":"type","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"yWD8YZYVXaFbtnsN24tuWg=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10043&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10043","type":"script","value":"ali_uid='';\nclient_ip=ip;\nevent_type='异常登录';\nevent_name='ECS被多个无效用户暴力破解失败(SSH)';\nlevel='medium';\n [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10044&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10044","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);", [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_rule_10012&&&&{"expressionStr":"type,==,'RDP'","scriptNames":"[]","expressionName":"11173","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_rule_10012","ruleStatus":"0","type":"rule","version":"1.0","exte [...]
-dipper.private.blink.rules&&&&express&&&&11173&&&&{"aesFlag":1,"varName":"type","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"XaH22EWkvLN7G+itcFtlvQ=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10045&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10045","type":"script","value":"ali_uid='';\nclient_ip=ip;\nevent_type='异常登录';\nevent_name='ECS被多个无效用户暴力破解失败(RDP)';\nlevel='medium';\n [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10046&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10046","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);", [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10047&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10047","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);", [...]
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_channel_10005&&&&{"offset":"0","isJsonData":"true","maxThread":"12","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.MetaqChannel","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_channel_10005","type":"channel","version":"1.0","timeout":"30000","activtyTimeOut":"3000","tags":"T_MSG_BRUTE_FORCE" [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10048&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_script_10048","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);", [...]
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_channel_10006&&&&{"offset":"0","isJsonData":"true","maxThread":"12","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.MetaqChannel","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_fail_alert_v4_channel_10006","type":"channel","version":"1.0","timeout":"30000","activtyTimeOut":"3000","tags":"T_MSG_BRUTE_FORCE" [...]
-dipper.private.blink.rules&&&&pipline&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"nextStageLables\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10002\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"cancelAfterConfigurableRefr [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4;json_concat_10020&&&&{"fullClassName":"com.aliyun.sec.lyra.udf.ext.JsonConcat","initMethodName":"open","functionName":"json_concat","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_succes [...]
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4&&&&{"offset":"0","isJsonData":"true","maxThread":"12","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.MetaqChannel","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4","type":"channel","version":"1.0","timeout":"30000","activtyTimeOut":"3000","tags":"T_MSG_LOGIN_INFO","syncTimeout":"5000"," [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10001","type":"script","value":"retainField(src_ip,port,success,ip,count,pid,type,uuid,user,log_time,invalid_user);","version":" [...]
-dipper.private.blink.rules&&&&nameList&&&&30.240.98.174;1616994994427;15&&&&{"indexs":"[\"account_name\"]","pollingTime":"60","className":"com.aliyun.filter.namelist.DBNameList","configureName":"30.240.98.174;1616994994427;15","userName":"blink.source.aegis.login.username","type":"nameList","version":"1.0","url":"blink.source.aegis.login.url","sql":"select account_name from brute_force_suspicious_account limit 1000000","cotainsIpFieldNames":"[]","password":"blink.source.aegis.login.passw [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10002","type":"script","value":"___dim_10015=left_join('dipper.private.blink.rules','30.240.98.174;1616994994427;15','(user,==,a [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10003&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10003","type":"script","value":"account_name=p.account_name;\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10004&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10004","type":"script","value":"retainField(account_name,src_ip,port,success,ip,count,pid,type,uuid,user,log_time,invalid_user); [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10005&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10005","type":"script","value":"rule_type='2';\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10006&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10006","type":"script","value":"retainField(rule_type,src_ip,port,success,ip,count,pid,type,uuid,user,log_time,invalid_user);"," [...]
-dipper.private.blink.rules&&&&nameList&&&&30.240.98.174;1616994994428;16&&&&{"indexs":"[\"rule_type\"]","pollingTime":"60","className":"com.aliyun.filter.namelist.DBNameList","configureName":"30.240.98.174;1616994994428;16","userName":"blink.source.aegis.login.account.username","type":"nameList","version":"1.0","url":"blink.source.aegis.login.account.url","sql":"select rule_type,content from brute_force_rule limit 1000000","cotainsIpFieldNames":"[]","password":"blink.source.aegis.login.a [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10007&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10007","type":"script","value":"___dim_10016=left_join('dipper.private.blink.rules','30.240.98.174;1616994994428;16','(rule_type [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10008&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10008","type":"script","value":"___json_get_10113=json_get(p.content,'$.winsize');\n___cast_10843=cast(___json_get_10113,'int'); [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10009&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10009","type":"script","value":"retainField(src_ip,rule_type,port,success,ip,count,pid,type,uuid,user,log_time,invalid_user,wins [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_rule_10001&&&&{"expressionStr":"uuid,!like,'inet-%'","scriptNames":"[]","expressionName":"11174","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_rule_10001","ruleStatus":"0","type":"rule","version" [...]
-dipper.private.blink.rules&&&&express&&&&11174&&&&{"aesFlag":1,"varName":"uuid","functionName":"!like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"KHAOzEfzUAKsbrVrfBuQcA=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10010&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10010","type":"script","value":"___cast_10853=cast(invalid_user,'long');\n___compare_10295=equals(___cast_10853,1);\nif(___compa [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10011&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10011","type":"script","value":"retainField(src_ip,port,success,ip,count,type,uuid,invalid_users,log_time,valid_users);","versio [...]
-dipper.private.blink.rules&&&&window&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_window_10001&&&&{"groupByFieldName":"type;src_ip;uuid;ip;port","waterMarkMinute":"0","sizeAdjust":"0","className":"com.aliyun.yundun.dipper.window.model.WindowMessageProcessor","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_window_10001","type":"window","version":"1.0","windowType":"hop","fireDelaySecond":"30","tim [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10012&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10012","type":"script","value":"retainField(succ_cns,invalid_users_list,ip,invalid_users_cnt,type,uuid,log_time,src_ip,winStart, [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10013&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10013","type":"script","value":"rule_type='1';\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10014&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10014","type":"script","value":"retainField(rule_type,src_ip,port,success,account_name,ip,count,pid,type,uuid,user,log_time,inva [...]
-dipper.private.blink.rules&&&&nameList&&&&30.240.98.174;1616994994428;17&&&&{"indexs":"[\"rule_type\"]","pollingTime":"60","className":"com.aliyun.filter.namelist.DBNameList","configureName":"30.240.98.174;1616994994428;17","userName":"blink.source.aegis.login.account.username","type":"nameList","version":"1.0","url":"blink.source.aegis.login.account.url","sql":"select rule_type,content from brute_force_rule limit 1000000","cotainsIpFieldNames":"[]","password":"blink.source.aegis.login.a [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10015&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10015","type":"script","value":"___dim_10017=left_join('dipper.private.blink.rules','30.240.98.174;1616994994428;17','(rule_type [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10016&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10016","type":"script","value":"___json_get_10111=json_get(p.content,'$.winsize');\n___cast_10841=cast(___json_get_10111,'int'); [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10017&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10017","type":"script","value":"retainField(fail_times,ip,count,pid,type,uuid,log_time,src_ip,rule_type,port,success,account_nam [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_rule_10002&&&&{"expressionStr":"uuid,!like,'inet-%'","scriptNames":"[]","expressionName":"11175","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_rule_10002","ruleStatus":"0","type":"rule","version" [...]
-dipper.private.blink.rules&&&&express&&&&11175&&&&{"aesFlag":1,"varName":"uuid","functionName":"!like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"KHAOzEfzUAKsbrVrfBuQcA=="}&&&&null
-dipper.private.blink.rules&&&&window&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_window_10002&&&&{"groupByFieldName":"type;src_ip;uuid;user;ip;port;winsize;fail_times","waterMarkMinute":"0","sizeAdjust":"1","sizeVariable":"winsize","className":"com.aliyun.yundun.dipper.window.model.WindowMessageProcessor","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_window_10002","type":"window","version":"1. [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10018&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10018","type":"script","value":"retainField(succ_cns,ip,winsize,type,uuid,log_time,src_ip,winStart,winEnd,port,account_name,fail [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_rule_10003&&&&{"expressionStr":"uuid,!like,'inet-%'","scriptNames":"[]","expressionName":"11176","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_rule_10003","ruleStatus":"0","type":"rule","version" [...]
-dipper.private.blink.rules&&&&express&&&&11176&&&&{"aesFlag":1,"varName":"uuid","functionName":"!like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"KHAOzEfzUAKsbrVrfBuQcA=="}&&&&null
-dipper.private.blink.rules&&&&window&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_window_10003&&&&{"groupByFieldName":"type;src_ip;uuid;user;ip;port;winsize;fail_times","waterMarkMinute":"0","sizeAdjust":"1","sizeVariable":"winsize","className":"com.aliyun.yundun.dipper.window.model.WindowMessageProcessor","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_window_10003","type":"window","version":"1. [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10019&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10019","type":"script","value":"retainField(succ_cns,ip,winsize,type,uuid,log_time,src_ip,winStart,winEnd,port,account_name,fail [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10020&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10020","type":"script","value":"__compare_value_10013=!((src_ip,regex,'^127\\.'))\n;___subtraction_10015=subtraction(fail_times, [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_rule_10004&&&&{"expressionStr":"((__compare_value_10013&(succ_cns,<=,double,3))&(succ_cns,>,double,0))&(___subtraction_10015,>=,double,0)","scriptNames":"[]","expressionName":"11182","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_ [...]
-dipper.private.blink.rules&&&&express&&&&11180&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"__compare_value_10013\",\"11177\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11182&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11181\",\"11179\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11181&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11180\",\"11178\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11177&&&&{"varName":"succ_cns","functionName":"<=","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"3.0"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11179&&&&{"varName":"___subtraction_10015","functionName":">=","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0.0"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11178&&&&{"varName":"succ_cns","functionName":">","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0.0"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10021&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10021","type":"script","value":"country='';\nprovince='';\ncity='';\nrule_type='1';\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10022&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10022","type":"script","value":"retainField(country,rule_type,province,city,succ_cns,ip,winsize,type,uuid,log_time,src_ip,winSta [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_rule_10005&&&&{"expressionStr":"type,==,'SSH'","scriptNames":"[]","expressionName":"11183","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_rule_10005","ruleStatus":"0","type":"rule","version":"1.0" [...]
-dipper.private.blink.rules&&&&express&&&&11183&&&&{"aesFlag":1,"varName":"type","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"yWD8YZYVXaFbtnsN24tuWg=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10023&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10023","type":"script","value":"ali_uid='';\nclient_ip=ip;\nevent_type='异常登录';\nevent_name='ECS被点对点暴力破解成功(SSH)';\nlevel='medium' [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10024&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10024","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uu [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_rule_10006&&&&{"expressionStr":"type,==,'RDP'","scriptNames":"[]","expressionName":"11184","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_rule_10006","ruleStatus":"0","type":"rule","version":"1.0" [...]
-dipper.private.blink.rules&&&&express&&&&11184&&&&{"aesFlag":1,"varName":"type","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"XaH22EWkvLN7G+itcFtlvQ=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10025&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10025","type":"script","value":"ali_uid='';\nclient_ip=ip;\nevent_type='异常登录';\nevent_name='ECS被点对点暴力破解成功(RDP)';\nlevel='medium' [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10026&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10026","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uu [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10027&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10027","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uu [...]
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_channel_10001&&&&{"offset":"0","isJsonData":"true","maxThread":"12","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.MetaqChannel","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_channel_10001","type":"channel","version":"1.0","timeout":"30000","activtyTimeOut":"3000","tags":"T_MSG_BRUTE_ [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10028&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10028","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uu [...]
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_channel_10002&&&&{"offset":"0","isJsonData":"true","maxThread":"12","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.MetaqChannel","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_channel_10002","type":"channel","version":"1.0","timeout":"30000","activtyTimeOut":"3000","tags":"T_MSG_BRUTE_ [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_rule_10007&&&&{"expressionStr":"uuid,!like,'inet-%'","scriptNames":"[]","expressionName":"11185","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_rule_10007","ruleStatus":"0","type":"rule","version" [...]
-dipper.private.blink.rules&&&&express&&&&11185&&&&{"aesFlag":1,"varName":"uuid","functionName":"!like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"KHAOzEfzUAKsbrVrfBuQcA=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10029&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10029","type":"script","value":"___cast_10859=cast(invalid_user,'long');\n___compare_10299=equals(___cast_10859,1);\nif(___compa [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10030&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10030","type":"script","value":"retainField(unsuccess_hosts,ip,count,winsize,type,uuid,success_hosts,log_time,valid_users,src_ip [...]
-dipper.private.blink.rules&&&&window&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_window_10004&&&&{"groupByFieldName":"type;src_ip;port;winsize;server_count","waterMarkMinute":"0","sizeAdjust":"1","sizeVariable":"winsize","className":"com.aliyun.yundun.dipper.window.model.WindowMessageProcessor","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_window_10004","type":"window","version":"1.0","windowT [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10031&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10031","type":"script","value":"retainField(succ_cns,ip,invalid_users_list,invalid_users_cnt,winsize,type,uuid,log_time,unsucces [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10032&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10032","type":"script","value":"__compare_value_10015=!((src_ip,regex,'^127\\.'))\n;___subtraction_10017=subtraction(success_hos [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_rule_10008&&&&{"expressionStr":"(__compare_value_10015&(valid_users_cnt,<=,double,3))&(((___subtraction_10017,>=,double,0)&(unsuccess_hosts_cnt,>,double,0))|((success_hosts_cnt,>,double,0)&(___subtraction_10018,>=,double,0)))","scriptNames":"[]","expressionName":"11195","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper [...]
-dipper.private.blink.rules&&&&express&&&&11191&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"__compare_value_10015\",\"11186\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11190&&&&{"varName":"___subtraction_10018","functionName":">=","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0.0"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11193&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11189\",\"11190\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11192&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11187\",\"11188\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11195&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11191\",\"11194\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11194&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11192\",\"11193\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11186&&&&{"varName":"valid_users_cnt","functionName":"<=","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"3.0"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11188&&&&{"varName":"unsuccess_hosts_cnt","functionName":">","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0.0"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11187&&&&{"varName":"___subtraction_10017","functionName":">=","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0.0"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11189&&&&{"varName":"success_hosts_cnt","functionName":">","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0.0"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10033&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10033","type":"script","value":"country='';\nprovince='';\ncity='';\nrule_type='2';\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10034&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10034","type":"script","value":"retainField(country,rule_type,province,city,succ_cns,ip,invalid_users_list,invalid_users_cnt,win [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10035&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10035","type":"script","value":"retainField(country,succ_cns,city,ip,invalid_users_list,invalid_users_cnt,winsize,type,uuid,log_ [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_rule_10009&&&&{"expressionStr":"type,==,'SSH'","scriptNames":"[]","expressionName":"11196","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_rule_10009","ruleStatus":"0","type":"rule","version":"1.0" [...]
-dipper.private.blink.rules&&&&express&&&&11196&&&&{"aesFlag":1,"varName":"type","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"yWD8YZYVXaFbtnsN24tuWg=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10036&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10036","type":"script","value":"uuid=uuid;\nali_uid='';\nclient_ip='';\nevent_type='异常登录';\nevent_name='批量异常登录成功(SSH）';\nlevel=' [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10037&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10037","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uu [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_rule_10010&&&&{"expressionStr":"type,==,'RDP'","scriptNames":"[]","expressionName":"11197","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_rule_10010","ruleStatus":"0","type":"rule","version":"1.0" [...]
-dipper.private.blink.rules&&&&express&&&&11197&&&&{"aesFlag":1,"varName":"type","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"XaH22EWkvLN7G+itcFtlvQ=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10038&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10038","type":"script","value":"uuid=uuid;\nali_uid='';\nclient_ip='';\nevent_type='异常登录';\nevent_name='批量异常登录成功(RDP）';\nlevel=' [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10039&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10039","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uu [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10040&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10040","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uu [...]
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_channel_10003&&&&{"offset":"0","isJsonData":"true","maxThread":"12","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.MetaqChannel","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_channel_10003","type":"channel","version":"1.0","timeout":"30000","activtyTimeOut":"3000","tags":"T_MSG_BRUTE_ [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10041&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10041","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uu [...]
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_channel_10004&&&&{"offset":"0","isJsonData":"true","maxThread":"12","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.MetaqChannel","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_channel_10004","type":"channel","version":"1.0","timeout":"30000","activtyTimeOut":"3000","tags":"T_MSG_BRUTE_ [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10042&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10042","type":"script","value":"__compare_value_10014=!((src_ip,regex,'^127\\.'))\n;___subtraction_10016=subtraction(fail_cns,in [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_rule_10011&&&&{"expressionStr":"((((__compare_value_10014&(succ_cns,<,double,5))&(succ_cns,>,double,0))&(___subtraction_10016,>=,double,0))&(invalid_users_cnt,>,double,3))&(valid_users_cnt,>,double,0)","scriptNames":"[]","expressionName":"11207","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","co [...]
-dipper.private.blink.rules&&&&express&&&&11207&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11206\",\"11202\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11206&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11205\",\"11201\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11199&&&&{"varName":"succ_cns","functionName":">","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0.0"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11198&&&&{"varName":"succ_cns","functionName":"<","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"5.0"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11201&&&&{"varName":"invalid_users_cnt","functionName":">","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"3.0"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11200&&&&{"varName":"___subtraction_10016","functionName":">=","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0.0"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11203&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"__compare_value_10014\",\"11198\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11202&&&&{"varName":"valid_users_cnt","functionName":">","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0.0"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11205&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11204\",\"11200\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11204&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11203\",\"11199\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10043&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10043","type":"script","value":"country='';\nprovince='';\ncity='';\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10044&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10044","type":"script","value":"retainField(country,province,city,succ_cns,invalid_users_list,ip,invalid_users_cnt,type,uuid,log [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_rule_10012&&&&{"expressionStr":"type,==,'SSH'","scriptNames":"[]","expressionName":"11208","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_rule_10012","ruleStatus":"0","type":"rule","version":"1.0" [...]
-dipper.private.blink.rules&&&&express&&&&11208&&&&{"aesFlag":1,"varName":"type","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"yWD8YZYVXaFbtnsN24tuWg=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10045&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10045","type":"script","value":"ali_uid='';\nclient_ip=ip;\nevent_type='异常登录';\nevent_name='ECS被多个无效用户暴力破解成功(SSH)';\nlevel='medi [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10046&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10046","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uu [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_rule_10013&&&&{"expressionStr":"type,==,'RDP'","scriptNames":"[]","expressionName":"11209","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_rule_10013","ruleStatus":"0","type":"rule","version":"1.0" [...]
-dipper.private.blink.rules&&&&express&&&&11209&&&&{"aesFlag":1,"varName":"type","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"XaH22EWkvLN7G+itcFtlvQ=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10047&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10047","type":"script","value":"ali_uid='';\nclient_ip=ip;\nevent_type='异常登录';\nevent_name='ECS被多个无效用户暴力破解成功(RDP)';\nlevel='medi [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10048&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10048","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uu [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10049&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10049","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uu [...]
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_channel_10005&&&&{"offset":"0","isJsonData":"true","maxThread":"12","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.MetaqChannel","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_channel_10005","type":"channel","version":"1.0","timeout":"30000","activtyTimeOut":"3000","tags":"T_MSG_BRUTE_ [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10050&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_script_10050","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uu [...]
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_channel_10006&&&&{"offset":"0","isJsonData":"true","maxThread":"12","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.MetaqChannel","configureName":"blink_source_aegis_login_adl_sas_apsara_abnormal_host_login_crack_success_alert_v4_channel_10006","type":"channel","version":"1.0","timeout":"30000","activtyTimeOut":"3000","tags":"T_MSG_BRUTE_ [...]
-dipper.private.blink.rules&&&&pipline&&&&blink.source.aegis.login&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"entityName\\\\\\\":\\\\\\\"script\\\\\\\",\\\\\\\"scriptName\\\\\\\":\\\\\\\"blink.source.aegis.login_script_10001\\\\\\\",\\\\\\\"lable\\\\\\\":\\\\\\\"blink. [...]
-dipper.private.blink.rules&&&&channel&&&&blink.source.aegis.login;channel&&&&{"offset":"0","isJsonData":"true","maxThread":"12","concurrentCount":"12","className":"com.aliyun.yundun.dipper.channel.adapter.MetaqChannel","configureName":"blink.source.aegis.login;channel","type":"channel","version":"1.0","timeout":"30000","activtyTimeOut":"3000","tags":"blink.source.aegis.login.tags","syncTimeout":"5000","groupName":"blink.source.aegis.login.group","isBatchMessage":"true","isAutoFlush":"fal [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.login_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.login_script_10001","type":"script","value":"aliUid=JSON_VALUE (meta_conf, '$.aliUid');portStr=JSON_VALUE (data, '$.port');port=cast(portStr,'long');rm(portStr);success=JSON_VALUE (data, '$.success');src_ip=JSON_VALUE (data, '$.ip');countStr=JSON_VALUE (dat [...]
-dipper.private.blink.rules&&&&pipline&&&&blink_source_aegis_net_sas_linux_alert_netstat_netscan&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"nextStageLables\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"blink_source_aegis_net_sas_linux_alert_netstat_netscan_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"cancelAfterConfigurableRefreshListerner\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"ownerSq [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_netscan;isIPv4Public_10001&&&&{"fullClassName":"com.aliyun.sec.lyra.udf.isIPv4Public","initMethodName":"open","functionName":"isIPv4Public","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_netscan;isIPv4Public_10001","type":"script","version" [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_netscan;json_concat_10021&&&&{"fullClassName":"com.aliyun.sec.lyra.udf.ext.JsonConcat","initMethodName":"open","functionName":"json_concat","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_netscan;json_concat_10021","type":"script","version": [...]
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_net_sas_linux_alert_netstat_netscan&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_netscan","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeInOrder":"true","maxFetchLogGroupSize":"100", [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_netscan_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_netscan_script_10001","type":"script","value":"___ISIPV4PUBLIC_10001=ISIPV4PUBLIC(dst_ip);\n___ISIPV4PUBLIC_10002=ISIPV4PUBLIC(dst_ip);\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_net_sas_linux_alert_netstat_netscan_rule_10001&&&&{"expressionStr":"(((((dst_port,in,'\\'21\\',\\'23\\',\\'25\\',\\'69\\',\\'81\\',\\'82\\',\\'389\\',\\'445\\',\\'873\\',\\'1433\\',\\'1521\\',\\'3306\\',\\'5900\\',\\'6379\\',\\'7001\\',\\'7002\\',\\'8009\\',\\'8080\\',\\'8081\\',\\'8082\\',\\'8443\\',\\'9200\\',\\'9300\\',\\'11211\\',\\'27017\\',\\'27018\\',\\'3389\\',\\'5632\\',\\'8000\\',\\'8161\\',\\'9043\\',\\'50000\\',\\'50070 [...]
-dipper.private.blink.rules&&&&express&&&&11218&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11217\",\"11212\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11217&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11210\",\"11211\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11219&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11218\",\"11213\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11210&&&&{"aesFlag":1,"varName":"dst_port","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"LEsonS2IlprVQ/co0XiLkvM6bIrx6/czU4m5Fi4n7MTlh5hCk8ASJPNdlIj8HwXRKVtn5pBb1jN4JfPhMkDraqiXUvKvOqNz/3SdzP3FowLBokTAp/9MTPE3wHlBW3ICtH6kYj/Zi3+/lHF19HSC0iAIv6fcxNHGs+NAMDIa1Pk3GqZGWq9ioiOlyjvQgdiBeAf+hl35+rnWCYnms2nRt1wxqEe5ZntLm4/9ZjKU [...]
-dipper.private.blink.rules&&&&express&&&&11221&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11215\",\"11216\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11220&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11219\",\"11214\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11212&&&&{"aesFlag":1,"varName":"status","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"TT4sWlpjQkv0uhwFIIa9c6Okraw7awBNUyPQnW/8dfk="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11211&&&&{"varName":"___ISIPV4PUBLIC_10001","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Boolean\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"false"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11222&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11220\",\"11221\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11214&&&&{"aesFlag":1,"varName":"ppid","functionName":"<>","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"2vQJrhr8/lZ2HEw+NK0Glw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11213&&&&{"aesFlag":1,"varName":"dir","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"kB68YBCbbpAqvn37U8lnhw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11216&&&&{"varName":"___ISIPV4PUBLIC_10002","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Boolean\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"false"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11215&&&&{"aesFlag":1,"varName":"proc_name","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"ky70l/8ginyvbfSpPOtje+ztK5ySxoW09dHhcreNc0dzYPlBfq5D/8m1SoealWg2"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_netscan_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_netscan_script_10002","type":"script","value":"retainField(k8s_namespace,proc_path,pid,proc_start_time,dir,host_uuid,dst_ip,parent_start_time,src_ip,k8s_node_name,cmdline,container_image_id,k8s_clust [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_netscan_script_10003&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_netscan_script_10003","type":"script","value":"___unixtime_10021=unixtime(scan_time,'yyyy-MM-dd HH:mm:ss');\n___division_10017=division(___unixtime_10021,1200);\n___floor_10008=floor(___division_1001 [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_netscan_script_10004&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_netscan_script_10004","type":"script","value":"retainField(cmdline,parent_proc_file_name,proc_path,proto,dst_port,time_win,scan_time,proc_name,host_uuid,dst_ip,parent_start_time,ppid);","version":"1. [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_netscan_script_10005&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_netscan_script_10005","type":"script","value":"retainField(cmdline,parent_proc_file_name,proc_path,proto,dst_port,time_win,scan_time,proc_name,host_uuid,dst_ip,parent_start_time,ppid);","version":"1. [...]
-dipper.private.blink.rules&&&&window&&&&blink_source_aegis_net_sas_linux_alert_netstat_netscan_window_10001&&&&{"groupByFieldName":"cmdline;dst_port;host_uuid;parent_proc_file_name;parent_start_time;ppid;proc_name;proc_path;proto;time_win","waterMarkMinute":"0","sizeAdjust":"0","className":"com.aliyun.yundun.dipper.window.model.WindowMessageProcessor","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_netscan_window_10001","type":"window","version":"1.0","windowType":"hop"," [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_netscan_script_10006&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_netscan_script_10006","type":"script","value":"retainField(dst_ip_cnt,proc_path,host_uuid,parent_start_time,ppid,start_time,dst_ip_info,cmdline,parent_proc_file_name,proto,dst_port,time_win,proc_name [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_net_sas_linux_alert_netstat_netscan_rule_10002&&&&{"expressionStr":"(((proc_name,!in,'\\'\\',\\'N/A\\',\\'nginx\\',\\'vk-nginx\\',\\'consul\\',\\'prometheus\\'')&(dst_ip_cnt,>=,double,55))|((proc_name,regex,'^(zmap|nmap|masscan|zenmap|unicornscan|hscan)')&(dst_ip_cnt,>=,double,10)))&(dst_port,!in,'\\'8000\\',\\'8161\\',\\'1099\\'')","scriptNames":"[]","expressionName":"11231","varNames":"[]","className":"com.aliyun.filter.processor [...]
-dipper.private.blink.rules&&&&express&&&&11229&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11225\",\"11226\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11228&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11223\",\"11224\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11230&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11228\",\"11229\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11231&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11230\",\"11227\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11223&&&&{"aesFlag":1,"varName":"proc_name","functionName":"!in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"qa44hJNByCCVqRLHchbLYgEvFHjbnaBkUhRpUQtNi15yibinvLdKrOkKC0RE6DL8IVuhTBcdykeEiyku9Q+UEg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11225&&&&{"aesFlag":1,"varName":"proc_name","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"ky70l/8ginyvbfSpPOtje+ztK5ySxoW09dHhcreNc0dzYPlBfq5D/8m1SoealWg2"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11224&&&&{"varName":"dst_ip_cnt","functionName":">=","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"55.0"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11227&&&&{"aesFlag":1,"varName":"dst_port","functionName":"!in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"+RBe6MtM/sgjf7I7hEq3EpXM1U9cuGcuBxW7yuyaRQg="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11226&&&&{"varName":"dst_ip_cnt","functionName":">=","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"10.0"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_netscan_script_10007&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_netscan_script_10007","type":"script","value":"is_white=0;\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_netscan_script_10008&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_netscan_script_10008","type":"script","value":"retainField(is_white,dst_ip_cnt,proc_path,host_uuid,parent_start_time,ppid,start_time,dst_ip_info,cmdline,parent_proc_file_name,proto,dst_port,time_win, [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_net_sas_linux_alert_netstat_netscan_rule_10003&&&&{"expressionStr":"is_white,==,double,0","scriptNames":"[]","expressionName":"11232","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_netscan_rule_10003","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[]"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11232&&&&{"varName":"is_white","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0.0"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_netscan_script_10009&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_netscan_script_10009","type":"script","value":"___concat_10041=concat(' - ','proc_path - ',proc_path,'pfilaname - ',parent_proc_file_name);\nabk_raw=___concat_10041;rm('___concat_10041');\n","version [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_netscan_script_10010&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_netscan_script_10010","type":"script","value":"retainField(dst_ip_cnt,proc_path,host_uuid,parent_start_time,ppid,start_time,dst_ip_info,cmdline,parent_proc_file_name,proto,dst_port,time_win,abk_raw,p [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_netscan_script_10011&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_netscan_script_10011","type":"script","value":"retainField(dst_ip_cnt,proc_path,host_uuid,parent_start_time,ppid,start_time,dst_ip_info,cmdline,parent_proc_file_name,proto,dst_port,time_win,abk_raw,p [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_netscan_script_10012&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_netscan_script_10012","type":"script","value":"uuid=host_uuid;\nali_uid='';\nclient_ip='';\nevent_type='异常网络连接';\nevent_name='内网扫描';\nlevel='high';\n___cast_10952=cast(cmdline,'string');\n___cast_109 [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_netscan_script_10013&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_netscan_script_10013","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_net_sas_linux_alert_netstat_netscan_channel_10001&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_netscan_channel_10001","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeInOrder":"true"," [...]
-dipper.private.blink.rules&&&&pipline&&&&blink_source_aegis_net_sas_linux_alert_netstat_lateral_movement&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.FilterChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"nextStageLables\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"blink_source_aegis_net_sas_linux_alert_netstat_lateral_movement_script_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"cancelAfterConfigurableRefreshListerner\\\\\\\":\\\\\\\"false\\\\ [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_lateral_movement;isIPv4Public_10002&&&&{"fullClassName":"com.aliyun.sec.lyra.udf.isIPv4Public","initMethodName":"open","functionName":"isIPv4Public","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_lateral_movement;isIPv4Public_10002","type": [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_lateral_movement;json_concat_10022&&&&{"fullClassName":"com.aliyun.sec.lyra.udf.ext.JsonConcat","initMethodName":"open","functionName":"json_concat","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_lateral_movement;json_concat_10022","type":" [...]
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_net_sas_linux_alert_netstat_lateral_movement&&&&{"offset":"0","isJsonData":"true","maxThread":"12","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.MetaqChannel","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_lateral_movement","type":"channel","version":"1.0","timeout":"30000","activtyTimeOut":"3000","syncTimeout":"5000","groupName":"blink_source_aegis_net_sas_linux_alert_netstat_later [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_net_sas_linux_alert_netstat_lateral_movement_rule_10001&&&&{"expressionStr":"(src_port,in,'\\'21\\',\\'23\\',\\'25\\',\\'69\\',\\'80\\',\\'389\\',\\'443\\',\\'873\\',\\'1433\\',\\'1434\\',\\'1521\\',\\'2375\\',\\'3306\\',\\'5900\\',\\'6379\\',\\'7001\\',\\'7002\\',\\'8080\\',\\'8443\\',\\'9200\\',\\'9300\\',\\'11211\\',\\'27017\\',\\'27018\\',\\'28017\\',\\'50030\\',\\'50070\\',\\'3389\\',\\'8081\\',\\'8080\\'')|(dst_port,in,'\\'21 [...]
-dipper.private.blink.rules&&&&express&&&&11234&&&&{"aesFlag":1,"varName":"dst_port","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"LEsonS2IlprVQ/co0XiLklo9FwYvW++krORC3ncjuYq4O6wKZ9PygiTvge5eK/8sI1wvrmWj3Bayf0UPN6pk/NlCtuo8AEsa4bZgAvbYaSidB7DDSwMH4/6o4HsZSkk30buO6A399CtpcMmAMke6RNyBi9Dgl1Uw+Y22Z+f1lOoccV7FmCuOMaeBVAGfaun8ChWK960f4hXvzYhYQ4zNIF3yzGUKYWktR7XZqKfY [...]
-dipper.private.blink.rules&&&&express&&&&11233&&&&{"aesFlag":1,"varName":"src_port","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"LEsonS2IlprVQ/co0XiLklo9FwYvW++krORC3ncjuYq4O6wKZ9PygiTvge5eK/8sI1wvrmWj3Bayf0UPN6pk/NlCtuo8AEsa4bZgAvbYaSidB7DDSwMH4/6o4HsZSkk30buO6A399CtpcMmAMke6RNyBi9Dgl1Uw+Y22Z+f1lOoccV7FmCuOMaeBVAGfaun8ChWK960f4hXvzYhYQ4zNIF3yzGUKYWktR7XZqKfY [...]
-dipper.private.blink.rules&&&&express&&&&11235&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11233\",\"11234\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_lateral_movement_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_lateral_movement_script_10001","type":"script","value":"retainField(k8s_namespace,proc_path,pid,proc_start_time,dir,host_uuid,dst_ip,parent_start_time,src_ip,k8s_node_name,cmdline,container_ [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_lateral_movement_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_lateral_movement_script_10002","type":"script","value":"___lower_proc_name_10010=lower(proc_name);\n___lower_cmdline_10099=lower(cmdline);\n___ISIPV4PUBLIC_10003=ISIPV4PUBLIC(dst_ip);\n","ve [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_net_sas_linux_alert_netstat_lateral_movement_rule_10002&&&&{"expressionStr":"((___lower_proc_name_10010,in,'\\'lcx\\',\\'ew\\',\\'ew_for_linux64\\',\\'ew_for_linux32\\',\\'htran\\',\\'rcsocks\\',\\'frpc\\'')|(___lower_cmdline_10099,regex,'ew_for_linux32|ew_for_linux64|(^|\\W+)htran|regeorgsocksproxy|(^|\\s+|/)rssocks\\s+|(^|\\s+|/)ssocksd\\s+|(^|\\s+|/)rcsocks\\s+|lcx_slave|lcx_tran|lcx_listen|((^|\\s+|/)frpc.*-s)|((^|\\s+|/)frpc.* [...]
-dipper.private.blink.rules&&&&express&&&&11239&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11236\",\"11237\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11240&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11239\",\"11238\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11236&&&&{"aesFlag":1,"varName":"___lower_proc_name_10010","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"yULyv2GPMaUFOae6UIbhCUMLUHrCa3Jotp8GNoo9qfbC2YF68YUz+LwjrBfi5Iw2SApZ0+OF1Xq9URrWUSWYvGeADOnB8V23goFx8ipWp6E="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11238&&&&{"varName":"___ISIPV4PUBLIC_10003","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Boolean\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"false"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11237&&&&{"aesFlag":1,"varName":"___lower_cmdline_10099","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"W0+STHJIqhBCXrNjId3ZBPV5qIQblgW+UlQ2WJDEXkqOiVG8RaI2603N5Q/5Gocb+24bg5f3n+auKo3Q5rQvpQLr7zxmiyzWTTzIwpKGn2M4GLlu63fJYXcGWRwsmOQilpzt7mOPVIMJP8r274FBXnNM+WwBWNvIFuUFOKv0Y3wz0e5krmrQPm1lyXOpW7ZE8qch7eeSW1q5i2sxlQJTKY7 [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_lateral_movement_script_10003&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_lateral_movement_script_10003","type":"script","value":"retainField(k8s_namespace,proc_path,pid,proc_start_time,dir,host_uuid,dst_ip,parent_start_time,src_ip,k8s_node_name,cmdline,container_ [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_net_sas_linux_alert_netstat_lateral_movement_rule_10003&&&&{"expressionStr":"ppid,<>,double,1","scriptNames":"[]","expressionName":"11241","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_lateral_movement_rule_10003","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":" [...]
-dipper.private.blink.rules&&&&express&&&&11241&&&&{"varName":"ppid","functionName":"<>","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"1.0"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_lateral_movement_script_10004&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_lateral_movement_script_10004","type":"script","value":"retainField(k8s_namespace,proc_path,pid,proc_start_time,dir,host_uuid,dst_ip,parent_start_time,src_ip,k8s_node_name,cmdline,container_ [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_lateral_movement_script_10005&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_lateral_movement_script_10005","type":"script","value":"retainField(k8s_namespace,proc_path,pid,proc_start_time,dir,host_uuid,dst_ip,parent_start_time,src_ip,k8s_node_name,cmdline,container_ [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_lateral_movement_script_10006&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_lateral_movement_script_10006","type":"script","value":"___REGEXP_REPLACE_10049=REGEXP_REPLACE(cmdline,'([^\\\\\\:\\>\\-\\&\\@\\=\\%\\s~~~~~\\/\\.\\(\\)\\[\\]]{1})','A');\n___concat_10042=co [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_lateral_movement_script_10007&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_lateral_movement_script_10007","type":"script","value":"retainField(k8s_namespace,proc_path,pid,proc_start_time,dir,host_uuid,dst_ip,parent_start_time,src_ip,k8s_node_name,cmdline,container_ [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_lateral_movement_script_10008&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_lateral_movement_script_10008","type":"script","value":"retainField(k8s_namespace,proc_path,pid,proc_start_time,dir,host_uuid,dst_ip,parent_start_time,src_ip,k8s_node_name,cmdline,abk_raw,co [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_lateral_movement_script_10009&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_lateral_movement_script_10009","type":"script","value":"uuid=host_uuid;\nali_uid='';\nclient_ip='';\nevent_type='异常网络连接';\nevent_name='疑似内网横向移动';\nlevel='high';\n___cast_10967=cast(cmdline,' [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_lateral_movement_script_10010&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_lateral_movement_script_10010","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField":"[ [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_lateral_movement_script_10011&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_lateral_movement_script_10011","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField":"[ [...]
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_net_sas_linux_alert_netstat_lateral_movement_channel_10001&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_lateral_movement_channel_10001","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consum [...]
-dipper.private.blink.rules&&&&pipline&&&&blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"nextStageLables\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"cancelAfterConfigurableRefreshListerner\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"ownerSqlN [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3;json_concat_10023&&&&{"fullClassName":"com.aliyun.sec.lyra.udf.ext.JsonConcat","initMethodName":"open","functionName":"json_concat","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3;json_concat_10023","type":"script","version":"1 [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3;b64_auto_10007&&&&{"fullClassName":"com.lyra.xs.udf.ext.b64_auto","initMethodName":"open","functionName":"b64_auto","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3;b64_auto_10007","type":"script","version":"1.0","closeMethodNam [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3;sas_black_rule_v2_10003&&&&{"fullClassName":"com.lyra.xs.udf.ext.sas_black_rule_v2","initMethodName":"open","functionName":"sas_black_rule_v2","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3;sas_black_rule_v2_10003","type":"scr [...]
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeInOrder":"true","maxFetchLogGroupSize":"100","s [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10001","type":"script","value":"___lower_proc_path_10001=lower(proc_path);\n___lower_parent_proc_file_name_10001=lower(parent_proc_file_name);\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_rule_10001&&&&{"expressionStr":"(dir,==,'out')&((((___lower_proc_path_10001,like,'%.exe')|(___lower_parent_proc_file_name_10001,like,'%.exe'))|(proc_path,like,'_:/%'))|(parent_proc_file_name,like,'_:/%'))","scriptNames":"[]","expressionName":"11250","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_sourc [...]
-dipper.private.blink.rules&&&&express&&&&11250&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11242\",\"11249\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11243&&&&{"aesFlag":1,"varName":"___lower_proc_path_10001","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"+hpefcjI0jxX3V74t6E7Ow=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11242&&&&{"aesFlag":1,"varName":"dir","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"kB68YBCbbpAqvn37U8lnhw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11245&&&&{"aesFlag":1,"varName":"proc_path","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"iEGyh/evZuz6gUfAf8QayQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11244&&&&{"aesFlag":1,"varName":"___lower_parent_proc_file_name_10001","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"+hpefcjI0jxX3V74t6E7Ow=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11247&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11243\",\"11244\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11246&&&&{"aesFlag":1,"varName":"parent_proc_file_name","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"iEGyh/evZuz6gUfAf8QayQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11249&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11248\",\"11246\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11248&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11247\",\"11245\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10002","type":"script","value":"logtime=scan_time;\n___unixtime_10022=unixtime(scan_time);\nunix_time=___unixtime_10022;rm('___unixtime_10022');\nuuid=host_uuid;\ncmd=cmdline;\npexe=proc [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10003&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10003","type":"script","value":"retainField(unix_time,pid,ppexe,dir,uuid,pproc_name,dst_ip,ppid,src_ip,src_port,pexe,proto,dst_port,cmd,proc_name,logtime,pcmd,status);","version":"1.0"," [...]
-dipper.private.blink.rules&&&&window&&&&blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_window_10001&&&&{"groupByFieldName":"uuid;dst_ip;dst_port;cmd;pexe;proc_name;ppexe;status;proto;___cast_10991","waterMarkMinute":"0","className":"com.aliyun.yundun.dipper.window.model.OverWindow","configureName":"blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_window_10001","type":"window","version":"1.0","fireDelaySecond":"30","timeout":"30000","activtyTimeOut":"3000","sizeInterval":"60", [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10004&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10004","type":"script","value":"___multiplication_10024=multiplication(60,10);\n___division_10018=division(unix_time,___multiplication_10024);\n___cast_10991=cast(___division_10018,'long [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10005&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10005","type":"script","value":"retainField(unix_time,pid,ppexe,dir,uuid,pproc_name,dst_ip,ppid,src_ip,src_port,pexe,proto,dst_port,cmd,proc_name,logtime,pcmd,status,r_dasudu19ud12y1827) [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_rule_10002&&&&{"expressionStr":"r_dasudu19ud12y1827,==,double,1","scriptNames":"[]","expressionName":"11251","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_rule_10002","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[]"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11251&&&&{"varName":"r_dasudu19ud12y1827","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"1.0"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10006&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10006","type":"script","value":"___regexp_10596=regex(dst_ip,'^((127\\.)|(10\\.)|(172\\.1[6-9]\\.)|(172\\.2[0-9]\\.)|(172\\.3[0-1]\\.)|(192\\.168\\.))');\nif(___regexp_10596){___if_dwd_y [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10007&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10007","type":"script","value":"retainField(unix_time,pid,ppexe,dir,uuid,pproc_name,dst_ip,ppid,src_ip,src_port,pexe,proto,dst_port,cmd,proc_name,logtime,pcmd,status,r_dasudu19ud12y1827, [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10008&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10008","type":"script","value":"___!_10072=!(((dir,==,'out')&(dst_ip,like,'100.100.%')));\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_rule_10003&&&&{"expressionStr":"(___!_10072)","scriptNames":"[]","expressionName":"11253","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_rule_10003","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[]"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11252&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"___!_10072\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11253&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11252\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10009&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10009","type":"script","value":"___REGEXP_EXTRACT_10021=REGEXP_EXTRACT(cmd,'([a-zA-Z0-9\\/+=]{50,})',1);\ncmd_b64_raw=___REGEXP_EXTRACT_10021;rm('___REGEXP_EXTRACT_10021');\n","version": [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10010&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10010","type":"script","value":"retainField(cmd_b64_raw,unix_time,pid,is_dstip_private,ppexe,dir,uuid,pproc_name,dst_ip,ppid,src_ip,src_port,pexe,proto,dst_port,cmd,proc_name,logtime,pcm [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10011&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10011","type":"script","value":"___lower_proc_name_10011=lower(proc_name);\nclean_proc=___lower_proc_name_10011;rm('___lower_proc_name_10011');\n___lower_pproc_name_10013=lower(pproc_nam [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10012&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10012","type":"script","value":"retainField(clean_proc,b64_decode_raw,clean_pexe,clean_pproc,clean_cmd,unix_time,pid,is_dstip_private,ppexe,dir,uuid,pproc_name,dst_ip,ppid,src_ip,src_por [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10013&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10013","type":"script","value":"___lower_b64_decode_raw_10003=lower(b64_decode_raw);\nb64_decode=___lower_b64_decode_raw_10003;rm('___lower_b64_decode_raw_10003');\n_tmp_qweyhkuyiyiuy='@ [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10014&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10014","type":"script","value":"retainField(b64_decode,is_real_out,_tmp_qweyhkuyiyiuy,clean_cmd,unix_time,pid,ppexe,dir,uuid,dst_ip,src_ip,pexe,cmd_b64_raw,b64_decode_raw,clean_pexe,is_d [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10015&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10015","type":"script","value":"___cast_10992=cast(cmd,'string');\n___cast_10993=cast(pcmd,'string');\n___cast_10994=cast(clean_proc,'string');\n___cast_10995=cast(clean_pproc,'string'); [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10016&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10016","type":"script","value":"retainField(clean_cmd,unix_time,pid,ppexe,dir,is_success,uuid,dst_ip,src_ip,is_real_out,pexe,cmd_b64_raw,b64_decode_raw,clean_pexe,_tmp_qweyhkuyiyiuy,is_d [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10017&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10017","type":"script","value":"___STRING_SPLIT_10003=STRING_SPLIT(threat_type_all,_tmp_qweyhkuyiyiuy);T.v=udtf.0;","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10018&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10018","type":"script","value":"___trim_10006=trim(T.v);\nthreat_type=___trim_10006;rm('___trim_10006');\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10019&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10019","type":"script","value":"retainField(threat_type,clean_cmd,unix_time,pid,ppexe,dir,is_success,uuid,dst_ip,src_ip,is_real_out,pexe,cmd_b64_raw,threat_type_all,b64_decode_raw,clean_ [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10020&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10020","type":"script","value":"___!null_10046=!null(threat_type);\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_rule_10004&&&&{"expressionStr":"___!null_10046&(threat_type,regex,'\\S+')","scriptNames":"[]","expressionName":"11255","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_rule_10004","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNam [...]
-dipper.private.blink.rules&&&&express&&&&11254&&&&{"aesFlag":1,"varName":"threat_type","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"BhPVyKanBl/NbolFClSKRQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11255&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"___!null_10046\",\"11254\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10021&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10021","type":"script","value":"retainField(clean_cmd,unix_time,pid,ppexe,dir,is_success,uuid,dst_ip,src_ip,is_real_out,pexe,cmd_b64_raw,threat_type,threat_type_all,b64_decode_raw,clean_ [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10022&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10022","type":"script","value":"___in_10107=contain(threat_type,'sas_black_rule_online','powershell_danger_port','installutil联网','msbuild联网','sensitive_path_N_danger_port')\n;___compare_ [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10023&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10023","type":"script","value":"retainField(event_name,clean_cmd,unix_time,pid,ppexe,dir,is_success,uuid,dst_ip,src_ip,is_real_out,pexe,cmd_b64_raw,threat_type,threat_type_all,b64_decode [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_rule_10005&&&&{"expressionStr":"event_name,<>,'ob'","scriptNames":"[]","expressionName":"11256","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_rule_10005","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[]"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11256&&&&{"aesFlag":1,"varName":"event_name","functionName":"<>","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"EN2hVlftSLdhuJOTt7HCwg=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10024&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10024","type":"script","value":"ali_uid='';\nclient_ip='';\nevent_type='异常网络连接';\nlevel='high';\n___cast_11004=cast(logtime,'string');\n___cast_11005=cast(uuid,'string');\n___cast_11006= [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10025&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_script_10025","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_channel_10001&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink_source_aegis_net_adl_sas_apsara_win_net_rule_v3_channel_10001","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeInOrder":"true","ma [...]
-dipper.private.blink.rules&&&&pipline&&&&blink_source_aegis_net_sas_linux_alert_netstat_black_feature&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"nextStageLables\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"blink_source_aegis_net_sas_linux_alert_netstat_black_feature_script_10002\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"cancelAfterConfigurableRefreshListerner\\\\\\\":\\\\\\\"false\\\\\\\",\ [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_black_feature;sas_black_rule_v2_10004&&&&{"fullClassName":"com.lyra.xs.udf.ext.sas_black_rule_v2","initMethodName":"open","functionName":"sas_black_rule_v2","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_black_feature;sas_black_rule_v2_1000 [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_black_feature;json_concat_10024&&&&{"fullClassName":"com.aliyun.sec.lyra.udf.ext.JsonConcat","initMethodName":"open","functionName":"json_concat","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_black_feature;json_concat_10024","type":"script [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_black_feature;b64_auto_10008&&&&{"fullClassName":"com.lyra.xs.udf.ext.b64_auto","initMethodName":"open","functionName":"b64_auto","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_black_feature;b64_auto_10008","type":"script","version":"1.0"," [...]
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_net_sas_linux_alert_netstat_black_feature&&&&{"offset":"0","isJsonData":"true","maxThread":"12","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.MetaqChannel","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_black_feature","type":"channel","version":"1.0","timeout":"30000","activtyTimeOut":"3000","syncTimeout":"5000","groupName":"blink_source_aegis_net_sas_linux_alert_netstat_black_featu [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_black_feature_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_black_feature_script_10001","type":"script","value":"___lower_proc_path_10002=lower(proc_path);\nstd_proc_path=___lower_proc_path_10002;rm('___lower_proc_path_10002');\n___lower_proc_name_10012 [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_black_feature_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_black_feature_script_10002","type":"script","value":"retainField(proc_path,std_proc_path,pid,dir,proc_start_time,host_uuid,dst_ip,parent_start_time,src_ip,cmdline,sas_black_rule_ob_result,std_c [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_black_feature_script_10003&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_black_feature_script_10003","type":"script","value":"retainField(proc_path,std_proc_path,pid,dir,proc_start_time,host_uuid,dst_ip,parent_start_time,src_ip,cmdline,sas_black_rule_ob_result,std_c [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_black_feature_script_10004&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_black_feature_script_10004","type":"script","value":"___compare_10316=great(sas_black_rule_online_result,0);\n___lower_std_proc_name_10001=lower(std_proc_name);\n___compare_10317=equals(___lowe [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_black_feature_script_10005&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_black_feature_script_10005","type":"script","value":"retainField(hit_result,proc_path,std_proc_path,pid,dir,proc_start_time,host_uuid,dst_ip,parent_start_time,src_ip,cmdline,sas_black_rule_ob_r [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_black_feature_script_10006&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_black_feature_script_10006","type":"script","value":"___in_10111=contain(hit_result,'黑特征','可疑端口监听','联网shell','联网系统命令','端口转发3','可疑网络外连')\n;if(___in_10111){___case_10165=0;}else{___case_10165=1;} [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_black_feature_script_10007&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_black_feature_script_10007","type":"script","value":"retainField(is_white,proc_path,std_proc_path,pid,dir,proc_start_time,host_uuid,dst_ip,parent_start_time,src_ip,cmdline,sas_black_rule_ob_res [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_black_feature_script_10008&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_black_feature_script_10008","type":"script","value":"retainField(proc_path,pid,dir,proc_start_time,host_uuid,dst_ip,parent_start_time,src_ip,cmdline,abk_raw,container_image_id,hit_result,logTim [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_net_sas_linux_alert_netstat_black_feature_rule_10001&&&&{"expressionStr":"is_white,==,double,0","scriptNames":"[]","expressionName":"11257","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_black_feature_rule_10001","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[] [...]
-dipper.private.blink.rules&&&&express&&&&11257&&&&{"varName":"is_white","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0.0"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_black_feature_script_10009&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_black_feature_script_10009","type":"script","value":"retainField(container_image_id,hit_result,proc_path,pid,dir,proc_start_time,host_uuid,dst_ip,parent_start_time,logTime,ppid,src_ip,src_port, [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_black_feature_script_10010&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_black_feature_script_10010","type":"script","value":"uuid=host_uuid;\nali_uid='';\nclient_ip='';\nevent_type='异常网络连接';\n___compare_10321=equals(hit_result,'可疑端口监听');\nif(___compare_10321){___ca [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_black_feature_script_10011&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_black_feature_script_10011","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_black_feature_script_10012&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_black_feature_script_10012","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_net_sas_linux_alert_netstat_black_feature_channel_10001&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_black_feature_channel_10001","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeInOrd [...]
-dipper.private.blink.rules&&&&pipline&&&&blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"nextStageLables\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"cancelAfterConfigurableRefreshListerner\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"ownerSq [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2;json_concat_10025&&&&{"fullClassName":"com.aliyun.sec.lyra.udf.ext.JsonConcat","initMethodName":"open","functionName":"json_concat","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2;json_concat_10025","type":"script","version": [...]
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeInOrder":"true","maxFetchLogGroupSize":"100", [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2_script_10001","type":"script","value":"___lower_proc_path_10003=lower(proc_path);\n___lower_parent_proc_file_name_10002=lower(parent_proc_file_name);\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2_rule_10001&&&&{"expressionStr":"(dir,==,'out')&((((___lower_proc_path_10003,like,'%.exe')|(___lower_parent_proc_file_name_10002,like,'%.exe'))|(proc_path,like,'_:/%'))|(parent_proc_file_name,like,'_:/%'))","scriptNames":"[]","expressionName":"11266","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_sour [...]
-dipper.private.blink.rules&&&&express&&&&11261&&&&{"aesFlag":1,"varName":"proc_path","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"iEGyh/evZuz6gUfAf8QayQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11260&&&&{"aesFlag":1,"varName":"___lower_parent_proc_file_name_10002","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"+hpefcjI0jxX3V74t6E7Ow=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11263&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11259\",\"11260\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11262&&&&{"aesFlag":1,"varName":"parent_proc_file_name","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"iEGyh/evZuz6gUfAf8QayQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11265&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11264\",\"11262\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11264&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11263\",\"11261\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11266&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11258\",\"11265\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11258&&&&{"aesFlag":1,"varName":"dir","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"kB68YBCbbpAqvn37U8lnhw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11259&&&&{"aesFlag":1,"varName":"___lower_proc_path_10003","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"+hpefcjI0jxX3V74t6E7Ow=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2_script_10002","type":"script","value":"logtime=scan_time;\n___unixtime_10023=unixtime(scan_time);\nunix_time=___unixtime_10023;rm('___unixtime_10023');\nuuid=host_uuid;\ncmd=cmdline;\npexe=pr [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2_script_10003&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2_script_10003","type":"script","value":"retainField(unix_time,pid,ppexe,dir,uuid,pproc_name,dst_ip,ppid,src_ip,src_port,pexe,proto,dst_port,cmd,proc_name,logtime,pcmd,status);","version":"1.0" [...]
-dipper.private.blink.rules&&&&window&&&&blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2_window_10001&&&&{"groupByFieldName":"uuid;dst_ip;dst_port;cmd;pexe;proc_name;ppexe;status;proto;___cast_11053","waterMarkMinute":"0","className":"com.aliyun.yundun.dipper.window.model.OverWindow","configureName":"blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2_window_10001","type":"window","version":"1.0","fireDelaySecond":"30","timeout":"30000","activtyTimeOut":"3000","sizeInterval":"60 [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2_script_10004&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2_script_10004","type":"script","value":"___multiplication_10025=multiplication(60,10);\n___division_10019=division(unix_time,___multiplication_10025);\n___cast_11053=cast(___division_10019,'lo [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2_script_10005&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2_script_10005","type":"script","value":"retainField(unix_time,pid,ppexe,dir,uuid,pproc_name,dst_ip,ppid,src_ip,src_port,pexe,proto,dst_port,cmd,proc_name,logtime,pcmd,status,r_dasudu19ud12y182 [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2_rule_10002&&&&{"expressionStr":"r_dasudu19ud12y1827,==,double,1","scriptNames":"[]","expressionName":"11267","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2_rule_10002","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[]" [...]
-dipper.private.blink.rules&&&&express&&&&11267&&&&{"varName":"r_dasudu19ud12y1827","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"1.0"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2_script_10006&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2_script_10006","type":"script","value":"___regexp_10611=regex(dst_ip,'^((127\\.)|(10\\.)|(172\\.1[6-9]\\.)|(172\\.2[0-9]\\.)|(172\\.3[0-1]\\.)|(192\\.168\\.))');\nif(___regexp_10611){___if_dwd [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2_script_10007&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2_script_10007","type":"script","value":"retainField(unix_time,pid,ppexe,dir,uuid,pproc_name,dst_ip,ppid,src_ip,src_port,pexe,proto,dst_port,cmd,proc_name,logtime,pcmd,status,r_dasudu19ud12y182 [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2_rule_10003&&&&{"expressionStr":"((dst_port,==,'445')&(status,==,'TCP_STATE_ESTABLISHED'))&(pexe,!in,'\\'N/A\\',\\'\\',\\'null\\'')","scriptNames":"[]","expressionName":"11272","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2_rule_10003","ruleStatus":"0", [...]
-dipper.private.blink.rules&&&&express&&&&11270&&&&{"aesFlag":1,"varName":"pexe","functionName":"!in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"pvra235d2zA7ATJsvgXMoQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11272&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11271\",\"11270\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11271&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11268\",\"11269\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11269&&&&{"aesFlag":1,"varName":"status","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"mBo8DRi0jqLEUMAk3fPIkEAFxUrWkYtflRV52aAcmFs="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11268&&&&{"aesFlag":1,"varName":"dst_port","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"H2mt3iCJRyVxRN/JBkPzZA=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2_script_10008&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2_script_10008","type":"script","value":"os='win';\n___unixtime_10024=unixtime(logtime);\n___multiplication_10026=multiplication(60,60);\n___multiplication_10027=multiplication(___multiplicatio [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2_script_10009&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2_script_10009","type":"script","value":"retainField(detail_len_max,os,time_part,unix_time,pid,is_dstip_private,ppexe,dir,uuid,pproc_name,dst_ip,ppid,src_ip,src_port,pexe,proto,dst_port,cmd,pro [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2_script_10010&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2_script_10010","type":"script","value":"___!null_10047=!null(time_part);\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2_rule_10004&&&&{"expressionStr":"(___!null_10047)","scriptNames":"[]","expressionName":"11274","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2_rule_10004","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[]"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11274&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11273\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11273&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"___!null_10047\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&window&&&&blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2_window_10002&&&&{"groupByFieldName":"time_part;uuid;pexe;dst_port;proto;detail_len_max;cmd;os","waterMarkMinute":"0","sizeAdjust":"0","className":"com.aliyun.yundun.dipper.window.model.WindowMessageProcessor","configureName":"blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2_window_10002","type":"window","version":"1.0","windowType":"hop","fireDelaySecond":"30","timeout":"30000","activtyTi [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2_script_10011&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2_script_10011","type":"script","value":"retainField(dst_ip_cnt,pproc_name_dtail,os,time_part,cmd_max,pid,uuid,min_time,pproc_name_max,cmd_detail,detail_len_max,pcmd_max,pexe,proto,dst_port,max [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2_rule_10005&&&&{"expressionStr":"dst_ip_cnt,>,double,10","scriptNames":"[]","expressionName":"11275","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2_rule_10005","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[]"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11275&&&&{"varName":"dst_ip_cnt","functionName":">","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"10.0"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2_script_10012&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2_script_10012","type":"script","value":"retainField(dst_ip_cnt,pproc_name_dtail,os,time_part,cmd_max,pid,uuid,min_time,pproc_name_max,cmd_detail,detail_len_max,pcmd_max,pexe,proto,dst_port,max [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2_rule_10006&&&&{"expressionStr":"1,==,double,1","scriptNames":"[]","expressionName":"11276","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2_rule_10006","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[]"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11276&&&&{"varName":"1","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"1.0"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2_script_10013&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2_script_10013","type":"script","value":"ali_uid='';\nclient_ip='';\nevent_type='异常网络连接';\nevent_name='疑似敏感端口扫描行为';\nlevel='medium';\n___cast_11054=cast(dst_ip_cnt,'string');\n___JSON_CONCAT_10 [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2_script_10014&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2_script_10014","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2_channel_10001&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink_source_aegis_net_adl_sas_apsara_win_port_scan_v2_channel_10001","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeInOrder":"true"," [...]
-dipper.private.blink.rules&&&&pipline&&&&blink_source_aegis_net_sas_linux_alert_netstat_reverse_shell&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.FilterChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"nextStageLables\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"blink_source_aegis_net_sas_linux_alert_netstat_reverse_shell_script_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"cancelAfterConfigurableRefreshListerner\\\\\\\":\\\\\\\"false\\\\\\\",\ [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_reverse_shell;b64_auto_10009&&&&{"fullClassName":"com.lyra.xs.udf.ext.b64_auto","initMethodName":"open","functionName":"b64_auto","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_reverse_shell;b64_auto_10009","type":"script","version":"1.0"," [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_reverse_shell;json_concat_10026&&&&{"fullClassName":"com.aliyun.sec.lyra.udf.ext.JsonConcat","initMethodName":"open","functionName":"json_concat","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_reverse_shell;json_concat_10026","type":"script [...]
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_net_sas_linux_alert_netstat_reverse_shell&&&&{"offset":"0","isJsonData":"true","maxThread":"12","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.MetaqChannel","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_reverse_shell","type":"channel","version":"1.0","timeout":"30000","activtyTimeOut":"3000","syncTimeout":"5000","groupName":"blink_source_aegis_net_sas_linux_alert_netstat_reverse_she [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_net_sas_linux_alert_netstat_reverse_shell_rule_10001&&&&{"expressionStr":"(dir,==,'out')&(dst_ip,!in,'\\'127.0.0.1\\'')","scriptNames":"[]","expressionName":"11279","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_reverse_shell_rule_10001","ruleStatus":"0","type":"rule","version":"1.0","extendFiel [...]
-dipper.private.blink.rules&&&&express&&&&11278&&&&{"aesFlag":1,"varName":"dst_ip","functionName":"!in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"qlkoz6ZzoWKyXcuI2NiwqQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11277&&&&{"aesFlag":1,"varName":"dir","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"kB68YBCbbpAqvn37U8lnhw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11279&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11277\",\"11278\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_reverse_shell_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_reverse_shell_script_10001","type":"script","value":"___lower_cmdline_10101=lower(cmdline);\n___REGEXP_REPLACE_10054=REGEXP_REPLACE(___lower_cmdline_10101,'\\s+',' ');\nstd_cmdline=___REGEXP_RE [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_reverse_shell_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_reverse_shell_script_10002","type":"script","value":"retainField(k8s_namespace,proc_path,pid,dir,proc_start_time,host_uuid,dst_ip,parent_start_time,src_ip,k8s_node_name,cmdline,std_cmdline,cont [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_net_sas_linux_alert_netstat_reverse_shell_rule_10002&&&&{"expressionStr":"(((((((((((((((((((((((((((((((((((((((((std_cmdline,regex,'^(((/?([a-zA-Z0-9_\\.\\-]+/){1,20})bin/)|/bin/|/|-)?(bash|sh|dash|ash|tcsh|csh|ksh)(\\s+[\\-a-z0-9]{1,5}){1,5}\\s*$')&(std_cmdline,regex,'-[a-z0-9]{0,4}i[a-z0-9]{0,4}'))|(std_cmdline,regex,'^(((/?([a-zA-Z0-9_\\.\\-]+/){1,20})bin/)|/bin/|/|-)?zsh\\s*$'))|(std_cmdline,regex,'(mkfifo|mknod).*&&\\s*(nc|t [...]
-dipper.private.blink.rules&&&&express&&&&11340&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11339\",\"11299\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11306&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"wY2TXDp84Np+fxCQw153qSOBRfZ5HzQhU8et9MwgBsM="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11305&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"ECKiR+VSLN0UJKMNnLRBzhCRKUJ1lwglvQPeOT3o6HZ1w1lsZlOOAWGqvcy19vuPUD2qnyEV5dcJYaWPL7dnuw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11349&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11348\",\"11308\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11308&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"WOspaXLAGxMn68UjGzgPMtwakB96I2wWn8tkcxUQ+tg6f1w3UvEhq/9YTeG7aUxpjSRn8vaLVPWw9v4R3pjTNQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11307&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"fKKxPOBqvGDBL8gf1En+jXSeH09f2NiEAxOCTMHPPVyyDXOfZVpRhzCifKHBQ/GhlUIK5Tys3qhw+gGDA0mkYQEMqY9rL7nmXZaTDMGrD7qQ64ORsh0j3HbW5XUxe1k9D/c+oPPB18fbkH7gxRCn6OocdswhUhuOUEKnNaN6gbTB60UM9BJfIrxCXN2xYG0s"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11309&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"mPvs4d/P/GyxoYG1nO5HKdG8cMpJ6tz8ID7jPha5bnSqCeJrUCkGBjKl18tBIg+QCUV0N4n7ecI6VRZC40pLoQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11342&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11341\",\"11301\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11341&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11340\",\"11300\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11300&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"S6+5nja7S9/5FD7Y9XWiMXlNNYLiwb+PpsTeyu0wTwmqXeVGKT8cz371234XRRs64E++v99R0XlwvhGSPJDPQlK2L2TuXLKVsrlfDgDFB4A="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11344&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11343\",\"11303\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11343&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11342\",\"11302\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11302&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"EBOsKA8HJr0TeID5FpnEGv1DYfkDqtN7zTZ9BqruY3GKaxnBlsl84RDlXhtxLN/VhPF7pfBk1150eJDAaSvD3EZDddj34f3nf7XJxONVt8o="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11346&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11345\",\"11305\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11301&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"NfOPvvUA2C02nnvJmlFVPIb1AQXWnScVSFTS0j3O1NPUlMwe38KWRFHo43FlwB+bI7Ls06CYxVy8xW4FGawtoQfHg7fPcVMJm4aGtUsThf0="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11345&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11344\",\"11304\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11304&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"EBOsKA8HJr0TeID5FpnEGk8OfW5gge289PC+T42dZMOqMmQwUl25j4sV82bdZQzFHfZVbbt0lB/EgrO9fQ3ZX9e8fK2i5SlvomB+6TObk/n9X5HKfEjzSrkiFR/EMKbKWGZ8bf+ALH7Vo+WH4og1eA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11348&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11347\",\"11307\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11303&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"EBOsKA8HJr0TeID5FpnEGhsz6jEjv19XPg1ML+aMKIi3VPwmHsnAX/AjthoMDVrxmmW3OL4PYANEf/gEOw3q9983zvSxEaeEZnnJHSA0vOr5et9uXOwEWhVBISH7pcah3E3j8pR9+z33jDX5I8XAFw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11347&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11346\",\"11306\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11351&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11350\",\"11310\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11350&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11349\",\"11309\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11317&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"1BklyQ3tw4vQH5b93Ia/vyXZH0RHgX3Eo5MdHJWKDCCCSYJ+YJig9HvriuFz25P/"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11316&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"WK0CcvVjcHNvKSz5NChmScbwq2rFCpGi/+AmRnM2Paxya0atBLfMta5qWg/EoJtHCpYOs/+DHGxFEpKLF+Kf7ln03RFewWw/XvdQeK0Xfu/I6WKFpntjkR4o3YBJQ2TyAtFf/6XQ9tPOQDo0tcHGO0W7t5CXwtLpL+hyUmUm3ig="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11319&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"xex36LJ6hs3NrKtc0smZnWbxKiHzkiCmnj1Lo3E4heuCCew6PLmc/yoE92e5dJzXMlJUjVZMn9/07Qt4Zz3t5w=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11318&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"hxdJkcCWlOLR0Z08v9P8nBuhHVB3C0BGnBHaWtJlSoBwAGmOVvMzJCR/hBaDbCnB27OvIRbjU3+4qErWPKZ9Xw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11353&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11352\",\"11312\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11352&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11351\",\"11311\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11311&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"EBOsKA8HJr0TeID5FpnEGk8OfW5gge289PC+T42dZMOqMmQwUl25j4sV82bdZQzF5BNzcp9OvbGB2bGGWWlAjnJojhiRyLmDQTBY12xBIRwL20Am54VnTPrDyKy4zSinqyVdT2dMZc5cJlrtZzw/Kw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11355&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11354\",\"11314\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11310&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"HDGdCUToGqETm9cV+Jw1DsDelVY01kGxLRoXbmDebmq0ptZlxf4r1YBlKFA+DgIYCIa6biOy7/UnU5VuvhpfoVaHGznnJEXLCs/fkUI3MN8="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11354&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11353\",\"11313\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11313&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"5ZNqmWb9CISLIDGXbtTJT8ot0Iyi6x5blCdB0qg2sMj0SPL4iR5NDF01E6AM1t+miwP0iC0BMCP+QR3Kg5S4XonytuBcbOBENg3EFJHmd8d+TGh6darQq1vbsbldr6jf"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11357&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11356\",\"11316\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11312&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"CXjr80lckkCQoRmGUctHXan4nAjLiopKSgP/26U9t4yuK6Bl8q0n/biqPgoe9SohDIqwDc9EOxlTF2d6HT/ydqFlHAq4xThHqrmrmzQuNME="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11356&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11355\",\"11315\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11315&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"EBOsKA8HJr0TeID5FpnEGk8OfW5gge289PC+T42dZMOqMmQwUl25j4sV82bdZQzF/V0UaVObgu2k9W0KBlU62EFR8eogS2ychnLDjmX0BJ0CXuVxm4ojZoY6jPBZDWhdLrPr1BzFFETnI/8hkSG8tg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11359&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11358\",\"11318\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11314&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"EBOsKA8HJr0TeID5FpnEGk8OfW5gge289PC+T42dZMOqMmQwUl25j4sV82bdZQzF6CkFqOebmPfTz2Zuknt0h22Hi1MUdiZ3F7EuATO0nSCxfs3mpVLkkj58hHieJCXtY8lwfKFrmxr4NwaO8GDsBQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11358&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11357\",\"11317\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11281&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0iakI5h93i0MVdEnQzv3Oayi2zdxP/QmjjNUk94pe/4="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11280&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"tC+6BVeaz7RbgiE7GI5nvI+3SNAfCktVEXCRJ2PiAZV6l5KuC1EsplbweRXMpO7VdK7uu993qm1aedLTtbg2Yklh5ogU1JJ5eSOwZIQRjAH0ze0cIG1oR0rs591s2jBozZaouf+j34WVK1W8XJXFNg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11283&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"1aiRaiqvOFibkrM/JSSjCApQH5ugQ7isbooFxL2VAN0YPlgmfmJWBvqP+6KjyHbp"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11360&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11359\",\"11319\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11282&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"tC+6BVeaz7RbgiE7GI5nvI+3SNAfCktVEXCRJ2PiAZV6l5KuC1EsplbweRXMpO7VtVy9YAY4uy+OTGXyoHY0kw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11285&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"1aiRaiqvOFibkrM/JSSjCHTCJIuyFUBJGoH9ymibBi5A5Q3hgNsyXVAomyYLunsX"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11362&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11361\",\"11321\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11284&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0yWv4wJ+Lh91yiZrWoWwA0QiYld/6Ag9ohQZOM8G0t+P6eS4ias9aG9XikdKtcAEOkSjBa5XjgVVzcEIy2GZZYXIX+Z5+7mUXqjc86ueoec="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11361&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11360\",\"11320\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11328&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11327\",\"11287\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11327&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11326\",\"11286\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11329&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11328\",\"11288\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11287&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"BSkRZytZ0ZpioJ9N79IarJlw/nTZlZcMiMSw3n+tiN9ikWIZJHE+2GgRPnYFXjF1qdxwMTXVMfh3Tp8oBnZ0W8k3sw3Hknl2ZWcbOIrw1pM="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11320&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"o3EBIPr0c0ZcyGjjhLGtU5S156e8TkmQE/omx4zSWpxN0jUy3o/vPrIz91zv/pZhLJPRxNK0HZPVzjbnMZmnPA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11286&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"1aiRaiqvOFibkrM/JSSjCPA3TJv0PJP6/if3K454Y1EEMUr5yjXddokuM6olNHE5pR0Fqfylep2/OERDuyXvlM31wnZ6b4Hmcm0zj0lIrEo="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11289&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"5ZNqmWb9CISLIDGXbtTJT8ot0Iyi6x5blCdB0qg2sMhCP2GecDQAW7YUtHs+dTM1VPwtc0NsZmXMjUWNLpF33w=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11322&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11280\",\"11281\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11288&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"BSkRZytZ0ZpioJ9N79IarJlw/nTZlZcMiMSw3n+tiN96X3+ZENqLYALk8+HKyZ/i/J5EgzH1u8CL8gd9m1QvpA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11321&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"Vs2VKxNtKiA0PJ0bP4XcHxxELX2RNqfaCC0F087m48+6IRw2wSZX0qa3CawbGZEvXc+rXK7S2iURJDOlHDRagA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11324&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11323\",\"11283\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11323&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11322\",\"11282\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11326&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11325\",\"11285\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11325&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11324\",\"11284\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11290&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"CXjr80lckkCQoRmGUctHXTXa3+u+NVZjaylmJ9j7ib0Rdv5J+hQ67ZDtEOUd+kfPxk0qvydCncT1KE5GPfFsxFupUKwdlNFqoVJmivUDgGStHKdpvruObX2chS8+izpO"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11292&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"CXjr80lckkCQoRmGUctHXY+hOB5x7uiMnhahKoGO+zdIycaMOFIcTXCy0Bi4oVNBkHXG5D5wXFsEKCrtV78ShdB2SaR1bFCQZH6VbGTqsiNglLFr4kZOoVN4SGPG2U8V/1q/02UvJuj5j7Rb1izY7w=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11291&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"CXjr80lckkCQoRmGUctHXTXa3+u+NVZjaylmJ9j7ib2qNExrI38l7Ehsz/Kv4MAUjYehEdzihIhLI+5cK5A25w=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11294&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"CXjr80lckkCQoRmGUctHXXU/9pTYOzIMQuESgsr3nzYaqYuWOSNdXJqD1EvgCQuWNxQlD0vVOWYsWkQOOGAHBRcpkWMRklY/I0raa28heKMptdZvXnYfrRUvXknkWhvU"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11293&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"CXjr80lckkCQoRmGUctHXTXa3+u+NVZjaylmJ9j7ib2K5GJwmtRq/Tvq7K2jOBCeYPDvJjMIoR6YcH8wM7LlSUaTo5Vst5WKNLPDWgl13sOitlwIatuJFAI+2fY3/Ww2"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11296&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"CXjr80lckkCQoRmGUctHXXNahm3tYU2TR6eNXFlHK8v4p55wK/41TMoXz8QSOep+x5+NkUo7biKiqiMSGjVxlA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11295&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"CXjr80lckkCQoRmGUctHXa4Q2n6hyzgpSv5CAmEKFcRaRRAKCcTvHbv41weYBBIV"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11339&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11338\",\"11298\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11338&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11337\",\"11297\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11298&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"hlmYfaWn4nmcgbGmwxlgbJq7JDgGmWvS7+SBYB1yh4XoURRiiwa3zg3/8mVyu6GGeKoS/h8NwkXKhKbDmaDl1A=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11331&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11330\",\"11290\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11297&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"hlmYfaWn4nmcgbGmwxlgbGg4r39BvWbW/3EaUWfR5oZjqkLVJv7+9sEv6Jw+y4LyuWv3CkZcMElOXxHbYVvfzWonh1aT6UQB6tdGcTW5mXw="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11330&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11329\",\"11289\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11333&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11332\",\"11292\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11299&&&&{"aesFlag":1,"varName":"std_cmdline","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"/BewLdePbTzbt8jB8hW7H+6IOTk4ajiWVsnWbh050ndXaeG2dWTfd8x1KLSiMSsYqmJ7UpHpYBubevn+6vm77urEv0HWYboyNITcRrsLeyvBjTvwQ0ZwM5CDghQDbzGaQArqtnE0k+bJUbMo3EqsyjxbjyrMtRyImECDCq7jOR+53cnUpTFeahjL8HY6I63A"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11332&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11331\",\"11291\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11335&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11334\",\"11294\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11334&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11333\",\"11293\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11337&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11336\",\"11296\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11336&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11335\",\"11295\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_reverse_shell_script_10003&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_reverse_shell_script_10003","type":"script","value":"___compare_10322=equals(b64_cmdline,'');\n___null_10075=null(b64_cmdline);\n___B64_AUTO_10007=B64_AUTO(b64_cmdline);\nif((___compare_10322|_ [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_reverse_shell_script_10004&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_reverse_shell_script_10004","type":"script","value":"retainField(k8s_namespace,proc_path,pid,dir,proc_start_time,host_uuid,dst_ip,parent_start_time,src_ip,k8s_node_name,cmdline,std_cmdline,cont [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_reverse_shell_script_10005&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_reverse_shell_script_10005","type":"script","value":"__compare_value_10016=!((b64_cmdline,regex,'connect\\({1,3}(#######|~~~~~)127\\.0\\.0\\.1'))\n;__compare_value_10017=!((parent_proc_file_nam [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_net_sas_linux_alert_netstat_reverse_shell_rule_10003&&&&{"expressionStr":"((b64_cmdline,==,'')|__compare_value_10016)&__compare_value_10017","scriptNames":"[]","expressionName":"11365","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_reverse_shell_rule_10003","ruleStatus":"0","type":"rule","versio [...]
-dipper.private.blink.rules&&&&express&&&&11364&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11363\",\"__compare_value_10016\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11363&&&&{"aesFlag":1,"varName":"b64_cmdline","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"wc8fi0p7qwBHnY1x2c6Qsw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11365&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11364\",\"__compare_value_10017\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_reverse_shell_script_10006&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_reverse_shell_script_10006","type":"script","value":"retainField(k8s_namespace,proc_path,pid,dir,proc_start_time,host_uuid,dst_ip,parent_start_time,src_ip,k8s_node_name,cmdline,std_cmdline,cont [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_reverse_shell_script_10007&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_reverse_shell_script_10007","type":"script","value":"source='netstat_ob';\n___regexp_10612=regex(std_cmdline,'^(((/?([a-zA-Z0-9_\\.\\-]+/){1,20})bin/)|/bin/|/|-)?(bash|sh|dash|ash|tcsh|csh|ksh) [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_reverse_shell_script_10008&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_reverse_shell_script_10008","type":"script","value":"retainField(k8s_namespace,proc_path,pid,source,dir,proc_start_time,host_uuid,dst_ip,parent_start_time,src_ip,k8s_node_name,cmdline,std_cmdli [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_reverse_shell_script_10009&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_reverse_shell_script_10009","type":"script","value":"source='netstat';\n___lower_cmdline_10102=lower(cmdline);\n___REGEXP_REPLACE_10055=REGEXP_REPLACE(___lower_cmdline_10102,'\\s+',' ');\n___RE [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_reverse_shell_script_10010&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_reverse_shell_script_10010","type":"script","value":"retainField(is_white,k8s_namespace,proc_path,pid,source,dir,proc_start_time,host_uuid,dst_ip,parent_start_time,src_ip,k8s_node_name,cmdline, [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_net_sas_linux_alert_netstat_reverse_shell_rule_10004&&&&{"expressionStr":"is_white,==,double,0","scriptNames":"[]","expressionName":"11366","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_reverse_shell_rule_10004","ruleStatus":"0","type":"rule","version":"1.0","extendField":"[]","actionNames":"[] [...]
-dipper.private.blink.rules&&&&express&&&&11366&&&&{"varName":"is_white","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"0.0"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_reverse_shell_script_10011&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_reverse_shell_script_10011","type":"script","value":"uuid=host_uuid;\nali_uid='';\nclient_ip='';\nevent_type='异常网络连接';\nevent_name='反弹shell网络外连';\nlevel='high';\n___cast_11055=cast(cmdline,'str [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_reverse_shell_script_10012&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_reverse_shell_script_10012","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_sas_linux_alert_netstat_reverse_shell_script_10013&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_reverse_shell_script_10013","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_net_sas_linux_alert_netstat_reverse_shell_channel_10001&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink_source_aegis_net_sas_linux_alert_netstat_reverse_shell_channel_10001","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeInOrd [...]
-dipper.private.blink.rules&&&&pipline&&&&blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"nextStageLables\\\\\\\":\\\\\\\"[\\\\\\\\\\\\\\\"blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_rule_10001\\\\\\\\\\\\\\\"]\\\\\\\",\\\\\\\"cancelAfterConfigurableRefreshListerner\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\ [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert;json_concat_10027&&&&{"fullClassName":"com.aliyun.sec.lyra.udf.ext.JsonConcat","initMethodName":"open","functionName":"json_concat","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert;json_concat_10027","type":"script", [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert;salt_hash_10003&&&&{"fullClassName":"com.aliyun.sec.lyra.hsh.udf.ext.SaltHash","initMethodName":"open","functionName":"salt_hash","isURL":"false","methodName":"eval","className":"com.aliyun.yundun.dipper.blink.script.udf.BlinkUDFScript","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert;salt_hash_10003","type":"script","ver [...]
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert&&&&{"offset":"0","isJsonData":"true","maxThread":"12","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.MetaqChannel","configureName":"blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert","type":"channel","version":"1.0","timeout":"30000","activtyTimeOut":"3000","tags":"T_MSG_NETSTAT","syncTimeout":"5000","groupName":"blink_source_aegis_net_adl_sas_apsara [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10001","type":"script","value":"___len_10008=len(dst_ip);\n___lower_proc_path_10004=lower(proc_path);\n___lower_proc_path_10005=lower(proc_path);\n___lower_proc_path_10006=lo [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_rule_10001&&&&{"expressionStr":"(((((((((((((((((((((((((((((status,==,'TCP_STATE_ESTABLISHED')&(___len_10008,>,double,9))&(dst_ip,!like,'10.%'))&(dst_ip,!like,'192.168.%'))&(dst_ip,!like,'172.1%'))&(dst_ip,!like,'172.2%'))&(dst_ip,!like,'172.3%'))&(dst_ip,!like,'169.254.%'))&(cmdline,!like,'%/jdk%'))&(cmdline,!like,'%kube%'))&(cmdline,!like,'%-proxy%'))&(cmdline,!like,'%config%'))&(cmdline, [...]
-dipper.private.blink.rules&&&&express&&&&11380&&&&{"aesFlag":1,"varName":"___lower_proc_path_10004","functionName":"!like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"bPd623S6p9+IKPQzgj7jqA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11382&&&&{"aesFlag":1,"varName":"___lower_proc_path_10006","functionName":"!like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"8ID5+or/oei0n0A6H6LFJA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11381&&&&{"aesFlag":1,"varName":"___lower_proc_path_10005","functionName":"!like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"a7G8lVqM8m5MpXv4yM6Sxw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11384&&&&{"aesFlag":1,"varName":"___lower_proc_path_10008","functionName":"!like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"St1CoCCEbKkQLulE3uJS8g=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11383&&&&{"aesFlag":1,"varName":"___lower_proc_path_10007","functionName":"!like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"PA1uuOgNPyGNxqvLTmj65Q=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11427&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11426\",\"11391\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11426&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11425\",\"11390\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11429&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11428\",\"___!null_10048\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11428&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11427\",\"11392\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11386&&&&{"aesFlag":1,"varName":"___lower_proc_path_10010","functionName":"!like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"UWCrbTDUfKWSAb52gmiQtQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11385&&&&{"aesFlag":1,"varName":"___lower_proc_path_10009","functionName":"!like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"pvtHFcqFW4vZaZY99L38Qg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11388&&&&{"aesFlag":1,"varName":"___lower_proc_path_10012","functionName":"!like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"YpHdvBoL8TkWiAMopF0ONQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11421&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11420\",\"11385\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11387&&&&{"aesFlag":1,"varName":"___lower_proc_path_10011","functionName":"!like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"BU6BD0E+0voWkMpBALEIKA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11420&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11419\",\"11384\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11423&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11422\",\"11387\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11389&&&&{"aesFlag":1,"varName":"___lower_proc_path_10013","functionName":"!like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"+RXwIsAQBwC7HHP8mq3kGw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11422&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11421\",\"11386\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11425&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11424\",\"11389\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11424&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11423\",\"11388\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11391&&&&{"aesFlag":1,"varName":"___lower_proc_path_10015","functionName":"!like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"2rXrGosCjY9Eoo/fh2tmNQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11390&&&&{"aesFlag":1,"varName":"___lower_proc_path_10014","functionName":"!like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"+LGAKQAr9naYgnz9L4g2qw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11393&&&&{"aesFlag":1,"varName":"proc_path","functionName":"<>","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"M7grO0DWTNleGtk17FTQSw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11392&&&&{"aesFlag":1,"varName":"___lower_proc_path_10016","functionName":"!like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"p0e7qODAqBvZC7YwC4+dUQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11395&&&&{"aesFlag":1,"varName":"proc_path","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"IPypLhG4WGJOP4nvKQJkUTdGwgZOsTnmJYeu8m72okdynCVkGAXpO9j0B4Vq6UzaLkSC6+yvsOUHb+liRzFr2w=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11394&&&&{"aesFlag":1,"varName":"proc_path","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"zsOmosoNjV3J8o5lspZXPVoLfXXbaN3Qj0t0cTtDsO+BgtgZ49F8dVvPFowl529f1mzFXSLM4VAAXb57g1jGYZZMz2tHyPsK6dABmE7XfNQVwXUgFc7kpdyA1VkTIMxR"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11438&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11436\",\"11437\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11437&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11400\",\"11401\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11439&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11438\",\"11402\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11397&&&&{"aesFlag":1,"varName":"cmdline","functionName":"!like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"TvIgwB7604F//ITo85BhIw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11430&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11429\",\"11393\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11396&&&&{"aesFlag":1,"varName":"cmdline","functionName":"<>","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":""}&&&&null
-dipper.private.blink.rules&&&&express&&&&11399&&&&{"aesFlag":1,"varName":"proc_path","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"uYyF//VCc6AARu+aqn+bzd/y/GbZnyidJYA6VF802arwHoTvChbKQTu/a98QleBOcZk0kEstJoLJmvRPhWuSuTh+dSDPNBPlqsi3lsOPxEpclTRlSZCoYuvCc++IYPOKDK+BSoNL3BVo2a8l07bgMw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11432&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11396\",\"11397\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11398&&&&{"aesFlag":1,"varName":"proc_path","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"3zhi4QJjswHAFuRJcKjmyPd6jKBPiEeqIPEaVnZbg4+KLfq8s8pwPsMpkb3mDiAeG8zvaQC6XEbpQhd1XtwNDhcLEecZ8x/Ll4PC3mGfjZbk5Fq/u+IU8ljdCAiP6rcJ"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11431&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11430\",\"___!_10081\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11434&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11394\",\"11433\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11433&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11395\",\"11432\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11436&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11435\",\"11399\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11435&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11434\",\"11398\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11405&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11404\",\"11369\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11404&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11367\",\"11368\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11407&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11406\",\"11371\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11406&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11405\",\"11370\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11409&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11408\",\"11373\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11408&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11407\",\"11372\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11441&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11431\",\"11440\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11440&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11439\",\"11403\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11368&&&&{"varName":"___len_10008","functionName":">","fieldFlag":false,"dataType":"{\"className\":\"java.lang.Double\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"9.0"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11401&&&&{"aesFlag":1,"varName":"proc_path","functionName":"like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"L185Aioyc9BQCz0KBWnP4Q=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11367&&&&{"aesFlag":1,"varName":"status","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"mBo8DRi0jqLEUMAk3fPIkEAFxUrWkYtflRV52aAcmFs="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11400&&&&{"aesFlag":1,"varName":"proc_path","functionName":"!like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"LKyatoBKYggsg12A2XP6Rw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11403&&&&{"aesFlag":1,"varName":"proc_path","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"rh+ZjYZrKxl0QJnnmQMT0OGOd5ll/Q3OE6zVk1JisdtdPZFXL2VaTxvd9DJALkHPp63cT4WML2XGOlUD5NutNqluIPk11+AXbHhVO5DWXYxIyoa8zspQBX6tswytMFABBcbZZyDceUF9B/YTsZyEGmJt01vyJZyRIxFF8gYmNPChJCcpvhM5kYIpLceNfzk/iau7xJsdPENSzvDANQL4m/s8jLr0THmXAgwP1wx [...]
-dipper.private.blink.rules&&&&express&&&&11369&&&&{"aesFlag":1,"varName":"dst_ip","functionName":"!like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"XsTI7crBOHY734eWxoY/qA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11402&&&&{"aesFlag":1,"varName":"proc_path","functionName":"regex","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"b20///b7od4W7KkQGuFdT3cHr44lJrhI3X1z8Cq1Bs5ZRX7eO0Bo5M6GOfU/kIC8XVajVzXlXnnFIoiNkNPQ1gzmrSsgq0G8UG9xF5kVYxfaHxST6lCqCillaz32KQgieMlR3vtfKiaL6Go4A2ioKw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11371&&&&{"aesFlag":1,"varName":"dst_ip","functionName":"!like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"wGyK3zHJFe7jFO91U8ICMw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11370&&&&{"aesFlag":1,"varName":"dst_ip","functionName":"!like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"LBXZ3QrQBCl9c4prF0Z32Q=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11373&&&&{"aesFlag":1,"varName":"dst_ip","functionName":"!like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"JR3F6m8Pp+PN2+wGtruNqg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11372&&&&{"aesFlag":1,"varName":"dst_ip","functionName":"!like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"LsYOeVl0Z98oLKV0r9DUQQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11416&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11415\",\"11380\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11415&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11414\",\"11379\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11418&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11417\",\"11382\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11417&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11416\",\"11381\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11419&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11418\",\"11383\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11375&&&&{"aesFlag":1,"varName":"cmdline","functionName":"!like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"UWToOwPPb4g69jkG3iOP+g=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11374&&&&{"aesFlag":1,"varName":"dst_ip","functionName":"!like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"7wRHqLNV+Tl7yWTGrn3peA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11377&&&&{"aesFlag":1,"varName":"cmdline","functionName":"!like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"KkN8/qSKzeLSLvMNjYf8WQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11410&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11409\",\"11374\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11376&&&&{"aesFlag":1,"varName":"cmdline","functionName":"!like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"xaum6KaJlG40DBtZRhY7rg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11379&&&&{"aesFlag":1,"varName":"cmdline","functionName":"!like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"zPJajGmrdtnl6YvUdxRjQA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11412&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11411\",\"11376\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11378&&&&{"aesFlag":1,"varName":"cmdline","functionName":"!like","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"vKn74iKokQGtNLyYt8PwVg=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11411&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11410\",\"11375\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11414&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11413\",\"11378\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11413&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11412\",\"11377\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10002&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10002","type":"script","value":"uuid=host_uuid;\n___lower_proc_path_10018=lower(proc_path);\nproc_path=___lower_proc_path_10018;rm('___lower_proc_path_10018');\nproc_time=sca [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10003&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10003","type":"script","value":"retainField(src_ip,cmdline,proc_path,dst_port,pid,uuid,proc_time,dst_ip,ppid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10004&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10004","type":"script","value":"retainField(src_ip,cmdline,proc_path,dst_port,pid,uuid,proc_time,dst_ip,ppid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10005&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10005","type":"script","value":"client_ip=src_ip;\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10006&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10006","type":"script","value":"retainField(cmdline,proc_path,dst_port,client_ip,pid,uuid,proc_time,dst_ip,ppid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10007&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10007","type":"script","value":"___SALT_HASH_10003=SALT_HASH(dst_ip);;","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&dataSource&&&&blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_dataSource_10001&&&&{"className":"com.aliyun.yundun.dipper.configurable.http.resource.JDBCDataSource","configureName":"blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_dataSource_10001","userName":"xxxxxxxxxx","type":"dataSource","version":"1.0","url":"intelligence.rds.jdbc.url","timeout":"30000","activtyTimeOut":"3000","password":"xxxxxxxxx","isAutoFlush":"false","outputThr [...]
-dipper.private.blink.rules&&&&intelligence&&&&blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_intelligence_10001&&&&{"className":"com.aliyun.filter.intelligence.IPIntelligenceCache","pollingTimeMintue":"30","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_intelligence_10001","idFieldName":"id","batchSize":"3000","datasourceName":"blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_dataSource_10001"," [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10008&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10008","type":"script","value":"intelligence('dipper.private.blink.rules','blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_intelligence_10001',___SALT_HASH_10003, [...]
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_rule_10002&&&&{"expressionStr":"(p.ip,!in,'\\'118.190.90.134\\',\\'101.132.192.134\\'')&(((((p.is_3rd,==,'1')|(p.is_c2,==,'1'))|(p.is_tor,==,'1'))|(p.is_mining_pool,==,'1'))|(p.is_malicious_source,==,'1'))","scriptNames":"[]","expressionName":"11452","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blin [...]
-dipper.private.blink.rules&&&&express&&&&11449&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11448\",\"11445\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11448&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11443\",\"11444\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11450&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11449\",\"11446\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11452&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11442\",\"11451\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11451&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11450\",\"11447\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11443&&&&{"aesFlag":1,"varName":"p.is_3rd","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"2vQJrhr8/lZ2HEw+NK0Glw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11442&&&&{"aesFlag":1,"varName":"p.ip","functionName":"!in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"3XsN7bjYH29AxCpN+9V+tgnyB8FflSqdNfVWv5iXBDtBVE0vLLQUDH1U8MyALVbI"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11445&&&&{"aesFlag":1,"varName":"p.is_tor","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"2vQJrhr8/lZ2HEw+NK0Glw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11444&&&&{"aesFlag":1,"varName":"p.is_c2","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"2vQJrhr8/lZ2HEw+NK0Glw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11447&&&&{"aesFlag":1,"varName":"p.is_malicious_source","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"2vQJrhr8/lZ2HEw+NK0Glw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11446&&&&{"aesFlag":1,"varName":"p.is_mining_pool","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"2vQJrhr8/lZ2HEw+NK0Glw=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10009&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10009","type":"script","value":"is_3rd=p.is_3rd;\nis_tor=p.is_tor;\nis_c2=p.is_c2;\nis_malicious_source=p.is_malicious_source;\nis_mining_pool=p.is_mining_pool;\nretainField( [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10010&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10010","type":"script","value":"___null_10076=null(is_3rd);\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_rule_10003&&&&{"expressionStr":"(is_malicious_source,in,'\\'1\\'')&((is_3rd,in,'\\'0\\'')|___null_10076)","scriptNames":"[]","expressionName":"11456","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_rule_10003","ruleStatus":"0","type":"rule","v [...]
-dipper.private.blink.rules&&&&express&&&&11454&&&&{"aesFlag":1,"varName":"is_3rd","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"JYRSmU5cpZWdyekvHqdfVw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11453&&&&{"aesFlag":1,"varName":"is_malicious_source","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"jUdckUKzQ+H+FRkCrZ4LEA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11456&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11453\",\"11455\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11455&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11454\",\"___null_10076\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10011&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10011","type":"script","value":"ali_uid='-';\nevent_type='威胁情报';\nevent_name='访问恶意IP(恶意下载源)';\nlevel='high';\n___cast_11087=cast(dst_ip,'string');\n___cast_11088=cast(dst_por [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10012&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10012","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_channel_10001&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_channel_10001","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeInOrder [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10013&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10013","type":"script","value":"___null_10077=null(is_3rd);\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_rule_10004&&&&{"expressionStr":"(is_tor,in,'\\'1\\'')&((is_3rd,in,'\\'0\\'')|___null_10077)","scriptNames":"[]","expressionName":"11460","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_rule_10004","ruleStatus":"0","type":"rule","version":"1.0" [...]
-dipper.private.blink.rules&&&&express&&&&11459&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11458\",\"___null_10077\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11460&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11457\",\"11459\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11458&&&&{"aesFlag":1,"varName":"is_3rd","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"JYRSmU5cpZWdyekvHqdfVw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11457&&&&{"aesFlag":1,"varName":"is_tor","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"jUdckUKzQ+H+FRkCrZ4LEA=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10014&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10014","type":"script","value":"ali_uid='-';\nevent_type='威胁情报';\nevent_name='访问恶意IP(暗网通信)';\nlevel='high';\n___cast_11093=cast(dst_ip,'string');\n___cast_11094=cast(dst_port [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10015&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10015","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_channel_10002&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_channel_10002","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeInOrder [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10016&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10016","type":"script","value":"___null_10078=null(is_3rd);\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_rule_10005&&&&{"expressionStr":"(is_c2,in,'\\'1\\'')&((is_3rd,in,'\\'0\\'')|___null_10078)","scriptNames":"[]","expressionName":"11464","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_rule_10005","ruleStatus":"0","type":"rule","version":"1.0", [...]
-dipper.private.blink.rules&&&&express&&&&11461&&&&{"aesFlag":1,"varName":"is_c2","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"jUdckUKzQ+H+FRkCrZ4LEA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11463&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11462\",\"___null_10078\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11462&&&&{"aesFlag":1,"varName":"is_3rd","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"JYRSmU5cpZWdyekvHqdfVw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11464&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11461\",\"11463\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10017&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10017","type":"script","value":"ali_uid='-';\nevent_type='威胁情报';\nevent_name='访问恶意IP(中控通信)';\nlevel='high';\n___cast_11099=cast(dst_ip,'string');\n___cast_11100=cast(dst_port [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10018&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10018","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_channel_10003&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_channel_10003","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeInOrder [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10019&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10019","type":"script","value":"___null_10079=null(is_3rd);\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_rule_10006&&&&{"expressionStr":"(is_mining_pool,in,'\\'1\\'')&((is_3rd,in,'\\'0\\'')|___null_10079)","scriptNames":"[]","expressionName":"11468","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_rule_10006","ruleStatus":"0","type":"rule","versio [...]
-dipper.private.blink.rules&&&&express&&&&11465&&&&{"aesFlag":1,"varName":"is_mining_pool","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"jUdckUKzQ+H+FRkCrZ4LEA=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11467&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11466\",\"___null_10079\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11466&&&&{"aesFlag":1,"varName":"is_3rd","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"JYRSmU5cpZWdyekvHqdfVw=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11468&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11465\",\"11467\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10020&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10020","type":"script","value":"ali_uid='-';\nevent_type='威胁情报';\nevent_name='访问恶意IP(矿池通信)';\nlevel='high';\n___cast_11105=cast(dst_ip,'string');\n___cast_11106=cast(dst_port [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10021&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10021","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_channel_10004&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_channel_10004","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeInOrder [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10022&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10022","type":"script","value":"___null_10080=null(is_c2);\n___null_10081=null(is_tor);\n___null_10082=null(is_mining_pool);\n","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&rule&&&&blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_rule_10007&&&&{"expressionStr":"(((is_3rd,in,'\\'1\\'')&((is_c2,==,'0')|___null_10080))&((is_tor,==,'0')|___null_10081))&((is_mining_pool,==,'0')|___null_10082)","scriptNames":"[]","expressionName":"11478","varNames":"[]","className":"com.aliyun.filter.processor.FilterMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_intelligenc [...]
-dipper.private.blink.rules&&&&express&&&&11470&&&&{"aesFlag":1,"varName":"is_c2","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"4ROl+e4g5JZAI9EFfD9CUQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11472&&&&{"aesFlag":1,"varName":"is_mining_pool","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"4ROl+e4g5JZAI9EFfD9CUQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11471&&&&{"aesFlag":1,"varName":"is_tor","functionName":"==","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"4ROl+e4g5JZAI9EFfD9CUQ=="}&&&&null
-dipper.private.blink.rules&&&&express&&&&11474&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11469\",\"11473\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11473&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11470\",\"___null_10080\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11476&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11474\",\"11475\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11475&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11471\",\"___null_10081\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11478&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11476\",\"11477\"]","relation":"and"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11477&&&&{"fieldFlag":false,"dataType":"{\"className\":\"java.util.List\",\"genericParameter\":\"java.util.List<java.lang.String>\"}","className":"com.aliyun.filter.rules.expression.RelationExpression","keyword":"","value":"[\"11472\",\"___null_10082\"]","relation":"or"}&&&&null
-dipper.private.blink.rules&&&&express&&&&11469&&&&{"aesFlag":1,"varName":"is_3rd","functionName":"in","fieldFlag":false,"dataType":"{\"className\":\"java.lang.String\"}","className":"com.aliyun.filter.rules.expression.SimpleExpression","keyword":"","value":"jUdckUKzQ+H+FRkCrZ4LEA=="}&&&&null
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10023&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10023","type":"script","value":"ali_uid='-';\nevent_type='威胁情报';\nevent_name='访问恶意IP(公开情报)';\nlevel='high';\n___cast_11111=cast(dst_ip,'string');\n___cast_11112=cast(dst_port [...]
-dipper.private.blink.rules&&&&script&&&&blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10024&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_script_10024","type":"script","value":"retainField(gmt_create,ali_uid,event_type,level,event_name,client_ip,ext_content,gmt_modified,uuid);","version":"1.0","extendField":"[]"}&&&&null
-dipper.private.blink.rules&&&&channel&&&&blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_channel_10005&&&&{"isJsonData":"true","project":"k8sblink","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"blink_source_aegis_net_adl_sas_apsara_intelligence_ip_alert_channel_10005","type":"channel","timeout":"30000","accessId":"replace_accessId_all","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeInOrder [...]
-dipper.private.blink.rules&&&&pipline&&&&blink.source.aegis.net&&&&{"isAutoStart":"false","stages":"[\"{\\\"className\\\":\\\"com.aliyun.yundun.dipper.common.pipline.ScriptChainStage\\\",\\\"configurable_value\\\":\\\"{\\\\\\\"prewStageLables\\\\\\\":\\\\\\\"[]\\\\\\\",\\\\\\\"closeSplitMode\\\\\\\":\\\\\\\"false\\\\\\\",\\\\\\\"entityName\\\\\\\":\\\\\\\"script\\\\\\\",\\\\\\\"scriptName\\\\\\\":\\\\\\\"blink.source.aegis.net_script_10001\\\\\\\",\\\\\\\"lable\\\\\\\":\\\\\\\"blink.sour [...]
-dipper.private.blink.rules&&&&channel&&&&blink.source.aegis.net;channel&&&&{"offset":"0","isJsonData":"true","maxThread":"12","concurrentCount":"12","className":"com.aliyun.yundun.dipper.channel.adapter.MetaqChannel","configureName":"blink.source.aegis.net;channel","type":"channel","version":"1.0","timeout":"30000","activtyTimeOut":"3000","tags":"blink.source.aegis.net.tags","syncTimeout":"5000","groupName":"blink.source.aegis.net.group","isBatchMessage":"true","isAutoFlush":"false","max [...]
-dipper.private.blink.rules&&&&script&&&&blink.source.aegis.net_script_10001&&&&{"className":"com.aliyun.yundun.dipper.script.script.ScriptMessageProcessor","nameSpace":"dipper.private.blink.rules","configureName":"blink.source.aegis.net_script_10001","type":"script","value":"logContent=data;logTime=now();splitArray('logContent');scan_time=time;cmdline=proc_cmdline;parent_proc_file_name=pfilename;parent_start_time=pstime;___compare_10001=equals(status,'0');___compare_10002=equals(status,' [...]
-dipper.private.blink.rules&&&&channel&&&&out.mock.switch_print&&&&{"offset":"0","isJsonData":"true","maxThread":"12","concurrentCount":"1","className":"com.aliyun.yundun.dipper.common.resource.channel.OutputTestChannel","configureName":"out.mock.switch_print","type":"channel","version":"1.0","timeout":"30000","activtyTimeOut":"3000","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","maxFetchLogGroupSize":"100","topic":"","syncCount":"1000","outputThreadCount":"-1","name [...]
-dipper.private.blink.rules&&&&channel&&&&out.mock.switch_sls&&&&{"isJsonData":"true","project":"out.mock.sls.project","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.SLSChannel","configureName":"out.mock.switch_sls","type":"channel","timeout":"30000","accessId":"out.mock.sls.accessId","syncTimeout":"5000","isBatchMessage":"false","isAutoFlush":"false","consumeInOrder":"true","maxFetchLogGroupSize":"100","syncCount":"1000","nameSpace":"dipper.private.blink.rule [...]
-dipper.private.blink.rules&&&&channel&&&&out.mock.switch_metaq&&&&{"offset":"0","isJsonData":"true","maxThread":"12","concurrentCount":"1","className":"com.aliyun.yundun.dipper.channel.adapter.MetaqChannel","configureName":"out.mock.switch_metaq","type":"channel","version":"1.0","timeout":"30000","activtyTimeOut":"3000","tags":"out.mock.metaq.tag","syncTimeout":"5000","isBatchMessage":"true","isAutoFlush":"false","maxFetchLogGroupSize":"100","topic":"out.mock.metaq.topic","syncCount":"10 [...]
diff --git a/rocketmq-streams-window/target/classes/dipper.properties b/rocketmq-streams-window/target/classes/dipper.properties
deleted file mode 100644
index c4591e4..0000000
--- a/rocketmq-streams-window/target/classes/dipper.properties
+++ /dev/null
@@ -1,21 +0,0 @@
-
-##如果是db类型，需要配置db连接信息
-dipper.rds.jdbc.password=xxxxxxx
-dipper.rds.jdbc.url=xxxxxx
-dipper.rds.jdbc.username=xxxxxx
-dipper.rds.table.name=dipper_sql_configure
-
-
-#window 异步消息分发需要的消息队列类型
-dipper.window.shuffle.rocketmq.dispatch.channel.type=rocketmq
-#如果是metaq，统计和join分别用的topoc
-dipper.window.shuffle.rocketmq.dispatch.channel.topic=TOPIC_DIPPER_WINDOW_STATISTICS
-#动态生成的消息队列的属性的key值，这个值会被动态赋值
-dipper.window.shuffle.rocketmq.dispatch.channel.dynamic.property=tag
-dipper.window.shuffle.rocketmq.dispatch.channel.thread.max.count=4
-dipper.window.shuffle.rocketmq.dispatch.channel.accessKey=XXX
-dipper.window.shuffle.rocketmq.dispatch.channel.secretKey=XXX
-dipper.window.shuffle.rocketmq.dispatch.channel.instanceId=xxxxxx
-dipper.window.shuffle.rocketmq.dispatch.channel.group=GID_dipper_shuffle_group
-dipper.window.shuffle.rocketmq.dispatch.channel.namesrvAddr=xxxxx
-dipper.window.shuffle.rocketmq.dispatch.channel.endpoint=xxxxxx
\ No newline at end of file
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/builder/WindowBuilder.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/builder/WindowBuilder.class
deleted file mode 100644
index acb24ce..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/builder/WindowBuilder.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/model/FunctionExecutor.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/model/FunctionExecutor.class
deleted file mode 100644
index 0dd3fb4..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/model/FunctionExecutor.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/model/WindowCache.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/model/WindowCache.class
deleted file mode 100644
index e077c40..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/model/WindowCache.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/model/WindowInstance.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/model/WindowInstance.class
deleted file mode 100644
index 2b577a9..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/model/WindowInstance.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/offset/IWindowMaxValueManager.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/offset/IWindowMaxValueManager.class
deleted file mode 100644
index 18f5cc3..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/offset/IWindowMaxValueManager.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/offset/WindowMaxValue.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/offset/WindowMaxValue.class
deleted file mode 100644
index e57f3b2..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/offset/WindowMaxValue.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/offset/WindowMaxValueManager.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/offset/WindowMaxValueManager.class
deleted file mode 100644
index cb18860..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/offset/WindowMaxValueManager.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/operator/AbstractShuffleWindow.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/operator/AbstractShuffleWindow.class
deleted file mode 100644
index 56e53f9..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/operator/AbstractShuffleWindow.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/operator/AbstractWindow$1.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/operator/AbstractWindow$1.class
deleted file mode 100644
index f64162f..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/operator/AbstractWindow$1.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/operator/AbstractWindow.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/operator/AbstractWindow.class
deleted file mode 100644
index 806e74d..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/operator/AbstractWindow.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/operator/impl/OverWindow.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/operator/impl/OverWindow.class
deleted file mode 100644
index ece1532..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/operator/impl/OverWindow.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/operator/impl/SessionWindow.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/operator/impl/SessionWindow.class
deleted file mode 100644
index 563f84b..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/operator/impl/SessionWindow.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/operator/impl/WindowOperator$1.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/operator/impl/WindowOperator$1.class
deleted file mode 100644
index 4da797b..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/operator/impl/WindowOperator$1.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/operator/impl/WindowOperator$WindowRowOperator.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/operator/impl/WindowOperator$WindowRowOperator.class
deleted file mode 100644
index 5a0dceb..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/operator/impl/WindowOperator$WindowRowOperator.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/operator/impl/WindowOperator.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/operator/impl/WindowOperator.class
deleted file mode 100644
index c2d4f11..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/operator/impl/WindowOperator.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/operator/join/DBOperator.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/operator/join/DBOperator.class
deleted file mode 100644
index eefe6c1..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/operator/join/DBOperator.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/operator/join/JoinWindow$1.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/operator/join/JoinWindow$1.class
deleted file mode 100644
index c23b4b3..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/operator/join/JoinWindow$1.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/operator/join/JoinWindow$2.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/operator/join/JoinWindow$2.class
deleted file mode 100644
index d47b70b..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/operator/join/JoinWindow$2.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/operator/join/JoinWindow.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/operator/join/JoinWindow.class
deleted file mode 100644
index 1708b94..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/operator/join/JoinWindow.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/operator/join/Operator.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/operator/join/Operator.class
deleted file mode 100644
index d68f4de..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/operator/join/Operator.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/shuffle/AbstractSystemChannel.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/shuffle/AbstractSystemChannel.class
deleted file mode 100644
index 163f25a..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/shuffle/AbstractSystemChannel.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/shuffle/ShuffleChannel$ShuffleOutputDataSource.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/shuffle/ShuffleChannel$ShuffleOutputDataSource.class
deleted file mode 100644
index 46a0202..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/shuffle/ShuffleChannel$ShuffleOutputDataSource.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/shuffle/ShuffleChannel.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/shuffle/ShuffleChannel.class
deleted file mode 100644
index 6a536a0..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/shuffle/ShuffleChannel.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/source/WindowRireSource$1$1.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/source/WindowRireSource$1$1.class
deleted file mode 100644
index ca22ccd..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/source/WindowRireSource$1$1.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/source/WindowRireSource$1.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/source/WindowRireSource$1.class
deleted file mode 100644
index 8801c36..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/source/WindowRireSource$1.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/source/WindowRireSource$WindowInstanceCache$1$1.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/source/WindowRireSource$WindowInstanceCache$1$1.class
deleted file mode 100644
index 221bcf3..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/source/WindowRireSource$WindowInstanceCache$1$1.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/source/WindowRireSource$WindowInstanceCache$1.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/source/WindowRireSource$WindowInstanceCache$1.class
deleted file mode 100644
index e6c1681..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/source/WindowRireSource$WindowInstanceCache$1.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/source/WindowRireSource$WindowInstanceCache.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/source/WindowRireSource$WindowInstanceCache.class
deleted file mode 100644
index e38b444..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/source/WindowRireSource$WindowInstanceCache.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/source/WindowRireSource.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/source/WindowRireSource.class
deleted file mode 100644
index 0952230..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/source/WindowRireSource.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/state/WindowBaseValue.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/state/WindowBaseValue.class
deleted file mode 100644
index 46a3e50..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/state/WindowBaseValue.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/state/impl/JoinLeftState.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/state/impl/JoinLeftState.class
deleted file mode 100644
index c3076fc..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/state/impl/JoinLeftState.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/state/impl/JoinRightState.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/state/impl/JoinRightState.class
deleted file mode 100644
index 559661e..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/state/impl/JoinRightState.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/state/impl/JoinState.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/state/impl/JoinState.class
deleted file mode 100644
index 5d30166..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/state/impl/JoinState.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/state/impl/WindowValue$1.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/state/impl/WindowValue$1.class
deleted file mode 100644
index 0040abe..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/state/impl/WindowValue$1.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/state/impl/WindowValue.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/state/impl/WindowValue.class
deleted file mode 100644
index 2a47cd2..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/state/impl/WindowValue.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/AbstractWindowStorage$1.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/AbstractWindowStorage$1.class
deleted file mode 100644
index 11756bf..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/AbstractWindowStorage$1.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/AbstractWindowStorage.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/AbstractWindowStorage.class
deleted file mode 100644
index 32c7fda..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/AbstractWindowStorage.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/ICommonStorage.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/ICommonStorage.class
deleted file mode 100644
index d62d93d..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/ICommonStorage.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/IKeyGenerator.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/IKeyGenerator.class
deleted file mode 100644
index a17eea8..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/IKeyGenerator.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/IShufflePartitionManager.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/IShufflePartitionManager.class
deleted file mode 100644
index 9d6f3a0..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/IShufflePartitionManager.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/IStorage.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/IStorage.class
deleted file mode 100644
index 2dfe78d..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/IStorage.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/IWindowStorage.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/IWindowStorage.class
deleted file mode 100644
index 9e35799..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/IWindowStorage.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/ShufflePartitionManager.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/ShufflePartitionManager.class
deleted file mode 100644
index 764fac2..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/ShufflePartitionManager.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/StorageManager.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/StorageManager.class
deleted file mode 100644
index b60e1e4..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/StorageManager.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/WindowStorage$1.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/WindowStorage$1.class
deleted file mode 100644
index 788adb8..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/WindowStorage$1.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/WindowStorage$WindowBaseValueIterator.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/WindowStorage$WindowBaseValueIterator.class
deleted file mode 100644
index 0e4d4d1..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/WindowStorage$WindowBaseValueIterator.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/WindowStorage.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/WindowStorage.class
deleted file mode 100644
index fd05d39..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/WindowStorage.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/db/DBStorage$1.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/db/DBStorage$1.class
deleted file mode 100644
index e6cc378..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/db/DBStorage$1.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/db/DBStorage$DBIterator.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/db/DBStorage$DBIterator.class
deleted file mode 100644
index 171639e..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/db/DBStorage$DBIterator.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/db/DBStorage.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/db/DBStorage.class
deleted file mode 100644
index 274b33a..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/db/DBStorage.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/file/FileStorage$1.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/file/FileStorage$1.class
deleted file mode 100644
index de1e604..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/file/FileStorage$1.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/file/FileStorage.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/file/FileStorage.class
deleted file mode 100644
index 0858297..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/file/FileStorage.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/rocksdb/RocksdbStorage$LocalIterator.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/rocksdb/RocksdbStorage$LocalIterator.class
deleted file mode 100644
index 8f6d7d0..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/rocksdb/RocksdbStorage$LocalIterator.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/rocksdb/RocksdbStorage.class b/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/rocksdb/RocksdbStorage.class
deleted file mode 100644
index 05ea41b..0000000
Binary files a/rocketmq-streams-window/target/classes/org/apache/rocketmq/streams/window/storage/rocksdb/RocksdbStorage.class and /dev/null differ
diff --git a/rocketmq-streams-window/target/maven-archiver/pom.properties b/rocketmq-streams-window/target/maven-archiver/pom.properties
deleted file mode 100644
index 02202d4..0000000
--- a/rocketmq-streams-window/target/maven-archiver/pom.properties
+++ /dev/null
@@ -1,5 +0,0 @@
-#Generated by Maven
-#Fri Jul 30 11:10:37 CST 2021
-version=2.0.0-SNAPSHOT
-groupId=org.apache.rocketmq
-artifactId=rocketmq-streams-window
diff --git a/rocketmq-streams-window/target/rocketmq-streams-window-2.0.0-SNAPSHOT-sources.jar b/rocketmq-streams-window/target/rocketmq-streams-window-2.0.0-SNAPSHOT-sources.jar
deleted file mode 100644
index 3630c77..0000000
Binary files a/rocketmq-streams-window/target/rocketmq-streams-window-2.0.0-SNAPSHOT-sources.jar and /dev/null differ
diff --git a/rocketmq-streams-window/target/rocketmq-streams-window-2.0.0-SNAPSHOT.jar b/rocketmq-streams-window/target/rocketmq-streams-window-2.0.0-SNAPSHOT.jar
deleted file mode 100644
index ba5a783..0000000
Binary files a/rocketmq-streams-window/target/rocketmq-streams-window-2.0.0-SNAPSHOT.jar and /dev/null differ
diff --git a/rocketmq-streams-window/target/test-classes/log4j.xml b/rocketmq-streams-window/target/test-classes/log4j.xml
deleted file mode 100644
index 7812fe7..0000000
--- a/rocketmq-streams-window/target/test-classes/log4j.xml
+++ /dev/null
@@ -1,20 +0,0 @@
-<?xml version="1.0" encoding="UTF-8" ?>
-<!DOCTYPE log4j:configuration SYSTEM "http://toolkit.alibaba-inc.com/dtd/log4j/log4j.dtd">
-<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
-
- <appender name="Console" class="org.apache.log4j.ConsoleAppender">
- <layout class="org.apache.log4j.PatternLayout">
- <param name="ConversionPattern" value="%d{ISO8601} %l [%t] %-5p - %m%n%n"/>
- </layout>
- <filter class="org.apache.log4j.varia.LevelRangeFilter">
- <param name="LevelMin" value="INFO"/>
- <param name="LevelMax" value="ERROR"/>
- </filter>
- </appender>
-
- <root>
- <priority value="INFO"/>
- <appender-ref ref="Console"/>
- </root>
-
-</log4j:configuration>
\ No newline at end of file
[rocketmq-streams] 30/35: Remove .iml file
Posted by du...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
duhengforever pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/rocketmq-streams.git
commit 4ce3b3be407faf63dbf17c7f5bad2749f136b02c
Author: 通融 <ro...@alibaba-inc.com>
AuthorDate: Mon Aug 2 19:54:02 2021 +0800
Remove .iml file
---
.../rocketmq-streams-channel-http.iml | 62 -------------------
.../rocketmq-streams-db-operator.iml | 16 -----
.../rocketmq-streams-filter.iml | 72 ----------------------
.../rocketmq-streams-schedule.iml | 65 -------------------
.../rocketmq-streams-transport-minio.iml | 17 -----
5 files changed, 232 deletions(-)
diff --git a/rocketmq-streams-channel-http/rocketmq-streams-channel-http.iml b/rocketmq-streams-channel-http/rocketmq-streams-channel-http.iml
deleted file mode 100644
index 51a5150..0000000
--- a/rocketmq-streams-channel-http/rocketmq-streams-channel-http.iml
+++ /dev/null
@@ -1,62 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<module org.jetbrains.idea.maven.project.MavenProjectsManager.isMavenModule="true" type="JAVA_MODULE" version="4">
- <component name="NewModuleRootManager" LANGUAGE_LEVEL="JDK_1_8">
- <output url="file://$MODULE_DIR$/target/classes" />
- <output-test url="file://$MODULE_DIR$/target/test-classes" />
- <content url="file://$MODULE_DIR$">
- <sourceFolder url="file://$MODULE_DIR$/src/main/java" isTestSource="false" />
- <sourceFolder url="file://$MODULE_DIR$/src/test/java" isTestSource="true" />
- <excludeFolder url="file://$MODULE_DIR$/target" />
- </content>
- <orderEntry type="inheritedJdk" />
- <orderEntry type="sourceFolder" forTests="false" />
- <orderEntry type="module" module-name="rocketmq-streams-commons" />
- <orderEntry type="library" name="Maven: org.apache.commons:commons-lang3:3.11" level="project" />
- <orderEntry type="library" name="Maven: junit:junit:4.12" level="project" />
- <orderEntry type="library" name="Maven: org.hamcrest:hamcrest-core:1.3" level="project" />
- <orderEntry type="library" name="Maven: com.alibaba:fastjson:1.2.27" level="project" />
- <orderEntry type="library" name="Maven: commons-logging:commons-logging:1.1" level="project" />
- <orderEntry type="library" name="Maven: logkit:logkit:1.0.1" level="project" />
- <orderEntry type="library" name="Maven: avalon-framework:avalon-framework:4.1.3" level="project" />
- <orderEntry type="library" name="Maven: javax.servlet:servlet-api:2.3" level="project" />
- <orderEntry type="library" name="Maven: log4j:log4j:1.2.17" level="project" />
- <orderEntry type="library" name="Maven: com.google.code.gson:gson:2.8.5" level="project" />
- <orderEntry type="library" name="Maven: com.google.auto.service:auto-service:1.0-rc5" level="project" />
- <orderEntry type="library" name="Maven: com.google.auto.service:auto-service-annotations:1.0-rc5" level="project" />
- <orderEntry type="library" name="Maven: com.google.auto:auto-common:0.10" level="project" />
- <orderEntry type="library" name="Maven: com.google.guava:guava:25.1-jre" level="project" />
- <orderEntry type="library" name="Maven: com.google.code.findbugs:jsr305:3.0.2" level="project" />
- <orderEntry type="library" name="Maven: org.checkerframework:checker-qual:2.0.0" level="project" />
- <orderEntry type="library" name="Maven: com.google.errorprone:error_prone_annotations:2.1.3" level="project" />
- <orderEntry type="library" name="Maven: com.google.j2objc:j2objc-annotations:1.1" level="project" />
- <orderEntry type="library" name="Maven: org.codehaus.mojo:animal-sniffer-annotations:1.14" level="project" />
- <orderEntry type="library" name="Maven: com.lmax:disruptor:3.2.0" level="project" />
- <orderEntry type="library" name="Maven: com.gliwka.hyperscan:hyperscan:5.4.0-2.0.0" level="project" />
- <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:5.4.0-1.0.0" level="project" />
- <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:linux-x86_64:5.4.0-1.0.0" level="project" />
- <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:windows-x86_64:5.4.0-1.0.0" level="project" />
- <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:macosx-x86_64:5.4.0-1.0.0" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp-platform:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-arm:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-arm64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-x86:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-x86_64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:ios-arm64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:ios-x86_64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-armhf:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-arm64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-ppc64le:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-x86:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-x86_64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:macosx-x86_64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:windows-x86:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:windows-x86_64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: net.java.dev.jna:platform:3.5.2" level="project" />
- <orderEntry type="library" name="Maven: net.java.dev.jna:jna:3.5.2" level="project" />
- <orderEntry type="library" name="Maven: commons-io:commons-io:2.5" level="project" />
- <orderEntry type="library" name="Maven: org.apache.httpcomponents:httpclient:4.5.2" level="project" />
- <orderEntry type="library" name="Maven: org.apache.httpcomponents:httpcore:4.4.4" level="project" />
- <orderEntry type="library" name="Maven: commons-codec:commons-codec:1.9" level="project" />
- </component>
-</module>
\ No newline at end of file
diff --git a/rocketmq-streams-db-operator/rocketmq-streams-db-operator.iml b/rocketmq-streams-db-operator/rocketmq-streams-db-operator.iml
deleted file mode 100644
index 38ffb14..0000000
--- a/rocketmq-streams-db-operator/rocketmq-streams-db-operator.iml
+++ /dev/null
@@ -1,16 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<module org.jetbrains.idea.maven.project.MavenProjectsManager.isMavenModule="true" type="JAVA_MODULE" version="4">
- <component name="NewModuleRootManager" LANGUAGE_LEVEL="JDK_1_5">
- <output url="file://$MODULE_DIR$/${project.build.directory}/classes" />
- <output-test url="file://$MODULE_DIR$/${project.build.directory}/test-classes" />
- <content url="file://$MODULE_DIR$">
- <sourceFolder url="file://$MODULE_DIR$/src/main/java" isTestSource="false" />
- <sourceFolder url="file://$MODULE_DIR$/src/test/java" isTestSource="true" />
- <excludeFolder url="file://$MODULE_DIR$/${project.build.directory}/classes" />
- <excludeFolder url="file://$MODULE_DIR$/${project.build.directory}/test-classes" />
- <excludeFolder url="file://$MODULE_DIR$/target" />
- </content>
- <orderEntry type="inheritedJdk" />
- <orderEntry type="sourceFolder" forTests="false" />
- </component>
-</module>
\ No newline at end of file
diff --git a/rocketmq-streams-filter/rocketmq-streams-filter.iml b/rocketmq-streams-filter/rocketmq-streams-filter.iml
deleted file mode 100644
index 70f1781..0000000
--- a/rocketmq-streams-filter/rocketmq-streams-filter.iml
+++ /dev/null
@@ -1,72 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<module org.jetbrains.idea.maven.project.MavenProjectsManager.isMavenModule="true" type="JAVA_MODULE" version="4">
- <component name="NewModuleRootManager" LANGUAGE_LEVEL="JDK_1_8">
- <output url="file://$MODULE_DIR$/target/classes" />
- <output-test url="file://$MODULE_DIR$/target/test-classes" />
- <content url="file://$MODULE_DIR$">
- <sourceFolder url="file://$MODULE_DIR$/src/main/java" isTestSource="false" />
- <sourceFolder url="file://$MODULE_DIR$/src/test/java" isTestSource="true" />
- <sourceFolder url="file://$MODULE_DIR$/src/test/resources" type="java-test-resource" />
- <excludeFolder url="file://$MODULE_DIR$/target" />
- </content>
- <orderEntry type="inheritedJdk" />
- <orderEntry type="sourceFolder" forTests="false" />
- <orderEntry type="module" module-name="rocketmq-streams-script" />
- <orderEntry type="module" module-name="rocketmq-streams-commons" />
- <orderEntry type="library" name="Maven: org.apache.commons:commons-lang3:3.11" level="project" />
- <orderEntry type="library" name="Maven: junit:junit:4.12" level="project" />
- <orderEntry type="library" name="Maven: org.hamcrest:hamcrest-core:1.3" level="project" />
- <orderEntry type="library" name="Maven: com.alibaba:fastjson:1.2.27" level="project" />
- <orderEntry type="library" name="Maven: commons-logging:commons-logging:1.1" level="project" />
- <orderEntry type="library" name="Maven: logkit:logkit:1.0.1" level="project" />
- <orderEntry type="library" name="Maven: avalon-framework:avalon-framework:4.1.3" level="project" />
- <orderEntry type="library" name="Maven: javax.servlet:servlet-api:2.3" level="project" />
- <orderEntry type="library" name="Maven: commons-io:commons-io:2.5" level="project" />
- <orderEntry type="library" name="Maven: log4j:log4j:1.2.17" level="project" />
- <orderEntry type="library" name="Maven: com.google.code.gson:gson:2.8.5" level="project" />
- <orderEntry type="library" name="Maven: com.google.auto.service:auto-service:1.0-rc5" level="project" />
- <orderEntry type="library" name="Maven: com.google.auto.service:auto-service-annotations:1.0-rc5" level="project" />
- <orderEntry type="library" name="Maven: com.google.auto:auto-common:0.10" level="project" />
- <orderEntry type="library" name="Maven: com.google.guava:guava:25.1-jre" level="project" />
- <orderEntry type="library" name="Maven: com.google.code.findbugs:jsr305:3.0.2" level="project" />
- <orderEntry type="library" name="Maven: org.checkerframework:checker-qual:2.0.0" level="project" />
- <orderEntry type="library" name="Maven: com.google.errorprone:error_prone_annotations:2.1.3" level="project" />
- <orderEntry type="library" name="Maven: com.google.j2objc:j2objc-annotations:1.1" level="project" />
- <orderEntry type="library" name="Maven: org.codehaus.mojo:animal-sniffer-annotations:1.14" level="project" />
- <orderEntry type="library" name="Maven: com.lmax:disruptor:3.2.0" level="project" />
- <orderEntry type="library" name="Maven: com.gliwka.hyperscan:hyperscan:5.4.0-2.0.0" level="project" />
- <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:5.4.0-1.0.0" level="project" />
- <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:linux-x86_64:5.4.0-1.0.0" level="project" />
- <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:windows-x86_64:5.4.0-1.0.0" level="project" />
- <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:macosx-x86_64:5.4.0-1.0.0" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp-platform:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-arm:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-arm64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-x86:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-x86_64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:ios-arm64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:ios-x86_64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-armhf:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-arm64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-ppc64le:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-x86:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-x86_64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:macosx-x86_64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:windows-x86:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:windows-x86_64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: net.java.dev.jna:platform:3.5.2" level="project" />
- <orderEntry type="library" name="Maven: net.java.dev.jna:jna:3.5.2" level="project" />
- <orderEntry type="library" name="Maven: io.krakens:java-grok:0.1.9" level="project" />
- <orderEntry type="library" name="Maven: org.codehaus.groovy:groovy-all:2.1.8" level="project" />
- <orderEntry type="library" name="Maven: org.python:jython-standalone:2.7.0" level="project" />
- <orderEntry type="module" module-name="rocketmq-streams-db-operator" />
- <orderEntry type="module" module-name="rocketmq-streams-configurable" />
- <orderEntry type="module" module-name="rocketmq-streams-serviceloader" />
- <orderEntry type="library" name="Maven: org.springframework:spring-jdbc:3.2.13.RELEASE" level="project" />
- <orderEntry type="library" name="Maven: org.springframework:spring-beans:3.2.13.RELEASE" level="project" />
- <orderEntry type="library" name="Maven: org.springframework:spring-core:3.2.13.RELEASE" level="project" />
- <orderEntry type="library" name="Maven: org.springframework:spring-tx:3.2.13.RELEASE" level="project" />
- <orderEntry type="library" name="Maven: mysql:mysql-connector-java:5.1.40" level="project" />
- </component>
-</module>
\ No newline at end of file
diff --git a/rocketmq-streams-schedule/rocketmq-streams-schedule.iml b/rocketmq-streams-schedule/rocketmq-streams-schedule.iml
deleted file mode 100644
index b51d7e1..0000000
--- a/rocketmq-streams-schedule/rocketmq-streams-schedule.iml
+++ /dev/null
@@ -1,65 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<module org.jetbrains.idea.maven.project.MavenProjectsManager.isMavenModule="true" type="JAVA_MODULE" version="4">
- <component name="NewModuleRootManager" LANGUAGE_LEVEL="JDK_1_8">
- <output url="file://$MODULE_DIR$/target/classes" />
- <output-test url="file://$MODULE_DIR$/target/test-classes" />
- <content url="file://$MODULE_DIR$">
- <sourceFolder url="file://$MODULE_DIR$/src/main/java" isTestSource="false" />
- <sourceFolder url="file://$MODULE_DIR$/src/test/java" isTestSource="true" />
- <sourceFolder url="file://$MODULE_DIR$/src/test/resources" type="java-test-resource" />
- <excludeFolder url="file://$MODULE_DIR$/target" />
- </content>
- <orderEntry type="inheritedJdk" />
- <orderEntry type="sourceFolder" forTests="false" />
- <orderEntry type="library" name="Maven: org.quartz-scheduler:quartz:2.2.1" level="project" />
- <orderEntry type="library" name="Maven: c3p0:c3p0:0.9.1.1" level="project" />
- <orderEntry type="library" name="Maven: org.slf4j:slf4j-api:1.6.6" level="project" />
- <orderEntry type="library" name="Maven: org.quartz-scheduler:quartz-jobs:2.2.1" level="project" />
- <orderEntry type="module" module-name="rocketmq-streams-serviceloader" />
- <orderEntry type="module" module-name="rocketmq-streams-commons" />
- <orderEntry type="library" name="Maven: org.apache.commons:commons-lang3:3.11" level="project" />
- <orderEntry type="library" name="Maven: junit:junit:4.12" level="project" />
- <orderEntry type="library" name="Maven: org.hamcrest:hamcrest-core:1.3" level="project" />
- <orderEntry type="library" name="Maven: com.alibaba:fastjson:1.2.27" level="project" />
- <orderEntry type="library" name="Maven: commons-logging:commons-logging:1.1" level="project" />
- <orderEntry type="library" name="Maven: logkit:logkit:1.0.1" level="project" />
- <orderEntry type="library" name="Maven: avalon-framework:avalon-framework:4.1.3" level="project" />
- <orderEntry type="library" name="Maven: javax.servlet:servlet-api:2.3" level="project" />
- <orderEntry type="library" name="Maven: commons-io:commons-io:2.5" level="project" />
- <orderEntry type="library" name="Maven: log4j:log4j:1.2.17" level="project" />
- <orderEntry type="library" name="Maven: com.google.code.gson:gson:2.8.5" level="project" />
- <orderEntry type="library" name="Maven: com.google.auto.service:auto-service:1.0-rc5" level="project" />
- <orderEntry type="library" name="Maven: com.google.auto.service:auto-service-annotations:1.0-rc5" level="project" />
- <orderEntry type="library" name="Maven: com.google.auto:auto-common:0.10" level="project" />
- <orderEntry type="library" name="Maven: com.google.guava:guava:25.1-jre" level="project" />
- <orderEntry type="library" name="Maven: com.google.code.findbugs:jsr305:3.0.2" level="project" />
- <orderEntry type="library" name="Maven: org.checkerframework:checker-qual:2.0.0" level="project" />
- <orderEntry type="library" name="Maven: com.google.errorprone:error_prone_annotations:2.1.3" level="project" />
- <orderEntry type="library" name="Maven: com.google.j2objc:j2objc-annotations:1.1" level="project" />
- <orderEntry type="library" name="Maven: org.codehaus.mojo:animal-sniffer-annotations:1.14" level="project" />
- <orderEntry type="library" name="Maven: com.lmax:disruptor:3.2.0" level="project" />
- <orderEntry type="library" name="Maven: com.gliwka.hyperscan:hyperscan:5.4.0-2.0.0" level="project" />
- <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:5.4.0-1.0.0" level="project" />
- <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:linux-x86_64:5.4.0-1.0.0" level="project" />
- <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:windows-x86_64:5.4.0-1.0.0" level="project" />
- <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:macosx-x86_64:5.4.0-1.0.0" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp-platform:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-arm:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-arm64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-x86:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-x86_64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:ios-arm64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:ios-x86_64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-armhf:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-arm64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-ppc64le:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-x86:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-x86_64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:macosx-x86_64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:windows-x86:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:windows-x86_64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: net.java.dev.jna:platform:3.5.2" level="project" />
- <orderEntry type="library" name="Maven: net.java.dev.jna:jna:3.5.2" level="project" />
- </component>
-</module>
\ No newline at end of file
diff --git a/rocketmq-streams-transport-minio/rocketmq-streams-transport-minio.iml b/rocketmq-streams-transport-minio/rocketmq-streams-transport-minio.iml
deleted file mode 100644
index af11f77..0000000
--- a/rocketmq-streams-transport-minio/rocketmq-streams-transport-minio.iml
+++ /dev/null
@@ -1,17 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<module org.jetbrains.idea.maven.project.MavenProjectsManager.isMavenModule="true" type="JAVA_MODULE" version="4">
- <component name="NewModuleRootManager" LANGUAGE_LEVEL="JDK_1_5">
- <output url="file://$MODULE_DIR$/${project.build.directory}/classes" />
- <output-test url="file://$MODULE_DIR$/${project.build.directory}/test-classes" />
- <content url="file://$MODULE_DIR$">
- <sourceFolder url="file://$MODULE_DIR$/src/main/java" isTestSource="false" />
- <sourceFolder url="file://$MODULE_DIR$/src/test/java" isTestSource="true" />
- <sourceFolder url="file://$MODULE_DIR$/src/test/resources" type="java-test-resource" />
- <excludeFolder url="file://$MODULE_DIR$/${project.build.directory}/classes" />
- <excludeFolder url="file://$MODULE_DIR$/${project.build.directory}/test-classes" />
- <excludeFolder url="file://$MODULE_DIR$/target" />
- </content>
- <orderEntry type="inheritedJdk" />
- <orderEntry type="sourceFolder" forTests="false" />
- </component>
-</module>
\ No newline at end of file
[rocketmq-streams] 05/35: modify README.md
Posted by du...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
duhengforever pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/rocketmq-streams.git
commit f9e16807cc20190dd314d4de057b70ba0dbf67f1
Author: 赤玄 <yu...@alibaba-inc.com>
AuthorDate: Mon Aug 2 12:03:22 2021 +0800
modify README.md
---
README.md | 110 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 109 insertions(+), 1 deletion(-)
diff --git a/README.md b/README.md
index 49cd84d..678bd5a 100644
--- a/README.md
+++ b/README.md
@@ -1 +1,109 @@
-# rocketmq-streams
+# Rocketmq Streams
+
+
+## Features
+
+* 轻量级部署：可以单独部署，也支持集群部署
+* 多种类型的数据输入以及输出，source支持 rocketmq ， sink支持db, rocketmq 等
+
+## DataStream Example
+
+```java
+ import org.apache.rocketmq.streams.client.transform.DataStream;
+
+
+
+ DataStreamSource source=StreamBuilder.dataStream("namespace","pipeline");
+
+ source
+ .fromFile("/Users/junjie.cheng/text.txt",false)
+ .map(message->message)
+ .toPrint(1)
+ .start();
+```
+
+## Maven Repository
+
+```xml
+
+<dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams-clients</artifactId>
+ <version>2.0.0-SNAPSHOT</version>
+</dependency>
+```
+
+# Core API
+
+rocketmq-stream 实现了一系列高级的API，可以让用户很方便的编写流计算的程序，实现自己的业务需求；
+
+## StreamBuilder
+
+StreamBuilder 用于构建流任务的源； 内部包含```dataStream()```和```tableStream()```俩个方法，分别返回DataStreamSource和TableStreamSource俩个源；
+
++ [dataStream(nameSpaceName,pipelineName)]() 返回DataStreamSource实例，用于分段编程实现流计算任务；
++ [tableStream(nameSpaceName,pipelineName)]()返回TableStreamSource实例， 用于脚本编程实现流计算任务；
+
+## DataStream API
+
+### Source
+DataStreamSource 是分段式编程的源头类，用于对接各种数据源， 从各大消息队列中获取数据；
+
++ ```fromFile``` 从文件中读取数据， 该方法包含俩个参数
+ + ```filePath``` 文件路径，必填参数
+ + ```isJsonData``` 是否json数据， 非必填参数， 默认为```true```
+
+
++ ```fromRocketmq``` 从rocketmq中获取数据，包含四个参数
+ + ```topic``` rocketmq消息队列的topic名称，必填参数
+ + ```groupName``` 消费者组的名称，必填参数
+ + ```isJson``` 是否json格式，非必填参数
+ + ```tags``` rocketmq消费的tags值，用于过滤消息，非必填参数
+
+
++ ```from``` 自定义的数据源， 通过实现ISource接口实现自己的数据源
+
+### transform
+transform 允许在流计算过程中对输入源的数据进行修改，进行下一步的操作；DataStream API中包括```DataStream```,```JoinStream```, ```SplitStream```,```WindowStream```等多个transform类；
+
+#### DataStream
+DataStream实现了一系列常见的流计算算子
+
++ ```map``` 通过将源的每个记录传递给函数func来返回一个新的DataStream
++ ```flatmap``` 与map类似，一个输入项对应0个或者多个输出项
++ ```filter``` 只选择func返回true的源DStream的记录来返回一个新的DStream
++ ```forEach``` 对每个记录执行一次函数func， 返回一个新的DataStream
++ ```selectFields``` 对每个记录返回对应的字段值，返回一个新的DataStream
++ ```operate``` 对每个记录执行一次自定义的函数，返回一个新的DataStream
++ ```script``` 针对每个记录的字段执行一段脚本，返回新的字段，生成一个新的DataStream
++ ```toPrint``` 将结果在控制台打印，生成新的DataStreamAction实例
++ ```toFile``` 将结果保存为文件，生成一个新的DataStreamAction实例
++ ```toDB``` 将结果保存到数据库
++ ```toRocketmq``` 将结果输出到rocketmq
++ ```toSls``` 将结果输出到sls
++ ```to``` 将结果经过自定义的ISink接口输出到指定的存储
++ ```window``` 在窗口内进行相关的统计分析，一般会与```groupBy```连用， ```window()```用来定义窗口的大小， ```groupBy()```用来定义统计分析的主key，可以指定多个
+ + ```count``` 在窗口内计数
+ + ```min``` 获取窗口内统计值的最小值
+ + ```max``` 获取窗口内统计值得最大值
+ + ```avg``` 获取窗口内统计值的平均值
+ + ```sum``` 获取窗口内统计值的加和值
+ + ```reduce``` 在窗口内进行自定义的汇总运算
++ ```join``` 根据条件将将俩个流进行关联， 合并为一个大流进行相关的运算
++ ```union``` 将俩个流进行合并
++ ```split``` 将一个数据流按照标签进行拆分，分为不同的数据流供下游进行分析计算
++ ```with``` with算子用来指定计算过程中的相关策略，包括checkpoint的存储策略，state的存储策略等
+
+
+# Strategy
+策略机制主要用来控制计算引擎运行过程中的底层逻辑，如checkpoint，state的存储方式等，后续还会增加对窗口、双流join等的控制；所有的控制策略通过```with```算子传入，可以同时传入多个策略类型；
+
+```java
+ //指定checkpoint的存储策略
+ source
+ .fromRocketmq("TSG_META_INFO", "")
+ .map(message -> message + "--")
+ .toPrint(1)
+ .with(CheckpointStrategy.db("jdbc:mysql://XXXXX:3306/XXXXX", "", "", 0L))
+ .start();
+```
[rocketmq-streams] 10/35: modify the README.md
Posted by du...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
duhengforever pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/rocketmq-streams.git
commit 30cb39bdbda5ac76d2b350867619ae7d70212979
Author: 刈刀 <ju...@alibaba-inc.com>
AuthorDate: Mon Aug 2 12:12:29 2021 +0800
modify the README.md
---
README.md | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/README.md b/README.md
index 2dfb623..649319c 100644
--- a/README.md
+++ b/README.md
@@ -56,8 +56,7 @@ DataStreamSource 是分段式编程的源头类，用于对接各种数据源，
+ ```groupName``` 消费者组的名称，必填参数
+ ```isJson``` 是否json格式，非必填参数
+ ```tags``` rocketmq消费的tags值，用于过滤消息，非必填参数
-
-
+
+ ```from``` 自定义的数据源， 通过实现ISource接口实现自己的数据源
### transform
[rocketmq-streams] 03/35: add lease、dim and client module
Posted by du...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
duhengforever pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/rocketmq-streams.git
commit 8c20f7314b838d853de39c90d19fd2e509a44713
Merge: b757f8d 997f3df
Author: 刈刀 <ju...@alibaba-inc.com>
AuthorDate: Mon Aug 2 11:30:22 2021 +0800
add lease、dim and client module
README.md | 28 +++++++++++++---------------
1 file changed, 13 insertions(+), 15 deletions(-)
diff --cc README.md
index 5f3cde5,49cd84d..2dfb623
--- a/README.md
+++ b/README.md
@@@ -1,108 -1,1 +1,106 @@@
-# rocketmq-streams
+# Rocketmq Streams
+
+## Features
+
+* 轻量级部署：可以单独部署，也支持集群部署
+* 多种类型的数据输入以及输出，source支持 rocketmq ， sink支持db, rocketmq 等
+
+## DataStream Example
+
+```java
+import org.apache.rocketmq.streams.client.transform.DataStream;
+
+DataStreamSource source=StreamBuilder.dataStream("namespace","pipeline");
+
+ source
+ .fromFile("/Users/junjie.cheng/text.txt",false)
+ .map(message->message)
+ .toPrint(1)
+ .start();
+```
+
+## Maven Repository
+
+```xml
+
+<dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams-clients</artifactId>
+ <version>2.0.0-SNAPSHOT</version>
+</dependency>
+```
+
+# Core API
+
+rocketmq-stream 实现了一系列高级的API，可以让用户很方便的编写流计算的程序，实现自己的业务需求；
+
+## StreamBuilder
+
+StreamBuilder 用于构建流任务的源； 内部包含```dataStream()```和```tableStream()```俩个方法，分别返回DataStreamSource和TableStreamSource俩个源；
+
++ [dataStream(nameSpaceName,pipelineName)]() 返回DataStreamSource实例，用于分段编程实现流计算任务；
+++ [tableStream(nameSpaceName,pipelineName)]()返回TableStreamSource实例， 用于脚本编程实现流计算任务；
+
+## DataStream API
+
+### Source
-
+DataStreamSource 是分段式编程的源头类，用于对接各种数据源， 从各大消息队列中获取数据；
+
++ ```fromFile``` 从文件中读取数据， 该方法包含俩个参数
+ + ```filePath``` 文件路径，必填参数
+ + ```isJsonData``` 是否json数据， 非必填参数， 默认为```true```
+
+
++ ```fromRocketmq``` 从rocketmq中获取数据，包含四个参数
+ + ```topic``` rocketmq消息队列的topic名称，必填参数
+ + ```groupName``` 消费者组的名称，必填参数
+ + ```isJson``` 是否json格式，非必填参数
+ + ```tags``` rocketmq消费的tags值，用于过滤消息，非必填参数
+
+
++ ```from``` 自定义的数据源， 通过实现ISource接口实现自己的数据源
+
+### transform
-
+transform 允许在流计算过程中对输入源的数据进行修改，进行下一步的操作；DataStream API中包括```DataStream```,```JoinStream```, ```SplitStream```,```WindowStream```等多个transform类；
+
+#### DataStream
-
+DataStream实现了一系列常见的流计算算子
+
++ ```map``` 通过将源的每个记录传递给函数func来返回一个新的DataStream
++ ```flatmap``` 与map类似，一个输入项对应0个或者多个输出项
++ ```filter``` 只选择func返回true的源DStream的记录来返回一个新的DStream
++ ```forEach``` 对每个记录执行一次函数func， 返回一个新的DataStream
++ ```selectFields``` 对每个记录返回对应的字段值，返回一个新的DataStream
++ ```operate``` 对每个记录执行一次自定义的函数，返回一个新的DataStream
++ ```script``` 针对每个记录的字段执行一段脚本，返回新的字段，生成一个新的DataStream
++ ```toPrint``` 将结果在控制台打印，生成新的DataStreamAction实例
++ ```toFile``` 将结果保存为文件，生成一个新的DataStreamAction实例
++ ```toDB``` 将结果保存到数据库
++ ```toRocketmq``` 将结果输出到rocketmq
++ ```toSls``` 将结果输出到sls
++ ```to``` 将结果经过自定义的ISink接口输出到指定的存储
++ ```window``` 在窗口内进行相关的统计分析，一般会与```groupBy```连用， ```window()```用来定义窗口的大小， ```groupBy()```用来定义统计分析的主key，可以指定多个
- + ```count``` 在窗口内计数
- + ```min``` 获取窗口内统计值的最小值
- + ```max``` 获取窗口内统计值得最大值
- + ```avg``` 获取窗口内统计值的平均值
- + ```sum``` 获取窗口内统计值的加和值
- + ```reduce``` 在窗口内进行自定义的汇总运算
++ + ```count``` 在窗口内计数
++ + ```min``` 获取窗口内统计值的最小值
++ + ```max``` 获取窗口内统计值得最大值
++ + ```avg``` 获取窗口内统计值的平均值
++ + ```sum``` 获取窗口内统计值的加和值
++ + ```reduce``` 在窗口内进行自定义的汇总运算
++ ```join``` 根据条件将将俩个流进行关联， 合并为一个大流进行相关的运算
++ ```union``` 将俩个流进行合并
++ ```split``` 将一个数据流按照标签进行拆分，分为不同的数据流供下游进行分析计算
++ ```with``` with算子用来指定计算过程中的相关策略，包括checkpoint的存储策略，state的存储策略等
+
- # Strategy
+
++# Strategy
+策略机制主要用来控制计算引擎运行过程中的底层逻辑，如checkpoint，state的存储方式等，后续还会增加对窗口、双流join等的控制；所有的控制策略通过```with```算子传入，可以同时传入多个策略类型；
+
+```java
+//指定checkpoint的存储策略
+source
- .fromRocketmq("TSG_META_INFO","")
- .map(message->message+"--")
- .toPrint(1)
- .with(CheckpointStrategy.db("jdbc:mysql://XXXXX:3306/XXXXX","","",0L))
- .start();
++.fromRocketmq("TSG_META_INFO", "")
++.map(message -> message + "--")
++.toPrint(1)
++.with(CheckpointStrategy.db("jdbc:mysql://XXXXX:3306/XXXXX", "", "", 0L))
++.start();
+```
[rocketmq-streams] 13/35: add module db-operator、transport-minio
Posted by du...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
duhengforever pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/rocketmq-streams.git
commit 9c898ecf0c5301f1ae1d0c144f15f96d2151ba96
Author: muyang <li...@alibaba-inc.com>
AuthorDate: Mon Aug 2 12:21:28 2021 +0800
add module db-operator、transport-minio
---
rocketmq-streams-db-operator/pom.xml | 34 ++
.../rocketmq-streams-db-operator.iml | 16 +
.../streams/db/configuable/DBConfigureService.java | 282 ++++++++++++
.../DBSupportParentConfigureService.java | 37 ++
.../rocketmq/streams/db/driver/DriverBuilder.java | 111 +++++
.../rocketmq/streams/db/driver/IDriverBudiler.java | 36 ++
.../rocketmq/streams/db/driver/JDBCDriver.java | 277 ++++++++++++
.../db/driver/batchloader/BatchRowLoader.java | 179 ++++++++
.../db/driver/batchloader/IRowOperator.java | 33 ++
.../rocketmq/streams/db/driver/orm/ORMUtil.java | 490 +++++++++++++++++++++
.../rocketmq/streams/db/operator/SQLOperator.java | 178 ++++++++
.../org/apache/rocketmq/streams/db/Person.java | 110 +++++
.../DBSupportParentConfigureServiceTest.java | 74 ++++
.../streams/db/driver/orm/ORMUtilTest.java | 86 ++++
rocketmq-streams-transport-minio/pom.xml | 25 ++
.../rocketmq-streams-transport-minio.iml | 17 +
.../transport/minio/MinioFileTransport.java | 141 ++++++
.../yundun/dipper/configurable/DataTpyeTest.java | 70 +++
.../streams/configuable/model/DataTpyeTest.java | 68 +++
.../rocketmq/streams/configuable/model/Person.java | 97 ++++
.../streams/configurable/model/Person.java | 97 ++++
.../component/ConfigurableComponent.properties | 7 +
.../src/test/resources/log4j.xml | 20 +
.../src/test/resources/pro-function.txt | 11 +
.../src/test/resources/python_script.py | 22 +
25 files changed, 2518 insertions(+)
diff --git a/rocketmq-streams-db-operator/pom.xml b/rocketmq-streams-db-operator/pom.xml
new file mode 100755
index 0000000..9a9b17b
--- /dev/null
+++ b/rocketmq-streams-db-operator/pom.xml
@@ -0,0 +1,34 @@
+<?xml version="1.0" encoding="utf-8"?>
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams</artifactId>
+ <version>2.0.0-SNAPSHOT</version>
+ </parent>
+ <artifactId>rocketmq-streams-db-operator</artifactId>
+ <name>ROCKETMQ STREAMS :: db-operator</name>
+ <packaging>jar</packaging>
+ <dependencies>
+
+ <dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams-configurable</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.springframework</groupId>
+ <artifactId>spring-jdbc</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>mysql</groupId>
+ <artifactId>mysql-connector-java</artifactId>
+ </dependency>
+
+
+ </dependencies>
+</project>
diff --git a/rocketmq-streams-db-operator/rocketmq-streams-db-operator.iml b/rocketmq-streams-db-operator/rocketmq-streams-db-operator.iml
new file mode 100644
index 0000000..38ffb14
--- /dev/null
+++ b/rocketmq-streams-db-operator/rocketmq-streams-db-operator.iml
@@ -0,0 +1,16 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<module org.jetbrains.idea.maven.project.MavenProjectsManager.isMavenModule="true" type="JAVA_MODULE" version="4">
+ <component name="NewModuleRootManager" LANGUAGE_LEVEL="JDK_1_5">
+ <output url="file://$MODULE_DIR$/${project.build.directory}/classes" />
+ <output-test url="file://$MODULE_DIR$/${project.build.directory}/test-classes" />
+ <content url="file://$MODULE_DIR$">
+ <sourceFolder url="file://$MODULE_DIR$/src/main/java" isTestSource="false" />
+ <sourceFolder url="file://$MODULE_DIR$/src/test/java" isTestSource="true" />
+ <excludeFolder url="file://$MODULE_DIR$/${project.build.directory}/classes" />
+ <excludeFolder url="file://$MODULE_DIR$/${project.build.directory}/test-classes" />
+ <excludeFolder url="file://$MODULE_DIR$/target" />
+ </content>
+ <orderEntry type="inheritedJdk" />
+ <orderEntry type="sourceFolder" forTests="false" />
+ </component>
+</module>
\ No newline at end of file
diff --git a/rocketmq-streams-db-operator/src/main/java/org/apache/rocketmq/streams/db/configuable/DBConfigureService.java b/rocketmq-streams-db-operator/src/main/java/org/apache/rocketmq/streams/db/configuable/DBConfigureService.java
new file mode 100644
index 0000000..ef319eb
--- /dev/null
+++ b/rocketmq-streams-db-operator/src/main/java/org/apache/rocketmq/streams/db/configuable/DBConfigureService.java
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.db.configuable;
+
+import com.alibaba.fastjson.JSONObject;
+import org.apache.rocketmq.streams.common.component.AbstractComponent;
+import org.apache.rocketmq.streams.common.configurable.AbstractConfigurable;
+import org.apache.rocketmq.streams.common.configurable.IConfigurable;
+import org.apache.rocketmq.streams.configurable.service.AbstractConfigurableService;
+import org.apache.rocketmq.streams.configurable.model.Configure;
+
+import org.apache.rocketmq.streams.common.configure.ConfigureFileKey;
+import org.apache.rocketmq.streams.common.interfaces.IPropertyEnable;
+import org.apache.rocketmq.streams.common.utils.AESUtil;
+import org.apache.rocketmq.streams.common.utils.MapKeyUtil;
+import org.apache.rocketmq.streams.common.utils.SQLUtil;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.rocketmq.streams.db.driver.JDBCDriver;
+import org.apache.rocketmq.streams.db.driver.DriverBuilder;
+
+import java.util.*;
+
+/**
+ * Configuable对象存储在db中，是生成环境常用的一种模式 数据库参数可以配置在配置文件中，ConfiguableComponent在启动时，会把参数封装在Properties中，调用DBConfigureService(Properties properties) 构造方法完成实例创建
+ */
+
+public class DBConfigureService extends AbstractConfigurableService implements IPropertyEnable {
+
+ private static final Log LOG = LogFactory.getLog(DBConfigureService.class);
+ private String jdbcdriver;
+ private String url;
+ private String userName;
+ private String password;
+ private String tableName = "dipper_configure";
+ @Deprecated
+ private boolean isCompatibilityOldRuleEngine = false;//兼容老规则引擎使用，正常场景不需要理会
+
+ public DBConfigureService(String jdbcdriver, String url, String userName, String password) {
+ this(jdbcdriver, url, userName, password, null);
+ }
+
+ public DBConfigureService(String jdbcdriver, String url, String userName, String password, String tableName) {
+ this.url = url;
+ this.jdbcdriver = jdbcdriver;
+ this.userName = userName;
+ this.password = password;
+ this.tableName = tableName;
+ LOG.info("DBConfigureService resource ,the info is: driver:" + this.jdbcdriver + ",url:" + this.url
+ + ",username:" + userName + ",password:" + password);
+ regJdbcDriver(jdbcdriver);
+ }
+
+ public DBConfigureService() {
+ }
+
+ /**
+ * @param properties
+ */
+ public DBConfigureService(Properties properties) {
+ super(properties);
+ initProperty(properties);
+ }
+
+ @Override
+ protected GetConfigureResult loadConfigurable(String namespace) {
+ GetConfigureResult result = new GetConfigureResult();
+ try {
+ List<Configure> configures = selectOpening(namespace);
+ List<IConfigurable> configurables = convert(configures);
+ result.setConfigurables(configurables);
+ result.setQuerySuccess(true);// 该字段标示查询是否成功，若不成功则不会更新配置
+ } catch (Exception e) {
+ result.setQuerySuccess(false);
+ LOG.error("load configurable error ", e);
+ }
+ return result;
+ }
+
+ protected List<Configure> selectOpening(String namespace) {
+ return queryConfigureByNamespace(namespace);
+ }
+
+ protected List<Configure> queryConfigureByNamespace(String... namespaces) {
+ return queryConfigureByNamespaceInner(null, namespaces);
+ }
+
+ protected List<Configure> queryConfigureByNamespaceInner(String type, String... namespaces) {
+ JDBCDriver resource = createResouce();
+ try {
+ String namespace = "namespace";
+ if (isCompatibilityOldRuleEngine && AbstractComponent.JDBC_COMPATIBILITY_RULEENGINE_TABLE_NAME.equals(tableName)) {
+ namespace = "name_space";
+ }
+ String sql = "SELECT * FROM `" + tableName + "` WHERE " + namespace + " in (" + SQLUtil.createInSql(namespaces) + ") and status =1";
+ if (StringUtil.isNotEmpty(type)) {
+ sql = sql + " and type='" + type + "'";
+ }
+ JSONObject jsonObject = new JSONObject();
+ jsonObject.put("namespace", MapKeyUtil.createKeyBySign(",", namespaces));
+ sql = SQLUtil.parseIbatisSQL(jsonObject, sql);
+ // String builder = "SELECT * FROM `" + tableName + "` WHERE namespace ='" + namespace + "' and status =1";
+ List<Map<String, Object>> result = resource.queryForList(sql);
+ if (result == null) {
+ return new ArrayList<Configure>();
+ }
+ // LOG.info("load configurable's count is " + result.size());
+ return convert2Configure(result);
+ } finally {
+ if (resource != null) {
+ resource.destroy();
+ }
+ }
+ }
+
+ @Override
+ public List<IConfigurable> queryConfiguableByNamespace(String... namespaces) {
+ List<Configure> configures = queryConfigureByNamespace(namespaces);
+ List<IConfigurable> configurables = convert(configures);
+ return configurables;
+ }
+
+ public static void main(String[] args) {
+ String[] namespaces = new String[] {"rule1", null};
+ String sql = "SELECT * FROM `dipper_configure` WHERE namespace in (" + SQLUtil.createInSql(namespaces) + ") and status =1";
+ JSONObject jsonObject = new JSONObject();
+ jsonObject.put("namespace", MapKeyUtil.createKeyBySign(",", namespaces));
+ sql = SQLUtil.parseIbatisSQL(jsonObject, sql);
+ System.out.println(sql);
+ }
+
+ protected void saveOrUpdate(IConfigurable configure) {
+ JDBCDriver jdbcDataSource = createResouce();
+ String sql = AbstractConfigurable.createSQL(configure, this.tableName);
+ try {
+ jdbcDataSource.executeInsert(sql);
+ } catch (Exception e) {
+ LOG.error("DBConfigureService saveOrUpdate error,sqlnode:" + sql);
+ throw new RuntimeException(e);
+ } finally {
+ if (jdbcDataSource != null) {
+ jdbcDataSource.destroy();
+ }
+ }
+ }
+
+ protected List<Configure> convert2Configure(List<Map<String, Object>> rows) {
+ List<Configure> configures = new ArrayList<Configure>();
+ for (Map<String, Object> row : rows) {
+ Configure configure = new Configure();
+ Long id = getColumnValue(row, "id");
+ configure.setId(id);
+ Date create = getColumnValue(row, "gmt_create");
+ configure.setGmtCreate(create);
+ Date modify = getColumnValue(row, "gmt_modified");
+ configure.setGmtModified(modify);
+ String namespace = getColumnValue(row, "namespace");
+ if (StringUtil.isEmpty(namespace)) {
+ namespace = getColumnValue(row, "name_space");
+ }
+ configure.setNameSpace(namespace);
+ String type = getColumnValue(row, "type");
+ configure.setType(type);
+ String name = getColumnValue(row, "name");
+ configure.setName(name);
+ String jsonValue = getColumnValue(row, "json_value");
+ try {
+ jsonValue = AESUtil.aesDecrypt(jsonValue, ConfigureFileKey.SECRECY);
+ } catch (Exception e) {
+ LOG.error("can't decrypt the value, reason:\t" + e.getCause());
+ throw new RuntimeException(e);
+ }
+ configure.setJsonValue(jsonValue);
+ configures.add(configure);
+ }
+ return configures;
+ }
+
+ @SuppressWarnings("unchecked")
+ protected <T> T getColumnValue(Map<String, Object> row, String columnName) {
+ Object value = row.get(columnName);
+ if (value == null) {
+ return null;
+ }
+ if (java.math.BigInteger.class.isInstance(value)) {
+ return (T)Long.valueOf(value.toString());
+ }
+ return (T)value;
+
+ }
+
+ protected JDBCDriver createResouce() {
+ JDBCDriver resource = DriverBuilder.createDriver(this.jdbcdriver, this.url, this.userName, this.password);
+ return resource;
+ }
+
+ public void setJdbcdriver(String jdbcdriver) {
+ this.jdbcdriver = jdbcdriver;
+ }
+
+ public void setUrl(String url) {
+ this.url = url;
+ }
+
+ public void setUserName(String userName) {
+ this.userName = userName;
+ }
+
+ public void setPassword(String password) {
+ this.password = password;
+ }
+
+ private void regJdbcDriver(String jdbcdriver) {
+ try {
+ if (StringUtil.isEmpty(jdbcdriver)) {
+ jdbcdriver = AbstractComponent.DEFAULT_JDBC_DRIVER;
+ }
+ Class.forName(jdbcdriver);
+ } catch (ClassNotFoundException e) {
+ LOG.error("DBConfigureService regJdbcDriver ClassNotFoundException error", e);
+ } catch (Exception e) {
+ LOG.error("DBConfigureService regJdbcDriver error", e);
+ }
+ }
+
+ @Override
+ public void initProperty(Properties properties) {
+ this.jdbcdriver = properties.getProperty(AbstractComponent.JDBC_DRIVER);
+ regJdbcDriver(jdbcdriver);
+ this.url = properties.getProperty(AbstractComponent.JDBC_URL);
+ this.userName = properties.getProperty(AbstractComponent.JDBC_USERNAME);
+ this.password = properties.getProperty(AbstractComponent.JDBC_PASSWORD);
+ String tableName = properties.getProperty(AbstractComponent.JDBC_TABLE_NAME);
+ String isCompatibilityOldRuleEngine = properties.getProperty(AbstractComponent.JDBC_COMPATIBILITY_OLD_RULEENGINE);
+ if (StringUtil.isNotEmpty(isCompatibilityOldRuleEngine)) {
+ this.isCompatibilityOldRuleEngine = true;
+ }
+ if (StringUtil.isNotEmpty(tableName)) {
+ this.tableName = tableName;
+ }
+ LOG.info(
+ "Properties resource ,the info is: driver:" + this.jdbcdriver + ",url:" + this.url + ",username:" + userName
+ + ",password:" + password);
+ }
+
+ @Override
+ protected void insertConfigurable(IConfigurable configurable) {
+ saveOrUpdate(configurable);
+ }
+
+ @Override
+ protected void updateConfigurable(IConfigurable configurable) {
+ saveOrUpdate(configurable);
+ }
+
+ @Override
+ public <T extends IConfigurable> List<T> loadConfigurableFromStorage(String type) {
+
+ List<Configure> configures = queryConfigureByNamespaceInner(type, namespace);
+ List<IConfigurable> configurables = convert(configures);
+ List<T> result = new ArrayList<>();
+ for (IConfigurable configurable : configurables) {
+ result.add((T)configurable);
+ }
+ return result;
+ }
+}
diff --git a/rocketmq-streams-db-operator/src/main/java/org/apache/rocketmq/streams/db/configuable/DBSupportParentConfigureService.java b/rocketmq-streams-db-operator/src/main/java/org/apache/rocketmq/streams/db/configuable/DBSupportParentConfigureService.java
new file mode 100644
index 0000000..77b82e0
--- /dev/null
+++ b/rocketmq-streams-db-operator/src/main/java/org/apache/rocketmq/streams/db/configuable/DBSupportParentConfigureService.java
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.db.configuable;
+
+import org.apache.rocketmq.streams.common.configurable.IConfigurableService;
+import org.apache.rocketmq.streams.configurable.service.ConfigurableServcieType;
+import org.apache.rocketmq.streams.common.model.ServiceName;
+import com.google.auto.service.AutoService;
+import org.apache.rocketmq.streams.configurable.service.AbstractSupportParentConfigureService;
+
+import java.util.Properties;
+
+@AutoService(IConfigurableService.class)
+@ServiceName(ConfigurableServcieType.DEFAULT_SERVICE_NAME)
+public class DBSupportParentConfigureService extends AbstractSupportParentConfigureService {
+
+ @Override
+ protected void initBeforeInitConfigurable(Properties property) {
+ this.parentConfigureService = new DBConfigureService(property);
+ this.configureService = new DBConfigureService(property);
+
+ }
+}
diff --git a/rocketmq-streams-db-operator/src/main/java/org/apache/rocketmq/streams/db/driver/DriverBuilder.java b/rocketmq-streams-db-operator/src/main/java/org/apache/rocketmq/streams/db/driver/DriverBuilder.java
new file mode 100644
index 0000000..c0e9f53
--- /dev/null
+++ b/rocketmq-streams-db-operator/src/main/java/org/apache/rocketmq/streams/db/driver/DriverBuilder.java
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.db.driver;
+
+import java.lang.reflect.Constructor;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.apache.rocketmq.streams.common.component.AbstractComponent;
+import org.apache.rocketmq.streams.common.component.ComponentCreator;
+import org.apache.rocketmq.streams.common.utils.ReflectUtil;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+/**
+ * 创建JDBCDriver，如果没有
+ */
+public class DriverBuilder {
+
+ private static final Log LOG = LogFactory.getLog(DriverBuilder.class);
+
+ public static final String DEFALUT_JDBC_DRIVER = "com.mysql.jdbc.Driver";
+
+ private static final Map<String, JDBCDriver> dataSourceMap = new ConcurrentHashMap<>();
+
+ private static AtomicInteger count = new AtomicInteger(0);
+
+ /**
+ * 使用ConfiguableComponent在属性文件配置的jdbc信息，dipper默认都是使用这个数据库连接 如果需要连接其他库，需要使用带参数的createDriver
+ *
+ * @return
+ */
+ public static JDBCDriver createDriver() {
+ String driver = ComponentCreator.getProperties().getProperty(AbstractComponent.JDBC_DRIVER);
+ String url = ComponentCreator.getProperties().getProperty(AbstractComponent.JDBC_URL);
+ String userName = ComponentCreator.getProperties().getProperty(AbstractComponent.JDBC_USERNAME);
+ String password = ComponentCreator.getProperties().getProperty(AbstractComponent.JDBC_PASSWORD);
+ return createDriver(driver, url, userName, password);
+ }
+
+ /**
+ * 根据数据库连接信息创建连接，并返回JDBCDriver
+ *
+ * @param driver 数据库驱动，如果为null，默认为mysql
+ * @param url 数据库连接url
+ * @param userName 用户名
+ * @param password 密码
+ * @return JDBCDriver
+ */
+ public static JDBCDriver createDriver(String driver, final String url, final String userName,
+ final String password) {
+ if (StringUtil.isEmpty(driver)) {
+ driver = DEFALUT_JDBC_DRIVER;
+ }
+ String className = ComponentCreator.getDBProxyClassName();
+ if (StringUtil.isNotEmpty(className)) {
+ Class clazz = ReflectUtil.forClass(className);
+ try {
+ Constructor constructor = clazz.getConstructor(
+ new Class[] {String.class, String.class, String.class, String.class});
+ JDBCDriver abstractDBDataSource = (JDBCDriver)constructor.newInstance(url, userName, password,
+ driver);
+ abstractDBDataSource.init();
+ return abstractDBDataSource;
+ } catch (Exception e) {
+ throw new RuntimeException(e.getMessage(), e);
+ }
+ }
+
+ final String jdbcdriver = driver;
+ ReflectUtil.forClass(jdbcdriver);
+ JDBCDriver resource = new JDBCDriver();
+ LOG.debug("jdbcdriver=" + jdbcdriver + ",url=" + url);
+ resource.setJdbcDriver(jdbcdriver);
+ resource.setUrl(url);
+ resource.setUserName(userName);
+ resource.setPassword(password);
+ resource.init();
+ return resource;
+ }
+
+ /**
+ * 生成拼接字符串
+ *
+ * @param url
+ * @param userName
+ * @param password
+ * @return
+ */
+ private static String genereateKey(String url, String userName, String password) {
+ return url + "_" + userName + "_" + password;
+ }
+
+}
+
diff --git a/rocketmq-streams-db-operator/src/main/java/org/apache/rocketmq/streams/db/driver/IDriverBudiler.java b/rocketmq-streams-db-operator/src/main/java/org/apache/rocketmq/streams/db/driver/IDriverBudiler.java
new file mode 100644
index 0000000..6be77eb
--- /dev/null
+++ b/rocketmq-streams-db-operator/src/main/java/org/apache/rocketmq/streams/db/driver/IDriverBudiler.java
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.db.driver;
+
+import org.apache.rocketmq.streams.common.dboperator.IDBDriver;
+
+/**
+ * 返回操作数据库的driver对象，并且提供方法，判断driver是否有效，以及销毁的方法
+ */
+public interface IDriverBudiler {
+
+ /**
+ * 和dipper系统同数据源
+ *
+ * @return
+ */
+ IDBDriver createDBDriver();
+
+ boolean isValidate();
+
+ void destroy();
+}
diff --git a/rocketmq-streams-db-operator/src/main/java/org/apache/rocketmq/streams/db/driver/JDBCDriver.java b/rocketmq-streams-db-operator/src/main/java/org/apache/rocketmq/streams/db/driver/JDBCDriver.java
new file mode 100644
index 0000000..356bce6
--- /dev/null
+++ b/rocketmq-streams-db-operator/src/main/java/org/apache/rocketmq/streams/db/driver/JDBCDriver.java
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.db.driver;
+
+import org.apache.rocketmq.streams.common.channel.sink.ISink;
+import org.apache.rocketmq.streams.common.configurable.BasedConfigurable;
+import org.apache.rocketmq.streams.common.configurable.annotation.ENVDependence;
+import org.apache.rocketmq.streams.common.dboperator.IDBDriver;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+import org.springframework.jdbc.core.JdbcTemplate;
+import org.springframework.jdbc.datasource.SingleConnectionDataSource;
+import org.springframework.jdbc.support.GeneratedKeyHolder;
+import org.springframework.jdbc.support.KeyHolder;
+
+import java.sql.PreparedStatement;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * 数据库常用操作的封装，核心实现的接口是IJdbcTemplate 这个对象实现了IConfigurable接口，可以序列化存储和网络传输 数据库参数，可以配置成名字，实际值在配置文件配置
+ * <p>
+ */
+public class JDBCDriver extends BasedConfigurable implements IDriverBudiler, IDBDriver {
+ private String jdbcDriver = DriverBuilder.DEFALUT_JDBC_DRIVER;
+ @ENVDependence
+ protected String url;
+ @ENVDependence
+ protected String userName;
+ @ENVDependence
+ protected String password;
+
+ protected transient javax.sql.DataSource dataSource;
+ private transient IDBDriver dbDriver = null;
+
+ public JDBCDriver(String url, String userName, String password,
+ String driver) {
+ setType(ISink.TYPE);
+ this.url = url;
+ this.userName = userName;
+ this.password = password;
+ if (StringUtil.isNotEmpty(driver)) {
+ this.jdbcDriver = driver;
+ }
+ }
+
+ public JDBCDriver() {
+ setType(ISink.TYPE);
+ }
+
+ protected IDBDriver createOrGetDriver() {
+ if (dbDriver == null) {
+ synchronized (this) {
+ if (dbDriver == null) {
+ dbDriver = createDBDriver();
+ if (dataSource == null) {
+ dataSource = createDBDataSource();
+ }
+ }
+ }
+ }
+ return dbDriver;
+ }
+
+ @Override
+ public IDBDriver createDBDriver() {
+ javax.sql.DataSource dataSource = createDBDataSource();
+ return new IDBDriver() {
+ private final JdbcTemplate jdbcTemplate = new JdbcTemplate(dataSource);
+
+ @Override
+ public int update(String sql) {
+ return jdbcTemplate.update(sql);
+ }
+
+ @Override
+ public void execute(String sql) {
+ jdbcTemplate.execute(sql);
+ }
+
+ @Override
+ public List<Map<String, Object>> queryForList(String sql) {
+ return jdbcTemplate.queryForList(sql);
+ }
+
+ @Override
+ public Map<String, Object> queryOneRow(String sql) {
+ return jdbcTemplate.queryForMap(sql);
+ }
+
+ @Override
+ public long executeInsert(String sql) {
+ try {
+ KeyHolder keyHolder = new GeneratedKeyHolder();
+ jdbcTemplate.update(con -> con.prepareStatement(sql, PreparedStatement.RETURN_GENERATED_KEYS), keyHolder);
+ if (keyHolder.getKeyList() == null || keyHolder.getKeyList().size() > 1 || keyHolder.getKey() == null) {
+ return 0;
+ }
+ return keyHolder.getKey().longValue();
+ } catch (Exception e) {
+ String errorMsg = "execute builder error ,the builder is " + sql + ". the error msg is " + e.getMessage();
+ throw new RuntimeException(errorMsg, e);
+ }
+ }
+
+ @Override
+ public void executSqls(String... sqls) {
+ jdbcTemplate.batchUpdate(sqls);
+ }
+
+ @Override
+ public void executSqls(Collection<String> sqlCollection) {
+ if (sqlCollection == null || sqlCollection.size() == 0) {
+ return;
+ }
+ String[] sqls = new String[sqlCollection.size()];
+ int i = 0;
+ Iterator<String> it = sqlCollection.iterator();
+ while (it.hasNext()) {
+ String sql = it.next();
+ sqls[i] = sql;
+ i++;
+ }
+ executSqls(sqls);
+ }
+
+ /**
+ * 分批获取数据，最终获取全量数据
+ * @param sql 可执行的SQL
+ * @return 结果数据
+ */
+ @Override
+ public List<Map<String, Object>> batchQueryBySql(String sql, int batchSize) {
+ List<Map<String, Object>> rows = new ArrayList<>();
+ int startBatch;
+ String baseSql = sql;
+ if (sql.contains(";")) {
+ baseSql = sql.substring(0, sql.indexOf(";"));
+ }
+ String batchSQL = baseSql + " limit 0," + batchSize;
+ List<Map<String, Object>> batchResult = queryForList(batchSQL);
+ int index = 1;
+ while (batchResult.size() >= batchSize) {
+ rows.addAll(batchResult);
+ startBatch = batchSize * index;
+ batchSQL = baseSql + " limit " + startBatch + "," + batchSize;
+ batchResult = queryForList(batchSQL);
+ index++;
+ }
+ rows.addAll(batchResult);
+
+ return rows;
+ }
+ };
+ }
+
+ protected javax.sql.DataSource createDBDataSource() {
+
+ SingleConnectionDataSource dataSource = new SingleConnectionDataSource(url, userName, password, true);
+
+ dataSource.setDriverClassName(jdbcDriver);// add by 林行0221
+ // 专有云落地运维中心联调时发现独立打包时必须加这句话，否则会报找不到驱动。(MYSQL驱动包虽然已经打进去了但还要在这里显示指定)
+ dataSource.setSuppressClose(true);
+ this.dataSource = dataSource;
+ return dataSource;
+ }
+
+ @Override
+ public boolean isValidate() {
+ try {
+ if (dataSource == null) {
+ dataSource = createDBDataSource();
+ }
+ dataSource.getConnection();
+ } catch (SQLException e) {
+ return false;
+ }
+ return true;
+ }
+
+ @Override
+ public void destroy() {
+ if (dataSource instanceof SingleConnectionDataSource) {
+ SingleConnectionDataSource data = (SingleConnectionDataSource)dataSource;
+ data.destroy();
+ }
+ }
+
+ public String getJdbcDriver() {
+ return jdbcDriver;
+ }
+
+ public void setJdbcDriver(String jdbcDriver) {
+ this.jdbcDriver = jdbcDriver;
+ }
+
+ public String getUrl() {
+ return url;
+ }
+
+ public void setUrl(String url) {
+ this.url = url;
+ }
+
+ public String getUserName() {
+ return userName;
+ }
+
+ public void setUserName(String userName) {
+ this.userName = userName;
+ }
+
+ public String getPassword() {
+ return password;
+ }
+
+ public void setPassword(String password) {
+ this.password = password;
+ }
+
+ @Override
+ public int update(String sql) {
+ return createOrGetDriver().update(sql);
+ }
+
+ @Override
+ public void execute(String sql) {
+ createOrGetDriver().execute(sql);
+ }
+
+ @Override
+ public List<Map<String, Object>> queryForList(String sql) {
+ return createOrGetDriver().queryForList(sql);
+ }
+
+ @Override
+ public Map<String, Object> queryOneRow(String sql) {
+ return createOrGetDriver().queryOneRow(sql);
+ }
+
+ @Override
+ public long executeInsert(String sql) {
+ return createOrGetDriver().executeInsert(sql);
+ }
+
+ @Override
+ public void executSqls(String... sqls) {
+ createOrGetDriver().executSqls(sqls);
+ }
+
+ @Override
+ public void executSqls(Collection<String> sqls) {
+ createOrGetDriver().executSqls(sqls);
+ }
+
+ @Override
+ public List<Map<String, Object>> batchQueryBySql(String sql, int batchSize) {
+ return createOrGetDriver().batchQueryBySql(sql, batchSize);
+ }
+}
diff --git a/rocketmq-streams-db-operator/src/main/java/org/apache/rocketmq/streams/db/driver/batchloader/BatchRowLoader.java b/rocketmq-streams-db-operator/src/main/java/org/apache/rocketmq/streams/db/driver/batchloader/BatchRowLoader.java
new file mode 100644
index 0000000..3e4548c
--- /dev/null
+++ b/rocketmq-streams-db-operator/src/main/java/org/apache/rocketmq/streams/db/driver/batchloader/BatchRowLoader.java
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.db.driver.batchloader;
+
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import com.alibaba.fastjson.JSONObject;
+
+import org.apache.rocketmq.streams.common.cache.compress.impl.IntValueKV;
+import org.apache.rocketmq.streams.common.utils.SQLUtil;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.rocketmq.streams.db.driver.JDBCDriver;
+import org.apache.rocketmq.streams.db.driver.DriverBuilder;
+
+/**
+ * 多线程批量加载数据，每加载一批数据后，通过IRowOperator回调接口处理数据 需要有递增的字段，这个字段有索引，不重复，如id字段
+ */
+public class BatchRowLoader {
+ private static final Log LOG = LogFactory.getLog(BatchRowLoader.class);
+ protected static final int MAX_LINE = 5000;//每个批次最大行数，根据这个值划分并行任务
+ protected static ExecutorService executorService = null;
+ protected String idFieldName;//配置字段名称，这个字段的值是数字的，且是递增的
+ protected String sql;//查询的sql语句，类似select * from table where idFieldName>#{idFieldName=0} order by idFieldName.不要加limit，系统会自动添加
+ protected int batchSize = 1000;//每批从数据库加载的数据量
+ protected IRowOperator dataRowProcessor;//加载的数据由这个回调接口处理
+ private JDBCDriver jdbcDriver;
+
+ public BatchRowLoader(String idFieldName, String sql, IRowOperator dataRowProcessor) {
+ this.idFieldName = idFieldName;
+ this.sql = sql;
+ this.dataRowProcessor = dataRowProcessor;
+ this.jdbcDriver = DriverBuilder.createDriver();
+ executorService = new ThreadPoolExecutor(20, 20,
+ 0L, TimeUnit.MILLISECONDS,
+ new LinkedBlockingQueue<Runnable>(1000));
+ }
+
+ public void startLoadData() {
+ try {
+ String statisticalSQL = sql;
+ int startIndex = sql.toLowerCase().indexOf("from");
+ statisticalSQL = "select count(1) as c, min(" + idFieldName + ") as min, max(" + idFieldName + ") as max "
+ + sql.substring(startIndex);
+ List<Map<String, Object>> rows = jdbcDriver.queryForList(statisticalSQL);
+ Map<String, Object> row = rows.get(0);
+ int count = Integer.valueOf(row.get("c").toString());
+ if (count == 0) {
+ LOG.warn("there is no data during execute sql: " + statisticalSQL);
+ return;
+ }
+
+ IntValueKV intValueKV = new IntValueKV(count);
+ //int maxBatch=count/maxSyncCount;//每1w条数据，一个并发。如果数据量比较大，为了提高性能，并行执行
+
+ long min = Long.valueOf(row.get("min").toString());
+ long max = Long.valueOf(row.get("max").toString());
+ int maxSyncCount = count / MAX_LINE + 1;
+ long step = (max - min + 1) / maxSyncCount;
+ CountDownLatch countDownLatch = new CountDownLatch(maxSyncCount + 1);
+ AtomicInteger finishedCount = new AtomicInteger(0);
+ String taskSQL = null;
+ if (sql.indexOf(" where ") != -1) {
+ taskSQL = sql + " and " + idFieldName + ">#{startIndex} and " + idFieldName + "<=#{endIndex} order by "
+ + idFieldName + " limit " + batchSize;
+ } else {
+ taskSQL = sql + " where " + idFieldName + ">#{startIndex} and " + idFieldName
+ + "<=#{endIndex} order by " + idFieldName + " limit " + batchSize;
+ }
+
+ int i = 0;
+ for (; i < maxSyncCount; i++) {
+ FetchDataTask
+ fetchDataTask = new FetchDataTask(taskSQL, (min - 1) + step * i,
+ (min - 1) + step * (i + 1), countDownLatch, finishedCount, jdbcDriver, count);
+ executorService.execute(fetchDataTask);
+ }
+ FetchDataTask
+ fetchDataTask = new FetchDataTask(taskSQL, (min - 1) + step * i, (min - 1) + step * (i + 1),
+ countDownLatch, finishedCount, jdbcDriver, count);
+ executorService.execute(fetchDataTask);
+
+ countDownLatch.await();
+
+ LOG.info(getClass().getSimpleName() + " load data finish, load data line size is " + count);
+ } catch (Exception e) {
+ LOG.error("failed loading data batch!", e);
+ } finally {
+ jdbcDriver.destroy();
+ }
+ }
+
+ protected class FetchDataTask implements Runnable {
+ long startIndex;
+ long endIndex;
+ String sql;
+ CountDownLatch countDownLatch;
+ JDBCDriver resource;
+ AtomicInteger finishedCount;//完成了多少条
+ int totalSize;//一共有多少条数据
+
+ public FetchDataTask(String sql, long startIndex, long endIndex, CountDownLatch countDownLatch,
+ AtomicInteger finishedCount, JDBCDriver resource, int totalSize) {
+ this.startIndex = startIndex;
+ this.endIndex = endIndex;
+ this.countDownLatch = countDownLatch;
+ this.sql = sql;
+ this.finishedCount = finishedCount;
+ this.resource = resource;
+ this.totalSize = totalSize;
+ }
+
+ @Override
+ public void run() {
+ long currentIndex = startIndex;
+ JSONObject msg = new JSONObject();
+ msg.put("endIndex", endIndex);
+ while (true) {
+ try {
+
+ msg.put("startIndex", currentIndex);
+
+ String sql = SQLUtil.parseIbatisSQL(msg, this.sql);
+ List<Map<String, Object>> rows = resource.queryForList(sql);
+ if (rows == null || rows.size() == 0) {
+ break;
+ }
+ currentIndex = Long.valueOf(rows.get(rows.size() - 1).get(idFieldName).toString());
+
+ int size = rows.size();
+ int count = finishedCount.addAndGet(size);
+ double progress = (double)count / (double)totalSize;
+ progress = progress * 100;
+ System.out.println(" finished count is " + count + " the total count is " + totalSize + ", the progress is " + String.format("%.2f", progress) + "%");
+ if (size < batchSize) {
+ if (size > 0) {
+
+ doProcess(rows);
+ }
+ break;
+ }
+ doProcess(rows);
+ } catch (Exception e) {
+ throw new RuntimeException("put data error ", e);
+ }
+ }
+
+ countDownLatch.countDown();
+ }
+ }
+
+ private void doProcess(List<Map<String, Object>> rows) {
+ for (Map<String, Object> row : rows) {
+ dataRowProcessor.doProcess(row);
+ }
+ }
+}
+
diff --git a/rocketmq-streams-db-operator/src/main/java/org/apache/rocketmq/streams/db/driver/batchloader/IRowOperator.java b/rocketmq-streams-db-operator/src/main/java/org/apache/rocketmq/streams/db/driver/batchloader/IRowOperator.java
new file mode 100644
index 0000000..f67393f
--- /dev/null
+++ b/rocketmq-streams-db-operator/src/main/java/org/apache/rocketmq/streams/db/driver/batchloader/IRowOperator.java
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.db.driver.batchloader;
+
+import java.util.Map;
+
+/**
+ * 操作一行数据
+ */
+public interface IRowOperator {
+
+ /**
+ * 处理一行数据
+ *
+ * @param row
+ */
+ void doProcess(Map<String, Object> row);
+
+}
diff --git a/rocketmq-streams-db-operator/src/main/java/org/apache/rocketmq/streams/db/driver/orm/ORMUtil.java b/rocketmq-streams-db-operator/src/main/java/org/apache/rocketmq/streams/db/driver/orm/ORMUtil.java
new file mode 100644
index 0000000..20529b0
--- /dev/null
+++ b/rocketmq-streams-db-operator/src/main/java/org/apache/rocketmq/streams/db/driver/orm/ORMUtil.java
@@ -0,0 +1,490 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.db.driver.orm;
+
+import java.lang.reflect.Field;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.rocketmq.streams.common.component.ComponentCreator;
+import org.apache.rocketmq.streams.common.configure.ConfigureFileKey;
+import org.apache.rocketmq.streams.common.model.Entity;
+import org.apache.rocketmq.streams.common.configurable.IFieldProcessor;
+import org.apache.rocketmq.streams.common.datatype.DataType;
+import org.apache.rocketmq.streams.common.metadata.MetaData;
+import org.apache.rocketmq.streams.common.utils.CollectionUtil;
+import org.apache.rocketmq.streams.common.utils.DataTypeUtil;
+import org.apache.rocketmq.streams.common.utils.DateUtil;
+import org.apache.rocketmq.streams.common.utils.ReflectUtil;
+import org.apache.rocketmq.streams.common.utils.SQLUtil;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.rocketmq.streams.db.driver.DriverBuilder;
+import org.apache.rocketmq.streams.db.driver.JDBCDriver;
+
+/**
+ * 轻量级的orm框架，如果pojo和table 符合驼峰的命名和下划线的命名规范，可以自动实现对象的orm
+ */
+public class ORMUtil {
+ private static final Log LOG = LogFactory.getLog(ORMUtil.class);
+
+ public static <T> T queryForObject(String sql, Object paras, Class<T> convertClass) {
+ return queryForObject(sql, paras, convertClass, null, null, null);
+ }
+
+ /**
+ * 通过sql查询一个唯一的对象出来，返回数据多于一条会报错
+ *
+ * @param sql 查询语句
+ * @param paras 如果有变参，这里面有变参的参数，可以是map，json或对象，只要key名或字段名和sql的参数名相同即可
+ * @param convertClass，如果有变参，这里面有变参的参数，可以是map，json或对象，只要key名或字段名和sql的参数名相同即可
+ * @param url 数据库连接url
+ * @param userName 用户名
+ * @param password 密码
+ * @param <T>
+ * @return 转换后的对象
+ */
+ public static <T> T queryForObject(String sql, Object paras, Class<T> convertClass, String url, String userName,
+ String password) {
+ List<T> result = queryForList(sql, paras, convertClass, url, userName, password);
+ if (result == null || result.size() == 0) {
+ return null;
+ }
+ if (result.size() > 1) {
+ throw new RuntimeException("expect only one row, actual " + result.size() + ". the builder is " + sql);
+ }
+ return result.get(0);
+ }
+
+ /**
+ * 根据sql查询一批对象
+ *
+ * @param sql 查询语句
+ * @param paras 如果有变参，这里面有变参的参数，可以是map，json或对象，只要key名或字段名和sql的参数名相同即可
+ * @param convertClass 如果有变参，这里面有变参的参数，可以是map，json或对象，只要key名或字段名和sql的参数名相同即可
+ * @param <T>
+ * @return 返回对象列表
+ */
+ public static <T> List<T> queryForList(String sql, Object paras, Class<T> convertClass) {
+ return queryForList(sql, paras, convertClass, null, null, null);
+ }
+
+ public static boolean hasConfigueDB() {
+ return ComponentCreator.getProperties().getProperty(ConfigureFileKey.JDBC_URL) != null;
+ }
+
+ /**
+ * @param sql 查询语句
+ * @param paras 如果有变参，这里面有变参的参数，可以是map，json或对象，只要key名或字段名和sql的参数名相同即可
+ * @param convertClass 需要转换成的对象类。类的字段应该符合列名的命名规范，下划线换成驼峰形式
+ * @param url 数据库连接url
+ * @param userName 用户名
+ * @param password 密码
+ * @param <T>
+ * @return 返回对象列表
+ */
+ public static <T> List<T> queryForList(String sql, Object paras, Class<T> convertClass, String url, String userName,
+ String password) {
+ sql = SQLUtil.parseIbatisSQL(paras, sql);
+ JDBCDriver dataSource = null;
+ try {
+ if (StringUtil.isEmpty(url) || StringUtil.isEmpty(userName)) {
+ dataSource = DriverBuilder.createDriver();
+ } else {
+ dataSource = DriverBuilder.createDriver(DriverBuilder.DEFALUT_JDBC_DRIVER, url, userName, password);
+ }
+
+ List<Map<String, Object>> rows = dataSource.queryForList(sql);
+ List<T> result = new ArrayList();
+ for (Map<String, Object> row : rows) {
+ T t = convert(row, convertClass);
+ result.add(t);
+ }
+ return result;
+ } catch (Exception e) {
+ String errorMsg = ("query for list error ,the builder is " + sql + ". the error msg is " + e.getMessage());
+ LOG.error(errorMsg);
+ e.printStackTrace();
+ throw new RuntimeException(errorMsg, e);
+ } finally {
+ if (dataSource != null) {
+ dataSource.destroy();
+ }
+ }
+ }
+
+ /**
+ * 执行sql，sql中可以有mybatis的参数#{name}
+ *
+ * @param sql insert语句
+ * @param paras 可以是map，json或对象，只要key名或字段名和sql的参数名相同即可
+ * @return
+ */
+ public static boolean executeSQL(String sql, Object paras) {
+ if (paras != null) {
+ sql = SQLUtil.parseIbatisSQL(paras, sql);
+ }
+ JDBCDriver dataSource = null;
+ try {
+ dataSource = DriverBuilder.createDriver();
+ dataSource.execute(sql);
+ return true;
+ } catch (Exception e) {
+ String errorMsg = ("execute sql error ,the sql is " + sql + ". the error msg is " + e.getMessage());
+ LOG.error(errorMsg);
+ e.printStackTrace();
+ throw new RuntimeException(errorMsg, e);
+ } finally {
+ if (dataSource != null) {
+ dataSource.destroy();
+ }
+ }
+ }
+
+ /**
+ * 把一个对象的字段拼接成where条件，如果字段值为null，不拼接
+ *
+ * @param object 带拼接的对象
+ * @param fieldNames 需要拼接的字段名，如果为null，返回null
+ * @return where 部分的sql
+ */
+ public static String createQueryWhereSql(Object object, String... fieldNames) {
+ if (fieldNames == null || fieldNames.length == 0) {
+ return "";
+ }
+ StringBuilder stringBuilder = new StringBuilder();
+ boolean isFirst = true;
+ for (String fieldName : fieldNames) {
+ Object value = ReflectUtil.getBeanFieldValue(object, fieldName);
+ if (object != null && value == null) {
+ continue;
+ }
+ if (isFirst) {
+ isFirst = false;
+ } else {
+ stringBuilder.append(" and ");
+ }
+ String columnName = getColumnNameFromFieldName(fieldName);
+ stringBuilder.append(" " + columnName + "=#{" + fieldName + "} ");
+ }
+ return stringBuilder.toString();
+ }
+
+ /**
+ * 把一行数据转换成一个对象，符合驼峰的命名和下划线的命名规范
+ *
+ * @param row 一行数据
+ * @param clazz 待转化的对象类型
+ * @param <T>
+ * @return 转化对象
+ */
+ public static <T> T convert(Map<String, Object> row, Class<T> clazz) {
+ T t = ReflectUtil.forInstance(clazz);
+ Iterator<Map.Entry<String, Object>> it = row.entrySet().iterator();
+ while (it.hasNext()) {
+ Map.Entry<String, Object> entry = it.next();
+ String columnName = entry.getKey();
+ Object value = entry.getValue();
+ if (value == null) {
+ continue;
+ }
+ String fieldName = getFieldNameFromColumnName(columnName);
+ DataType datatype = DataTypeUtil.createFieldDataType(clazz, fieldName);
+ Object columnValue = datatype.convert(value);
+ ReflectUtil.setBeanFieldValue(t, fieldName, columnValue);
+ }
+ return t;
+ }
+
+ /**
+ * 把列名转换成字段名称，把下划线转化成驼峰
+ *
+ * @param columnName
+ * @return
+ */
+ protected static String getFieldNameFromColumnName(String columnName) {
+ String[] values = columnName.split("_");
+ if (values.length == 1) {
+ return columnName;
+ }
+ StringBuilder stringBuilder = new StringBuilder();
+ stringBuilder.append(values[0]);
+ for (int i = 1; i < values.length; i++) {
+ String value = values[i];
+ value = value.substring(0, 1).toUpperCase() + value.substring(1);
+ stringBuilder.append(value);
+ }
+ return stringBuilder.toString();
+ }
+
+ /**
+ * 对象批量替换，会生成replace into语句，多个对象会拼接成一个sql，提升效率
+ *
+ * @param values 待插入对象
+ */
+ public static void batchReplaceInto(Collection values) {
+ List list = new ArrayList<>();
+ list.addAll(values);
+ batchReplaceInto(list);
+ }
+
+ public static void batchReplaceInto(Object... valueArray) {
+ List values = new ArrayList();
+ if (valueArray != null) {
+ for (Object value : valueArray) {
+ values.add(value);
+ }
+ }
+ batchReplaceInto(values);
+ }
+
+ public static void batchReplaceInto(List<?> values) {
+ batchIntoByFlag(values, 1);
+ }
+
+ /**
+ * 对象批量插入，如果主键冲突会忽略，会生成insert ignore into语句，多个对象会拼接成一个sql，提升效率
+ *
+ * @param values 待插入对象
+ */
+ public static void batchIgnoreInto(List<?> values) {
+ batchIntoByFlag(values, -1);
+ }
+
+ /**
+ * 对象批量插入，如果主键冲突会忽略，会生成insert ignore into语句，多个对象会拼接成一个sql，提升效率
+ *
+ * @param values 待插入对象
+ */
+ public static void batchInsertInto(List<?> values) {
+ batchIntoByFlag(values, 0);
+ }
+
+ /**
+ * 批量插入对象，多个对象会拼接成一个sql flag==1 then replace into flag=-1 then insert ignore int flag=0 then insert int
+ *
+ * @param values
+ * @param flag
+ */
+ protected static void batchIntoByFlag(List<?> values, int flag) {
+ if (CollectionUtil.isEmpty(values)) {
+ return;
+ }
+ Object object = values.get(0);
+ Map<String, Object> paras = new HashMap<>(16);
+ MetaData metaData = createMetaDate(object, paras);
+ boolean containsIdField = false;
+ if (metaData.getIdFieldName() != null) {
+ for (Object o : values) {
+ Object id = ReflectUtil.getDeclaredField(o, metaData.getIdFieldName());
+ if (id == null) {
+ containsIdField = false;
+ break;
+ }
+ if (id instanceof Number) {
+ if (Long.valueOf(id.toString()) == 0) {
+ containsIdField = false;
+ break;
+ }
+ }
+ if (id instanceof String) {
+ String idStr = (String)id;
+ if (StringUtil.isEmpty(idStr)) {
+ containsIdField = false;
+ break;
+ }
+ }
+ }
+ }
+
+ String sql = null;
+ if (flag == 0) {
+ sql = SQLUtil.createInsertSql(metaData, paras, containsIdField);
+ } else if (flag == 1) {
+ sql = SQLUtil.createInsertSql(metaData, paras, containsIdField);
+ } else if (flag == -1) {
+ sql = SQLUtil.createIgnoreInsertSql(metaData, paras, containsIdField);
+ } else {
+ throw new RuntimeException("the flag is not valdate " + flag);
+ }
+
+ List<Map<String, Object>> rows = new ArrayList<>();
+ for (int i = 1; i < values.size(); i++) {
+ Map<String, Object> row = createRow(metaData, values.get(i));
+ rows.add(row);
+ }
+ String valuesSQL = SQLUtil.createInsertValuesSQL(metaData, rows);
+ sql = sql + valuesSQL + " ON DUPLICATE KEY UPDATE " + SQLUtil.createDuplicateKeyUpdateSQL(metaData);
+ ;
+ JDBCDriver dataSource = DriverBuilder.createDriver();
+ try {
+ dataSource.execute(sql);
+ } catch (Exception e) {
+ e.printStackTrace();
+ throw new RuntimeException(e);
+ } finally {
+ if (dataSource != null) {
+ dataSource.destroy();
+ }
+ }
+ }
+
+ private static Map<String, Object> createRow(MetaData metaData, Object object) {
+ Map<String, Object> row = new HashMap<>();
+ ReflectUtil.scanFields(object, new IFieldProcessor() {
+ @Override
+ public void doProcess(Object o, Field field) {
+ String fieldName = field.getName();
+ String columnName = getColumnNameFromFieldName(fieldName);
+ Object value = ReflectUtil.getBeanFieldValue(o, fieldName);
+ row.put(columnName, value);
+
+ }
+ });
+ return row;
+ }
+
+ public static void replaceInto(Object object) {
+ replaceInto(object, null, null, null);
+ }
+
+ /**
+ * 把一个对象插入到数据库，对象符合插入规范，表名是对象名转小写后加下划线。如果有重复到会被替换成最新的
+ *
+ * @param object
+ * @param url
+ * @param userName
+ * @param password
+ */
+ public static void replaceInto(Object object, String url, String userName, String password) {
+ Map<String, Object> paras = new HashMap<>();
+ if (Entity.class.isInstance(object)) {
+ Entity newEntity = (Entity)object;
+ newEntity.setGmtModified(new Date());
+ if (newEntity.getGmtCreate() == null) {
+ newEntity.setGmtCreate(new Date());
+ }
+ }
+ MetaData metaData = createMetaDate(object, paras);
+ String sql = SQLUtil.createReplacesInsertSql(metaData, paras, metaData.getIdFieldName() != null);
+ JDBCDriver dataSource = null;
+ try {
+ if (StringUtil.isEmpty(url) || StringUtil.isEmpty(userName)) {
+ dataSource = DriverBuilder.createDriver();
+ } else {
+ dataSource = DriverBuilder.createDriver(DriverBuilder.DEFALUT_JDBC_DRIVER, url, userName, password);
+ }
+ long id = dataSource.executeInsert(sql);
+ if (Entity.class.isInstance(object)) {
+ Entity newEntity = (Entity)object;
+ newEntity.setId(id);
+ }
+ } catch (Exception e) {
+ String errorMsg = ("replace into error ,the builder is " + sql + ". the error msg is " + e.getMessage());
+ LOG.error(errorMsg);
+ throw new RuntimeException(errorMsg, e);
+ } finally {
+ if (dataSource != null) {
+ dataSource.destroy();
+ }
+ }
+ }
+
+ public static void insertInto(Object object, boolean ignoreRepeateRow) {
+ insertInto(object, ignoreRepeateRow, null, null, null);
+ }
+
+ /**
+ * 把一个对象插入到数据库，对象符合插入规范，表名是对象名转小写后加下划线
+ *
+ * @param object
+ * @param ignoreRepeateRow，如果是重复数据，则不插入。基于唯一建做判断
+ * @param url
+ * @param userName
+ * @param password
+ */
+ public static void insertInto(Object object, boolean ignoreRepeateRow, String url, String userName,
+ String password) {
+ Map<String, Object> paras = new HashMap<>();
+ if (Entity.class.isInstance(object)) {
+ Entity newEntity = (Entity)object;
+ newEntity.setGmtCreate(DateUtil.getCurrentTime());
+ newEntity.setGmtModified(DateUtil.getCurrentTime());
+ }
+ MetaData metaData = createMetaDate(object, paras);
+ String sql = null;
+ if (ignoreRepeateRow) {
+ sql = SQLUtil.createIgnoreInsertSql(metaData, paras, metaData.getIdFieldName() != null);
+ } else {
+ sql = SQLUtil.createInsertSql(metaData, paras, metaData.getIdFieldName() != null);
+ }
+ JDBCDriver dataSource = null;
+ try {
+ if (StringUtil.isEmpty(url) || StringUtil.isEmpty(userName)) {
+ dataSource = DriverBuilder.createDriver();
+ } else {
+ dataSource = DriverBuilder.createDriver(DriverBuilder.DEFALUT_JDBC_DRIVER, url, userName, password);
+ }
+ long id = dataSource.executeInsert(sql);
+ if (Entity.class.isInstance(object)) {
+ Entity newEntity = (Entity)object;
+ newEntity.setId(id);
+ }
+ } catch (Exception e) {
+ String errorMsg = ("insert into error ,the builder is " + sql + ". the error msg is " + e.getMessage());
+ LOG.error(errorMsg);
+ throw new RuntimeException(errorMsg, e);
+ } finally {
+ if (dataSource != null) {
+ dataSource.destroy();
+ }
+ }
+ }
+
+ /**
+ * 创建meta信息
+ *
+ * @param object
+ * @param paras
+ * @return
+ */
+ public static MetaData createMetaDate(Object object, Map<String, Object> paras) {
+ MetaData metaData = MetaData.createMetaDate(object, paras);
+ return metaData;
+ }
+
+ public static String getTableName(Class clazz) {
+ return getColumnNameFromFieldName(clazz.getSimpleName());
+ }
+
+ /**
+ * 把驼峰转换成下划线的形式
+ *
+ * @param para
+ * @return
+ */
+ protected static String getColumnNameFromFieldName(String para) {
+ return MetaData.getColumnNameFromFieldName(para);
+ }
+
+}
diff --git a/rocketmq-streams-db-operator/src/main/java/org/apache/rocketmq/streams/db/operator/SQLOperator.java b/rocketmq-streams-db-operator/src/main/java/org/apache/rocketmq/streams/db/operator/SQLOperator.java
new file mode 100644
index 0000000..1f42470
--- /dev/null
+++ b/rocketmq-streams-db-operator/src/main/java/org/apache/rocketmq/streams/db/operator/SQLOperator.java
@@ -0,0 +1,178 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.db.operator;
+
+import java.util.List;
+import java.util.Map;
+
+import org.apache.rocketmq.streams.common.component.AbstractComponent;
+import org.apache.rocketmq.streams.db.driver.JDBCDriver;
+import org.apache.rocketmq.streams.common.configurable.BasedConfigurable;
+import org.apache.rocketmq.streams.common.configurable.annotation.Changeable;
+import org.apache.rocketmq.streams.common.configurable.annotation.ENVDependence;
+import org.apache.rocketmq.streams.common.interfaces.IStreamOperator;
+import org.apache.rocketmq.streams.common.context.AbstractContext;
+import org.apache.rocketmq.streams.common.context.IMessage;
+import org.apache.rocketmq.streams.common.topology.ChainStage;
+import org.apache.rocketmq.streams.common.topology.builder.IStageBuilder;
+import org.apache.rocketmq.streams.common.topology.stages.NewSQLChainStage;
+import org.apache.rocketmq.streams.common.topology.builder.PipelineBuilder;
+import org.apache.rocketmq.streams.common.utils.SQLUtil;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+import org.apache.rocketmq.streams.db.driver.DriverBuilder;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+/**
+ * sql算法，执行一个sql，sql中可以有变量，会用message的值做替换。
+ */
+public class SQLOperator extends BasedConfigurable implements IStreamOperator<IMessage, IMessage>, IStageBuilder<ChainStage> {
+ private static final Log LOG = LogFactory.getLog(SQLOperator.class);
+ public static final String DEFALUT_DATA_KEY = "data";
+
+ @ENVDependence
+ protected String jdbcDriver = AbstractComponent.DEFAULT_JDBC_DRIVER;
+ @ENVDependence
+ protected String url;
+ @ENVDependence
+ protected String userName;
+ @ENVDependence
+ protected String password;
+
+ @Changeable
+ protected String sql;//查询的sql，支持ibatis的语法和变量.因为会被替换，所以不自动感知。select * from table where name=#{name=chris}
+
+ public SQLOperator() {
+ setType(IStreamOperator.TYPE);
+ }
+
+ public SQLOperator(String sql, String url, String userName, String password) {
+ this();
+ this.sql = sql;
+ this.url = url;
+ this.password = password;
+ this.userName = userName;
+ }
+
+ /**
+ * db串多数是名字，可以取个名字前缀，如果值为空，默认为此类的name，name为空，默认为简单类名
+ *
+ * @param sql
+ * @param dbInfoNamePrex
+ */
+ public SQLOperator(String sql, String dbInfoNamePrex) {
+ this();
+ if (StringUtil.isEmpty(dbInfoNamePrex)) {
+ dbInfoNamePrex = getConfigureName();
+ }
+ if (StringUtil.isEmpty(dbInfoNamePrex)) {
+ dbInfoNamePrex = this.getClass().getSimpleName();
+ }
+ this.sql = sql;
+ this.url = dbInfoNamePrex + ".url";
+ this.password = dbInfoNamePrex + ".password";
+ ;
+ this.userName = dbInfoNamePrex + ".userName";
+ }
+
+ @Override
+ public IMessage doMessage(IMessage message, AbstractContext context) {
+ String querySQL = SQLUtil.parseIbatisSQL(message.getMessageBody(), sql);
+ List<Map<String, Object>> result = query(querySQL);
+ message.getMessageBody().put(DEFALUT_DATA_KEY, result);
+ return message;
+ }
+
+ /**
+ * 查询数据库数据
+ *
+ * @return
+ */
+ protected List<Map<String, Object>> query(String querySQL) {
+
+ JDBCDriver dataSource = null;
+ try {
+ dataSource = createDBDataSource();
+ List<Map<String, Object>> result = null;
+ result = dataSource.queryForList(sql);
+
+ return result;
+ } finally {
+ if (dataSource != null) {
+ dataSource.destroy();
+ }
+ }
+
+ }
+
+ public JDBCDriver createDBDataSource() {
+ return DriverBuilder.createDriver(jdbcDriver, url, userName, password);
+ }
+
+ @Override
+ public void addConfigurables(PipelineBuilder pipelineBuilder) {
+ pipelineBuilder.addConfigurables(this);
+ }
+
+ @Override
+ public ChainStage createStageChain(PipelineBuilder pipelineBuilder) {
+ NewSQLChainStage sqlChainStage = new NewSQLChainStage();
+ sqlChainStage.setMessageProcessor(this);
+ return sqlChainStage;
+ }
+
+ public String getSql() {
+ return sql;
+ }
+
+ public void setSql(String sql) {
+ this.sql = sql;
+ }
+
+ public String getJdbcDriver() {
+ return jdbcDriver;
+ }
+
+ public void setJdbcDriver(String jdbcDriver) {
+ this.jdbcDriver = jdbcDriver;
+ }
+
+ public String getUrl() {
+ return url;
+ }
+
+ public void setUrl(String url) {
+ this.url = url;
+ }
+
+ public String getUserName() {
+ return userName;
+ }
+
+ public void setUserName(String userName) {
+ this.userName = userName;
+ }
+
+ public String getPassword() {
+ return password;
+ }
+
+ public void setPassword(String password) {
+ this.password = password;
+ }
+
+}
diff --git a/rocketmq-streams-db-operator/src/test/java/org/apache/rocketmq/streams/db/Person.java b/rocketmq-streams-db-operator/src/test/java/org/apache/rocketmq/streams/db/Person.java
new file mode 100644
index 0000000..3d5e51b
--- /dev/null
+++ b/rocketmq-streams-db-operator/src/test/java/org/apache/rocketmq/streams/db/Person.java
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.db;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.rocketmq.streams.common.configurable.BasedConfigurable;
+import org.apache.rocketmq.streams.common.configurable.annotation.ENVDependence;
+
+public class Person extends BasedConfigurable {
+ @ENVDependence
+ private String name;
+ private int age;
+ private Boolean isMale;
+ private List<String> addresses;
+ private Map<String, Integer> childName2Age;
+
+ public static Person createPerson(String namespace) {
+ Person person = new Person();
+ person.setNameSpace(namespace);
+ person.setType("person");
+ person.setConfigureName("Chris");
+ person.setName("Chris");
+ List<String> addresses = new ArrayList<>();
+ addresses.add("huilongguan");
+ addresses.add("shangdi");
+ person.setAddresses(addresses);
+ Map<String, Integer> childName2Age = new HashMap<>();
+ childName2Age.put("yuanyahan", 8);
+ childName2Age.put("yuanruxi", 4);
+ person.setChildName2Age(childName2Age);
+ person.setMale(true);
+ person.setAge(18);
+ return person;
+ }
+
+ @Override
+ public String toString() {
+ return "Person{" + "name='" + name + '\'' + ", age=" + age + ", isMale=" + isMale + ", addresses=" + addresses
+ + ", childName2Age=" + childName2Age + '}';
+ }
+
+ public String getName() {
+ return name;
+ }
+
+ public void setName(String name) {
+ this.name = name;
+ }
+
+ public int getAge() {
+ return age;
+ }
+
+ public void setAge(int age) {
+ this.age = age;
+ }
+
+ public Boolean getMale() {
+ return isMale;
+ }
+
+ public void setMale(Boolean male) {
+ isMale = male;
+ }
+
+ public List<String> getAddresses() {
+ return addresses;
+ }
+
+ public void setAddresses(List<String> addresses) {
+ this.addresses = addresses;
+ }
+
+ public Map<String, Integer> getChildName2Age() {
+ return childName2Age;
+ }
+
+ public void setChildName2Age(Map<String, Integer> childName2Age) {
+ this.childName2Age = childName2Age;
+ }
+
+ @Override
+ public Object clone() {
+ Person person = null;
+ try {
+ person = (Person)super.clone();
+ } catch (CloneNotSupportedException e) {
+ System.out.println("clone error " + e);
+ }
+ return person;
+ }
+}
diff --git a/rocketmq-streams-db-operator/src/test/java/org/apache/rocketmq/streams/db/configuable/DBSupportParentConfigureServiceTest.java b/rocketmq-streams-db-operator/src/test/java/org/apache/rocketmq/streams/db/configuable/DBSupportParentConfigureServiceTest.java
new file mode 100644
index 0000000..3baa65d
--- /dev/null
+++ b/rocketmq-streams-db-operator/src/test/java/org/apache/rocketmq/streams/db/configuable/DBSupportParentConfigureServiceTest.java
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.db.configuable;
+
+import org.apache.rocketmq.streams.common.component.ComponentCreator;
+import org.apache.rocketmq.streams.common.configure.ConfigureFileKey;
+import org.apache.rocketmq.streams.configurable.ConfigurableComponent;
+import org.apache.rocketmq.streams.configurable.model.Configure;
+import org.apache.rocketmq.streams.db.Person;
+import org.apache.rocketmq.streams.db.driver.DriverBuilder;
+import org.junit.Test;
+
+import static junit.framework.TestCase.assertTrue;
+
+/**
+ * 数据库的存储，需要配置存储的连接参数，请先完成配置，后执行单元用例 如果未建表，可以通过Configure.createTableSQL() 获取建表语句，创建表后，测试
+ */
+public class DBSupportParentConfigureServiceTest {
+ private String URL = "";
+ protected String USER_NAME = "";
+ protected String PASSWORD = "";
+ protected String TABLE_NAME = "dipper_configure_source";
+
+ @Test
+ public void testDBConfigurableService() {
+ String namespace = "streams.db.configuable";
+
+ //正式使用时，在配置文件配置
+ ComponentCreator.getProperties().put(ConfigureFileKey.CONNECT_TYPE, "DB");
+ ComponentCreator.getProperties().put(ConfigureFileKey.JDBC_URL, URL);//数据库连接url
+ ComponentCreator.getProperties().put(ConfigureFileKey.JDBC_USERNAME, USER_NAME);//用户名
+ ComponentCreator.getProperties().put(ConfigureFileKey.JDBC_PASSWORD, PASSWORD);//password
+ ComponentCreator.getProperties().put(ConfigureFileKey.JDBC_TABLE_NAME, TABLE_NAME);
+
+ //如果表不存在，创建表
+ String sql = (Configure.createTableSQL(TABLE_NAME));
+ DriverBuilder.createDriver().execute(sql);
+ ConfigurableComponent configurableComponent = ConfigurableComponent.getInstance(namespace);
+ configurableComponent.insert(createPerson(namespace));
+ configurableComponent.refreshConfigurable(namespace);
+ Person person = configurableComponent.queryConfigurable("person", "peronName");
+ assertTrue(person != null);
+ }
+
+ /**
+ * 创建configuable对象
+ *
+ * @param namespace
+ * @return
+ */
+ protected Person createPerson(String namespace) {
+ Person person = new Person();
+ person.setName("chris");
+ person.setAge(18);
+ person.setNameSpace(namespace);
+ person.setConfigureName("peronName");
+ person.setType("person");
+ return person;
+ }
+}
diff --git a/rocketmq-streams-db-operator/src/test/java/org/apache/rocketmq/streams/db/driver/orm/ORMUtilTest.java b/rocketmq-streams-db-operator/src/test/java/org/apache/rocketmq/streams/db/driver/orm/ORMUtilTest.java
new file mode 100644
index 0000000..e9b1fa2
--- /dev/null
+++ b/rocketmq-streams-db-operator/src/test/java/org/apache/rocketmq/streams/db/driver/orm/ORMUtilTest.java
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.db.driver.orm;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.rocketmq.streams.common.component.ComponentCreator;
+import org.apache.rocketmq.streams.common.configure.ConfigureFileKey;
+import org.apache.rocketmq.streams.db.Person;
+import org.junit.Test;
+
+public class ORMUtilTest {
+ private String URL = "";
+ protected String USER_NAME = "";
+ protected String PASSWORD = "";
+
+ public ORMUtilTest() {
+ //正式使用时，在配置文件配置
+ ComponentCreator.getProperties().put(ConfigureFileKey.JDBC_URL, URL);//数据库连接url
+ ComponentCreator.getProperties().put(ConfigureFileKey.JDBC_USERNAME, USER_NAME);//用户名
+ ComponentCreator.getProperties().put(ConfigureFileKey.JDBC_PASSWORD, PASSWORD);//password
+ }
+
+ @Test
+ public void testInsert() {
+ String namespace = "org.apache.configuable.test";
+ List<Person> personList = new ArrayList<>();
+ for (int i = 0; i < 10; i++) {
+ personList.add(createPerson(namespace, "chris" + i));
+ }
+ /**
+ * 不带数据库连接信息（url,userName,Password），使用ConfiguableComponet的连接信息
+ */
+ ORMUtil.batchIgnoreInto(personList);//批量插入，如果有唯一键冲突，替换
+ ORMUtil.batchIgnoreInto(personList);//批量插入，如果有唯一键冲突，忽略
+ ORMUtil.batchIntoByFlag(personList, 0);////批量插入，如果有唯一键冲突，跑错
+ }
+
+ @Test
+ public void testQueryList() {
+ Map<String, Integer> paras = new HashMap<>();
+ paras.put("age", 18);
+ List<Person> personList = ORMUtil.queryForList("select * from person where age >${age} limit 100", paras, Person.class);
+ }
+
+ @Test
+ public void testQueryOneRow() {
+ Person personPara = new Person();
+ personPara.setAge(18);
+ personPara.setName("chris1");
+ Person person = ORMUtil.queryForObject("select * from person where age =${age} and name='${name}' ", personPara, Person.class, URL, USER_NAME, PASSWORD);
+ }
+
+ /**
+ * 创建configuable对象
+ *
+ * @param namespace
+ * @return
+ */
+ protected Person createPerson(String namespace, String name) {
+ Person person = new Person();
+ person.setName(name);
+ person.setAge(18);
+ person.setNameSpace(namespace);
+ person.setConfigureName("peronName");
+ person.setType("person");
+ return person;
+ }
+}
diff --git a/rocketmq-streams-transport-minio/pom.xml b/rocketmq-streams-transport-minio/pom.xml
new file mode 100755
index 0000000..510b8cc
--- /dev/null
+++ b/rocketmq-streams-transport-minio/pom.xml
@@ -0,0 +1,25 @@
+<?xml version="1.0" encoding="utf-8"?>
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams</artifactId>
+ <version>2.0.0-SNAPSHOT</version>
+ </parent>
+ <artifactId>rocketmq-streams-transport-minio</artifactId>
+ <packaging>jar</packaging>
+ <name>ROCKETMQ STREAMS :: transport-minio</name>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams-serviceloader</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>io.minio</groupId>
+ <artifactId>minio</artifactId>
+ </dependency>
+ </dependencies>
+</project>
diff --git a/rocketmq-streams-transport-minio/rocketmq-streams-transport-minio.iml b/rocketmq-streams-transport-minio/rocketmq-streams-transport-minio.iml
new file mode 100644
index 0000000..af11f77
--- /dev/null
+++ b/rocketmq-streams-transport-minio/rocketmq-streams-transport-minio.iml
@@ -0,0 +1,17 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<module org.jetbrains.idea.maven.project.MavenProjectsManager.isMavenModule="true" type="JAVA_MODULE" version="4">
+ <component name="NewModuleRootManager" LANGUAGE_LEVEL="JDK_1_5">
+ <output url="file://$MODULE_DIR$/${project.build.directory}/classes" />
+ <output-test url="file://$MODULE_DIR$/${project.build.directory}/test-classes" />
+ <content url="file://$MODULE_DIR$">
+ <sourceFolder url="file://$MODULE_DIR$/src/main/java" isTestSource="false" />
+ <sourceFolder url="file://$MODULE_DIR$/src/test/java" isTestSource="true" />
+ <sourceFolder url="file://$MODULE_DIR$/src/test/resources" type="java-test-resource" />
+ <excludeFolder url="file://$MODULE_DIR$/${project.build.directory}/classes" />
+ <excludeFolder url="file://$MODULE_DIR$/${project.build.directory}/test-classes" />
+ <excludeFolder url="file://$MODULE_DIR$/target" />
+ </content>
+ <orderEntry type="inheritedJdk" />
+ <orderEntry type="sourceFolder" forTests="false" />
+ </component>
+</module>
\ No newline at end of file
diff --git a/rocketmq-streams-transport-minio/src/main/java/org/apache/rocketmq/streams/transport/minio/MinioFileTransport.java b/rocketmq-streams-transport-minio/src/main/java/org/apache/rocketmq/streams/transport/minio/MinioFileTransport.java
new file mode 100644
index 0000000..c2a6884
--- /dev/null
+++ b/rocketmq-streams-transport-minio/src/main/java/org/apache/rocketmq/streams/transport/minio/MinioFileTransport.java
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.transport.minio;
+
+import org.apache.rocketmq.streams.common.component.ComponentCreator;
+import org.apache.rocketmq.streams.common.configure.ConfigureFileKey;
+import org.apache.rocketmq.streams.common.model.ServiceName;
+import org.apache.rocketmq.streams.common.transport.AbstractFileTransport;
+import org.apache.rocketmq.streams.common.transport.IFileTransport;
+import org.apache.rocketmq.streams.common.utils.FileUtil;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+import com.google.auto.service.AutoService;
+import io.minio.MinioClient;
+
+import java.io.BufferedReader;
+import java.io.BufferedWriter;
+import java.io.File;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+
+@AutoService(IFileTransport.class)
+@ServiceName(MinioFileTransport.NAME)
+public class MinioFileTransport extends AbstractFileTransport {
+ public static final String NAME = "minio";
+ protected String ak;
+ protected String sk;
+ protected String endpoint;
+ protected String dirpperDir;
+ protected MinioClient minioClient;
+
+ public MinioFileTransport() {
+ this.ak = ComponentCreator.getProperties().getProperty(ConfigureFileKey.FILE_TRANSPORT_AK);
+ this.sk = ComponentCreator.getProperties().getProperty(ConfigureFileKey.FILE_TRANSPORT_SK);
+ this.endpoint = ComponentCreator.getProperties().getProperty(ConfigureFileKey.FILE_TRANSPORT_ENDPOINT);
+ this.dirpperDir = ComponentCreator.getProperties().getProperty(ConfigureFileKey.FILE_TRANSPORT_DIPPER_DIR);
+ if (StringUtil.isEmpty(this.dirpperDir)) {
+ this.dirpperDir = "dipper_files";
+ }
+ }
+
+ @Override
+ public File download(String remoteFileName, String localDir, String localFileName) {
+ MinioClient minioClient = getOrCreateMinioClient();
+ BufferedWriter bw = null;
+ BufferedReader br = null;
+ try {
+ InputStream input = minioClient.getObject(dirpperDir, remoteFileName);
+ File file = new File(FileUtil.concatFilePath(localDir, localFileName));
+ bw = new BufferedWriter(new FileWriter(file));
+ br = new BufferedReader(new InputStreamReader(input));
+ String line = br.readLine();
+ while (line != null) {
+ bw.write(line);
+ line = br.readLine();
+ }
+ bw.flush();
+ return file;
+ } catch (Exception e) {
+ throw new RuntimeException("dowload file error " + dirpperDir + "/" + remoteFileName, e);
+
+ } finally {
+ if (bw != null) {
+ try {
+ bw.close();
+ } catch (IOException e) {
+ e.printStackTrace();
+ }
+ }
+ if (br != null) {
+ try {
+ br.close();
+ } catch (IOException e) {
+ e.printStackTrace();
+ }
+ }
+ }
+
+ }
+
+ @Override
+ public Boolean upload(File file, String remoteFileName) {
+ MinioClient minioClient = getOrCreateMinioClient();
+ try {
+ minioClient.putObject(dirpperDir, remoteFileName, file.getAbsolutePath());
+ } catch (Exception e) {
+ throw new RuntimeException("upload file error " + dirpperDir + "/" + remoteFileName, e);
+ }
+ return true;
+ }
+
+ @Override
+ public boolean delete(String remoteFileName) {
+ MinioClient minioClient = getOrCreateMinioClient();
+ try {
+ minioClient.removeObject(dirpperDir, remoteFileName);
+ } catch (Exception e) {
+ throw new RuntimeException("delete file error " + dirpperDir + "/" + remoteFileName, e);
+ }
+ return true;
+ }
+
+ protected MinioClient getOrCreateMinioClient() {
+ if (this.minioClient == null) {
+ synchronized (this) {
+ if (minioClient == null) {
+ try {
+ MinioClient minioClient = new MinioClient(endpoint, ak, sk);
+ boolean existDir = minioClient.bucketExists(dirpperDir);
+
+ if (!existDir) {
+ minioClient.makeBucket(dirpperDir);
+ }
+ this.minioClient = minioClient;
+ } catch (Exception e) {
+ throw new RuntimeException("create minio client error", e);
+ }
+
+ }
+ }
+ }
+ return this.minioClient;
+ }
+}
+
+
diff --git a/rocketmq-streams-transport-minio/src/test/java/com/aliyun/yundun/dipper/configurable/DataTpyeTest.java b/rocketmq-streams-transport-minio/src/test/java/com/aliyun/yundun/dipper/configurable/DataTpyeTest.java
new file mode 100644
index 0000000..a3ebf6b
--- /dev/null
+++ b/rocketmq-streams-transport-minio/src/test/java/com/aliyun/yundun/dipper/configurable/DataTpyeTest.java
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.aliyun.yundun.dipper.configurable;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+import org.junit.Test;
+
+import org.apache.rocketmq.streams.common.utils.FileUtil;
+import org.apache.rocketmq.streams.configurable.model.Person;
+
+public class DataTpyeTest {
+ @Test
+ public void testDataType() {
+ Person person = Person.createPerson("com.dipper.test");
+ String jsonValue = person.toJson();
+
+ Person person1 = new Person();
+ person1.toObject(jsonValue);
+ System.out.println(person1);
+ }
+
+ @Test
+ public void testV2() {
+ Set<String> set = new HashSet<>();
+ set.add("北斗");
+ set.add("福建jz");
+ set.add("甘肃jz");
+ set.add("广东省气象micaps云");
+ set.add("贵州公安科信");
+ set.add("贵州警务云");
+ set.add("杭州税友");
+ set.add("江西公安大数据平台");
+ set.add("昆仑项目");
+ set.add("新华网");
+ set.add("浙江气象高时空分辨率气象预报专有云");
+ List<String> v2 = FileUtil.loadFileLine("/Users/yuanxiaodong/Documents/workdir/项目名称.txt");
+ List<String> zyy = FileUtil.loadFileLine("/Users/yuanxiaodong/Documents/workdir/专有云.txt");
+ int count = 0;
+ for (String v2Line : v2) {
+ boolean match = false;
+ for (String zyyLine : zyy) {
+ if (zyyLine.indexOf(v2Line) != -1 || v2Line.indexOf(zyyLine) != -1) {
+ match = true;
+ count++;
+ }
+ }
+ if (match == false) {
+ System.out.println(v2Line);
+ }
+ }
+ System.out.println(count);
+ }
+}
diff --git a/rocketmq-streams-transport-minio/src/test/java/org/apache/rocketmq/streams/configuable/model/DataTpyeTest.java b/rocketmq-streams-transport-minio/src/test/java/org/apache/rocketmq/streams/configuable/model/DataTpyeTest.java
new file mode 100644
index 0000000..a1570c3
--- /dev/null
+++ b/rocketmq-streams-transport-minio/src/test/java/org/apache/rocketmq/streams/configuable/model/DataTpyeTest.java
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.streams.configuable.model;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import org.apache.rocketmq.streams.common.utils.FileUtil;
+import org.junit.Test;
+
+public class DataTpyeTest {
+ @Test
+ public void testDataType() {
+ Person person = Person.createPerson("com.dipper.test");
+ String jsonValue = person.toJson();
+
+ Person person1 = new Person();
+ person1.toObject(jsonValue);
+ System.out.println(person1);
+ }
+
+ @Test
+ public void testV2() {
+ Set<String> set = new HashSet<>();
+ set.add("北斗");
+ set.add("福建jz");
+ set.add("甘肃jz");
+ set.add("广东省气象micaps云");
+ set.add("贵州公安科信");
+ set.add("贵州警务云");
+ set.add("杭州税友");
+ set.add("江西公安大数据平台");
+ set.add("昆仑项目");
+ set.add("新华网");
+ set.add("浙江气象高时空分辨率气象预报专有云");
+ List<String> v2 = FileUtil.loadFileLine("/Users/yuanxiaodong/Documents/workdir/项目名称.txt");
+ List<String> zyy = FileUtil.loadFileLine("/Users/yuanxiaodong/Documents/workdir/专有云.txt");
+ int count = 0;
+ for (String v2Line : v2) {
+ boolean match = false;
+ for (String zyyLine : zyy) {
+ if (zyyLine.indexOf(v2Line) != -1 || v2Line.indexOf(zyyLine) != -1) {
+ match = true;
+ count++;
+ }
+ }
+ if (match == false) {
+ System.out.println(v2Line);
+ }
+ }
+ System.out.println(count);
+ }
+}
diff --git a/rocketmq-streams-transport-minio/src/test/java/org/apache/rocketmq/streams/configuable/model/Person.java b/rocketmq-streams-transport-minio/src/test/java/org/apache/rocketmq/streams/configuable/model/Person.java
new file mode 100644
index 0000000..04b99bb
--- /dev/null
+++ b/rocketmq-streams-transport-minio/src/test/java/org/apache/rocketmq/streams/configuable/model/Person.java
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.streams.configuable.model;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.rocketmq.streams.common.configurable.BasedConfigurable;
+
+public class Person extends BasedConfigurable {
+ private String name;
+ private int age;
+ private Boolean isMale;
+ private List<String> addresses;
+ private Map<String, Integer> childName2Age;
+
+ public static Person createPerson(String namespace) {
+ Person person = new Person();
+ person.setNameSpace(namespace);
+ person.setType("person");
+ person.setConfigureName("Chris");
+ person.setName("Chris");
+ List<String> addresses = new ArrayList<>();
+ addresses.add("huilongguan");
+ addresses.add("shangdi");
+ person.setAddresses(addresses);
+ Map<String, Integer> childName2Age = new HashMap<>();
+ childName2Age.put("yuanyahan", 8);
+ childName2Age.put("yuanruxi", 4);
+ person.setChildName2Age(childName2Age);
+ person.setMale(true);
+ person.setAge(18);
+ return person;
+ }
+
+ @Override
+ public String toString() {
+ return "Person{" + "name='" + name + '\'' + ", age=" + age + ", isMale=" + isMale + ", addresses=" + addresses
+ + ", childName2Age=" + childName2Age + '}';
+ }
+
+ public String getName() {
+ return name;
+ }
+
+ public void setName(String name) {
+ this.name = name;
+ }
+
+ public int getAge() {
+ return age;
+ }
+
+ public void setAge(int age) {
+ this.age = age;
+ }
+
+ public Boolean getMale() {
+ return isMale;
+ }
+
+ public void setMale(Boolean male) {
+ isMale = male;
+ }
+
+ public List<String> getAddresses() {
+ return addresses;
+ }
+
+ public void setAddresses(List<String> addresses) {
+ this.addresses = addresses;
+ }
+
+ public Map<String, Integer> getChildName2Age() {
+ return childName2Age;
+ }
+
+ public void setChildName2Age(Map<String, Integer> childName2Age) {
+ this.childName2Age = childName2Age;
+ }
+}
diff --git a/rocketmq-streams-transport-minio/src/test/java/org/apache/rocketmq/streams/configurable/model/Person.java b/rocketmq-streams-transport-minio/src/test/java/org/apache/rocketmq/streams/configurable/model/Person.java
new file mode 100644
index 0000000..709978a
--- /dev/null
+++ b/rocketmq-streams-transport-minio/src/test/java/org/apache/rocketmq/streams/configurable/model/Person.java
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.configurable.model;
+
+import org.apache.rocketmq.streams.common.configurable.BasedConfigurable;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class Person extends BasedConfigurable {
+ private String name;
+ private int age;
+ private Boolean isMale;
+ private List<String> addresses;
+ private Map<String, Integer> childName2Age;
+
+ public static Person createPerson(String namespace) {
+ Person person = new Person();
+ person.setNameSpace(namespace);
+ person.setType("person");
+ person.setConfigureName("Chris");
+ person.setName("Chris");
+ List<String> addresses = new ArrayList<>();
+ addresses.add("huilongguan");
+ addresses.add("shangdi");
+ person.setAddresses(addresses);
+ Map<String, Integer> childName2Age = new HashMap<>();
+ childName2Age.put("yuanyahan", 8);
+ childName2Age.put("yuanruxi", 4);
+ person.setChildName2Age(childName2Age);
+ person.setMale(true);
+ person.setAge(18);
+ return person;
+ }
+
+ @Override
+ public String toString() {
+ return "Person{" + "name='" + name + '\'' + ", age=" + age + ", isMale=" + isMale + ", addresses=" + addresses
+ + ", childName2Age=" + childName2Age + '}';
+ }
+
+ public String getName() {
+ return name;
+ }
+
+ public void setName(String name) {
+ this.name = name;
+ }
+
+ public int getAge() {
+ return age;
+ }
+
+ public void setAge(int age) {
+ this.age = age;
+ }
+
+ public Boolean getMale() {
+ return isMale;
+ }
+
+ public void setMale(Boolean male) {
+ isMale = male;
+ }
+
+ public List<String> getAddresses() {
+ return addresses;
+ }
+
+ public void setAddresses(List<String> addresses) {
+ this.addresses = addresses;
+ }
+
+ public Map<String, Integer> getChildName2Age() {
+ return childName2Age;
+ }
+
+ public void setChildName2Age(Map<String, Integer> childName2Age) {
+ this.childName2Age = childName2Age;
+ }
+}
diff --git a/rocketmq-streams-transport-minio/src/test/resources/component/ConfigurableComponent.properties b/rocketmq-streams-transport-minio/src/test/resources/component/ConfigurableComponent.properties
new file mode 100644
index 0000000..598511e
--- /dev/null
+++ b/rocketmq-streams-transport-minio/src/test/resources/component/ConfigurableComponent.properties
@@ -0,0 +1,7 @@
+dipper.configurable.service.type=resource_support_parent
+dipper.channle.ak=xxxxxx
+dipper.channle.sk=xxxxxx
+dipper.rds.jdbc.driver=com.mysql.jdbc.Driver
+dipper.rds.jdbc.url=xxxxxxx
+dipper.rds.jdbc.username=xxxxxx
+dipper.rds.jdbc.password=xxxxx
\ No newline at end of file
diff --git a/rocketmq-streams-transport-minio/src/test/resources/log4j.xml b/rocketmq-streams-transport-minio/src/test/resources/log4j.xml
new file mode 100755
index 0000000..7812fe7
--- /dev/null
+++ b/rocketmq-streams-transport-minio/src/test/resources/log4j.xml
@@ -0,0 +1,20 @@
+<?xml version="1.0" encoding="UTF-8" ?>
+<!DOCTYPE log4j:configuration SYSTEM "http://toolkit.alibaba-inc.com/dtd/log4j/log4j.dtd">
+<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
+
+ <appender name="Console" class="org.apache.log4j.ConsoleAppender">
+ <layout class="org.apache.log4j.PatternLayout">
+ <param name="ConversionPattern" value="%d{ISO8601} %l [%t] %-5p - %m%n%n"/>
+ </layout>
+ <filter class="org.apache.log4j.varia.LevelRangeFilter">
+ <param name="LevelMin" value="INFO"/>
+ <param name="LevelMax" value="ERROR"/>
+ </filter>
+ </appender>
+
+ <root>
+ <priority value="INFO"/>
+ <appender-ref ref="Console"/>
+ </root>
+
+</log4j:configuration>
\ No newline at end of file
diff --git a/rocketmq-streams-transport-minio/src/test/resources/pro-function.txt b/rocketmq-streams-transport-minio/src/test/resources/pro-function.txt
new file mode 100644
index 0000000..34a186f
--- /dev/null
+++ b/rocketmq-streams-transport-minio/src/test/resources/pro-function.txt
@@ -0,0 +1,11 @@
+paserBySplit(@,uuid,file_path,pid,ppid,pfile_path,group_name,group_id,user_name,uid,euid,egroup_id,time,cmd_line,index,perm,tty,pcmdline,sid,cwd,filename);
+addRandom(messageId,10);
+rename(groupname,group_name);
+rename(username,user_name);
+rename(seq,index);
+rename(egourpid,egroup_id);
+rename(filepath,file_path);
+rename(groupid,group_id);
+rename(pfilename,pfile_name);
+rename(safe_mode,perm);
+rename(cmdline,cmd_line);
\ No newline at end of file
diff --git a/rocketmq-streams-transport-minio/src/test/resources/python_script.py b/rocketmq-streams-transport-minio/src/test/resources/python_script.py
new file mode 100644
index 0000000..f4e7252
--- /dev/null
+++ b/rocketmq-streams-transport-minio/src/test/resources/python_script.py
@@ -0,0 +1,22 @@
+#!/usr/bin/env python
+# coding=utf-8
+import json;
+import re;
+import time;
+regex = '^/(.*)/\w+'
+pattern = re.compile(regex)
+
+def pythonTest(*processLine):
+ try:
+ jsonObject = json.loads(processLine[0])
+
+ if jsonObject.has_key('filepath'):
+ filePath = jsonObject['filepath']
+ match = pattern.search(filePath)
+ if match:
+ return match.group(1)
+ else:
+ pass # print "does not has key filepath"
+ except BaseException as e:
+ pass # print "process one line cause exception %s" %e
+ return "does not match"
[rocketmq-streams] 18/35: Merge pull request #1 from
programer-0/develop
Posted by du...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
duhengforever pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/rocketmq-streams.git
commit 6d89c9ad60b17bac2d5d706f587ee306862d25a6
Merge: 997f3df 5555b67
Author: Heng Du <du...@apache.org>
AuthorDate: Mon Aug 2 13:30:04 2021 +0800
Merge pull request #1 from programer-0/develop
add lease、dim and client module
.gitignore | 25 ++
README.md | 108 ++++-
pom.xml | 410 +++++++++++++++++++
rocketmq-streams-clients/pom.xml | 46 +++
.../rocketmq/streams/client/DataStreamAction.java | 101 +++++
.../rocketmq/streams/client/StreamBuilder.java | 28 ++
.../streams/client/source/DataStreamSource.java | 76 ++++
.../client/strategy/CheckpointStrategy.java | 69 ++++
.../streams/client/strategy/StateStrategy.java | 37 ++
.../rocketmq/streams/client/strategy/Strategy.java | 25 ++
.../streams/client/transform/DataStream.java | 437 +++++++++++++++++++++
.../streams/client/transform/JoinStream.java | 212 ++++++++++
.../streams/client/transform/SplitStream.java | 61 +++
.../streams/client/transform/WindowStream.java | 210 ++++++++++
.../client/transform/window/HoppingWindow.java | 32 ++
.../client/transform/window/SessionWindow.java | 32 ++
.../streams/client/transform/window/Time.java | 45 +++
.../client/transform/window/TumblingWindow.java | 33 ++
.../client/transform/window/WindowInfo.java | 83 ++++
.../rocketmq/streams/client/DBDriverTest.java | 75 ++++
.../rocketmq/streams/client/DataStreamTest.java | 107 +++++
.../apache/rocketmq/streams/client/FilterTest.java | 49 +++
.../apache/rocketmq/streams/client/JoinTest.java | 89 +++++
.../apache/rocketmq/streams/client/LeaseTest.java | 98 +++++
.../rocketmq/streams/client/ORMUtilTest.java | 172 ++++++++
.../apache/rocketmq/streams/client/SplitTest.java | 86 ++++
.../apache/rocketmq/streams/client/UnionTest.java | 82 ++++
.../apache/rocketmq/streams/client/WindowTest.java | 86 ++++
.../client/windows/AbstractWindowFireModeTest.java | 189 +++++++++
.../streams/client/windows/WindowFromFileTest.java | 158 ++++++++
.../streams/client/windows/WindowFromMetaq.java | 47 +++
.../client/windows/WindowHighAvailabilityTest.java | 131 ++++++
.../src/test/resources/log4j.xml | 36 ++
rocketmq-streams-dim/pom.xml | 47 +++
.../apache/rocketmq/streams/dim/DimComponent.java | 63 +++
.../rocketmq/streams/dim/builder/DimBuilder.java | 94 +++++
.../function/expression/InExpressionResource.java | 80 ++++
.../expression/NotInExpressionResource.java | 45 +++
.../dim/function/script/IntelligenceFunction.java | 81 ++++
.../script/IntelligenceNameListFunction.java | 24 ++
.../dim/function/script/NameListFunction.java | 203 ++++++++++
.../rocketmq/streams/dim/index/DimIndex.java | 319 +++++++++++++++
.../rocketmq/streams/dim/index/IndexExecutor.java | 258 ++++++++++++
.../intelligence/AbstractIntelligenceCache.java | 395 +++++++++++++++++++
.../dim/intelligence/AccountIntelligenceCache.java | 77 ++++
.../dim/intelligence/DomainIntelligenceCache.java | 83 ++++
.../dim/intelligence/IPIntelligenceCache.java | 108 +++++
.../dim/intelligence/URLIntelligenceCache.java | 80 ++++
.../rocketmq/streams/dim/model/AbstractDim.java | 312 +++++++++++++++
.../streams/dim/model/BooleanFieldDBDim.java | 55 +++
.../apache/rocketmq/streams/dim/model/DBDim.java | 140 +++++++
.../rocketmq/streams/dim/service/IDimService.java | 65 +++
.../streams/dim/service/impl/DimServiceImpl.java | 92 +++++
.../com/aliyun/service/ConfigureLoaderTest.java | 37 ++
.../com/aliyun/service/ExpressionExecutorTest.java | 80 ++++
.../java/com/aliyun/service/JsonParserTest.java | 40 ++
.../com/aliyun/service/NameListFunctionTest.java | 90 +++++
.../java/com/aliyun/service/TableCompressTest.java | 26 ++
rocketmq-streams-lease/pom.xml | 25 ++
.../rocketmq/streams/lease/LeaseComponent.java | 103 +++++
.../rocketmq/streams/lease/model/LeaseInfo.java | 127 ++++++
.../streams/lease/service/ILeaseGetCallback.java | 30 ++
.../streams/lease/service/ILeaseService.java | 136 +++++++
.../streams/lease/service/ILeaseStorage.java | 73 ++++
.../streams/lease/service/ILeaseStorasge.java | 63 +++
.../lease/service/impl/BasedLesaseImpl.java | 404 +++++++++++++++++++
.../lease/service/impl/LeaseServiceImpl.java | 275 +++++++++++++
.../streams/lease/service/impl/MockLeaseImpl.java | 95 +++++
.../lease/service/storages/DBLeaseStorage.java | 229 +++++++++++
.../rocketmq/streams/lease/LeaseComponentTest.java | 119 ++++++
.../src/test/resources/log4j.xml | 20 +
71 files changed, 8067 insertions(+), 1 deletion(-)
[rocketmq-streams] 14/35: modify the README.md
Posted by du...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
duhengforever pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/rocketmq-streams.git
commit ec252775987b1acbeb1fb848a819ccf176e7fae4
Author: 刈刀 <je...@gmail.com>
AuthorDate: Mon Aug 2 12:23:37 2021 +0800
modify the README.md
---
README.md | 1 -
1 file changed, 1 deletion(-)
diff --git a/README.md b/README.md
index 5325cdd..649319c 100644
--- a/README.md
+++ b/README.md
@@ -7,7 +7,6 @@
## DataStream Example
-
```java
import org.apache.rocketmq.streams.client.transform.DataStream;
[rocketmq-streams] 02/35: add lease、dim and client module
Posted by du...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
duhengforever pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/rocketmq-streams.git
commit b757f8db8141d282dd793d37af9b509bcc54b7ca
Author: 刈刀 <ju...@alibaba-inc.com>
AuthorDate: Mon Aug 2 11:21:45 2021 +0800
add lease、dim and client module
---
.gitignore | 25 ++
README.md | 108 +++++
pom.xml | 410 +++++++++++++++++++
rocketmq-streams-clients/pom.xml | 46 +++
.../rocketmq/streams/client/DataStreamAction.java | 101 +++++
.../rocketmq/streams/client/StreamBuilder.java | 28 ++
.../streams/client/source/DataStreamSource.java | 76 ++++
.../client/strategy/CheckpointStrategy.java | 69 ++++
.../streams/client/strategy/StateStrategy.java | 37 ++
.../rocketmq/streams/client/strategy/Strategy.java | 25 ++
.../streams/client/transform/DataStream.java | 437 +++++++++++++++++++++
.../streams/client/transform/JoinStream.java | 212 ++++++++++
.../streams/client/transform/SplitStream.java | 61 +++
.../streams/client/transform/WindowStream.java | 210 ++++++++++
.../client/transform/window/HoppingWindow.java | 32 ++
.../client/transform/window/SessionWindow.java | 32 ++
.../streams/client/transform/window/Time.java | 45 +++
.../client/transform/window/TumblingWindow.java | 33 ++
.../client/transform/window/WindowInfo.java | 83 ++++
.../rocketmq/streams/client/DBDriverTest.java | 75 ++++
.../rocketmq/streams/client/DataStreamTest.java | 107 +++++
.../apache/rocketmq/streams/client/FilterTest.java | 49 +++
.../apache/rocketmq/streams/client/JoinTest.java | 89 +++++
.../apache/rocketmq/streams/client/LeaseTest.java | 98 +++++
.../rocketmq/streams/client/ORMUtilTest.java | 172 ++++++++
.../apache/rocketmq/streams/client/SplitTest.java | 86 ++++
.../apache/rocketmq/streams/client/UnionTest.java | 82 ++++
.../apache/rocketmq/streams/client/WindowTest.java | 86 ++++
.../client/windows/AbstractWindowFireModeTest.java | 189 +++++++++
.../streams/client/windows/WindowFromFileTest.java | 158 ++++++++
.../streams/client/windows/WindowFromMetaq.java | 47 +++
.../client/windows/WindowHighAvailabilityTest.java | 131 ++++++
.../src/test/resources/log4j.xml | 36 ++
rocketmq-streams-dim/pom.xml | 47 +++
.../apache/rocketmq/streams/dim/DimComponent.java | 63 +++
.../rocketmq/streams/dim/builder/DimBuilder.java | 94 +++++
.../function/expression/InExpressionResource.java | 80 ++++
.../expression/NotInExpressionResource.java | 45 +++
.../dim/function/script/IntelligenceFunction.java | 81 ++++
.../script/IntelligenceNameListFunction.java | 24 ++
.../dim/function/script/NameListFunction.java | 203 ++++++++++
.../rocketmq/streams/dim/index/DimIndex.java | 319 +++++++++++++++
.../rocketmq/streams/dim/index/IndexExecutor.java | 258 ++++++++++++
.../intelligence/AbstractIntelligenceCache.java | 395 +++++++++++++++++++
.../dim/intelligence/AccountIntelligenceCache.java | 77 ++++
.../dim/intelligence/DomainIntelligenceCache.java | 83 ++++
.../dim/intelligence/IPIntelligenceCache.java | 108 +++++
.../dim/intelligence/URLIntelligenceCache.java | 80 ++++
.../rocketmq/streams/dim/model/AbstractDim.java | 312 +++++++++++++++
.../streams/dim/model/BooleanFieldDBDim.java | 55 +++
.../apache/rocketmq/streams/dim/model/DBDim.java | 140 +++++++
.../rocketmq/streams/dim/service/IDimService.java | 65 +++
.../streams/dim/service/impl/DimServiceImpl.java | 92 +++++
.../com/aliyun/service/ConfigureLoaderTest.java | 37 ++
.../com/aliyun/service/ExpressionExecutorTest.java | 80 ++++
.../java/com/aliyun/service/JsonParserTest.java | 40 ++
.../com/aliyun/service/NameListFunctionTest.java | 90 +++++
.../java/com/aliyun/service/TableCompressTest.java | 26 ++
rocketmq-streams-lease/pom.xml | 25 ++
.../rocketmq/streams/lease/LeaseComponent.java | 103 +++++
.../rocketmq/streams/lease/model/LeaseInfo.java | 127 ++++++
.../streams/lease/service/ILeaseGetCallback.java | 30 ++
.../streams/lease/service/ILeaseService.java | 136 +++++++
.../streams/lease/service/ILeaseStorage.java | 73 ++++
.../streams/lease/service/ILeaseStorasge.java | 63 +++
.../lease/service/impl/BasedLesaseImpl.java | 404 +++++++++++++++++++
.../lease/service/impl/LeaseServiceImpl.java | 275 +++++++++++++
.../streams/lease/service/impl/MockLeaseImpl.java | 95 +++++
.../lease/service/storages/DBLeaseStorage.java | 229 +++++++++++
.../rocketmq/streams/lease/LeaseComponentTest.java | 119 ++++++
.../src/test/resources/log4j.xml | 20 +
71 files changed, 8068 insertions(+)
diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..1471653
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,25 @@
+target/
+.DS_Store
+!.mvn/wrapper/maven-wrapper.jar
+
+### STS ###
+.apt_generated
+.classpath
+.factorypath
+.project
+.settings
+.springBeans
+
+### IntelliJ IDEA ###
+.idea
+*.iws
+*.iml
+*.ipr
+
+### NetBeans ###
+nbproject/private/
+build/
+nbbuild/
+dist/
+nbdist/
+
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..5f3cde5
--- /dev/null
+++ b/README.md
@@ -0,0 +1,108 @@
+# Rocketmq Streams
+
+## Features
+
+* 轻量级部署：可以单独部署，也支持集群部署
+* 多种类型的数据输入以及输出，source支持 rocketmq ， sink支持db, rocketmq 等
+
+## DataStream Example
+
+```java
+import org.apache.rocketmq.streams.client.transform.DataStream;
+
+DataStreamSource source=StreamBuilder.dataStream("namespace","pipeline");
+
+ source
+ .fromFile("/Users/junjie.cheng/text.txt",false)
+ .map(message->message)
+ .toPrint(1)
+ .start();
+```
+
+## Maven Repository
+
+```xml
+
+<dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams-clients</artifactId>
+ <version>2.0.0-SNAPSHOT</version>
+</dependency>
+```
+
+# Core API
+
+rocketmq-stream 实现了一系列高级的API，可以让用户很方便的编写流计算的程序，实现自己的业务需求；
+
+## StreamBuilder
+
+StreamBuilder 用于构建流任务的源； 内部包含```dataStream()```和```tableStream()```俩个方法，分别返回DataStreamSource和TableStreamSource俩个源；
+
++ [dataStream(nameSpaceName,pipelineName)]() 返回DataStreamSource实例，用于分段编程实现流计算任务；
+
+## DataStream API
+
+### Source
+
+DataStreamSource 是分段式编程的源头类，用于对接各种数据源， 从各大消息队列中获取数据；
+
++ ```fromFile``` 从文件中读取数据， 该方法包含俩个参数
+ + ```filePath``` 文件路径，必填参数
+ + ```isJsonData``` 是否json数据， 非必填参数， 默认为```true```
+
+
++ ```fromRocketmq``` 从rocketmq中获取数据，包含四个参数
+ + ```topic``` rocketmq消息队列的topic名称，必填参数
+ + ```groupName``` 消费者组的名称，必填参数
+ + ```isJson``` 是否json格式，非必填参数
+ + ```tags``` rocketmq消费的tags值，用于过滤消息，非必填参数
+
+
++ ```from``` 自定义的数据源， 通过实现ISource接口实现自己的数据源
+
+### transform
+
+transform 允许在流计算过程中对输入源的数据进行修改，进行下一步的操作；DataStream API中包括```DataStream```,```JoinStream```, ```SplitStream```,```WindowStream```等多个transform类；
+
+#### DataStream
+
+DataStream实现了一系列常见的流计算算子
+
++ ```map``` 通过将源的每个记录传递给函数func来返回一个新的DataStream
++ ```flatmap``` 与map类似，一个输入项对应0个或者多个输出项
++ ```filter``` 只选择func返回true的源DStream的记录来返回一个新的DStream
++ ```forEach``` 对每个记录执行一次函数func， 返回一个新的DataStream
++ ```selectFields``` 对每个记录返回对应的字段值，返回一个新的DataStream
++ ```operate``` 对每个记录执行一次自定义的函数，返回一个新的DataStream
++ ```script``` 针对每个记录的字段执行一段脚本，返回新的字段，生成一个新的DataStream
++ ```toPrint``` 将结果在控制台打印，生成新的DataStreamAction实例
++ ```toFile``` 将结果保存为文件，生成一个新的DataStreamAction实例
++ ```toDB``` 将结果保存到数据库
++ ```toRocketmq``` 将结果输出到rocketmq
++ ```toSls``` 将结果输出到sls
++ ```to``` 将结果经过自定义的ISink接口输出到指定的存储
++ ```window``` 在窗口内进行相关的统计分析，一般会与```groupBy```连用， ```window()```用来定义窗口的大小， ```groupBy()```用来定义统计分析的主key，可以指定多个
+ + ```count``` 在窗口内计数
+ + ```min``` 获取窗口内统计值的最小值
+ + ```max``` 获取窗口内统计值得最大值
+ + ```avg``` 获取窗口内统计值的平均值
+ + ```sum``` 获取窗口内统计值的加和值
+ + ```reduce``` 在窗口内进行自定义的汇总运算
++ ```join``` 根据条件将将俩个流进行关联， 合并为一个大流进行相关的运算
++ ```union``` 将俩个流进行合并
++ ```split``` 将一个数据流按照标签进行拆分，分为不同的数据流供下游进行分析计算
++ ```with``` with算子用来指定计算过程中的相关策略，包括checkpoint的存储策略，state的存储策略等
+
+# Strategy
+
+策略机制主要用来控制计算引擎运行过程中的底层逻辑，如checkpoint，state的存储方式等，后续还会增加对窗口、双流join等的控制；所有的控制策略通过```with```算子传入，可以同时传入多个策略类型；
+
+```java
+//指定checkpoint的存储策略
+source
+ .fromRocketmq("TSG_META_INFO","")
+ .map(message->message+"--")
+ .toPrint(1)
+ .with(CheckpointStrategy.db("jdbc:mysql://XXXXX:3306/XXXXX","","",0L))
+ .start();
+```
\ No newline at end of file
diff --git a/pom.xml b/pom.xml
new file mode 100644
index 0000000..3675792
--- /dev/null
+++ b/pom.xml
@@ -0,0 +1,410 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams</artifactId>
+ <version>2.0.0-SNAPSHOT</version>
+ <name>ROCKETMQ STREAMS</name>
+ <packaging>pom</packaging>
+
+ <modules>
+ <module>rocketmq-streams-commons</module>
+ <module>rocketmq-streams-dim</module>
+ <module>rocketmq-streams-transport-minio</module>
+ <module>rocketmq-streams-script</module>
+ <module>rocketmq-streams-script-python</module>
+ <module>rocketmq-streams-configurable</module>
+ <module>rocketmq-streams-serviceloader</module>
+ <module>rocketmq-streams-filter</module>
+ <module>rocketmq-streams-schedule</module>
+ <module>rocketmq-streams-lease</module>
+ <module>rocketmq-streams-db-operator</module>
+ <module>rocketmq-streams-window</module>
+ <module>rocketmq-streams-clients</module>
+ <module>rocketmq-streams-channel-rocketmq</module>
+ <module>rocketmq-streams-channel-db</module>
+ <module>rocketmq-streams-channel-http</module>
+ </modules>
+
+ <properties>
+ <java.version>1.8</java.version>
+ <java.encoding>UTF-8</java.encoding>
+ <project.build.sourceEncoding>${java.encoding}</project.build.sourceEncoding>
+ <log4j.version>1.2.17</log4j.version>
+ <commons-logging.version>1.1</commons-logging.version>
+ <spring.version>3.2.13.RELEASE</spring.version>
+ <auto-service.version>1.0-rc5</auto-service.version>
+ <mysql-connector.version>5.1.40</mysql-connector.version>
+ <fastjson.version>1.2.27</fastjson.version>
+ <quartz.version>2.2.1</quartz.version>
+ <httpclient.version>4.5.2</httpclient.version>
+ <commons-io.version>2.5</commons-io.version>
+ <junit.version>4.12</junit.version>
+ <guava.version>25.1-jre</guava.version>
+ <groovy.version>2.1.8</groovy.version>
+ <disruptor.version>3.2.0</disruptor.version>
+ <rocksdbjni.version>6.6.4</rocksdbjni.version>
+ <rocketmq.version>4.5.2</rocketmq.version>
+ <hyperscan.version>5.4.0-2.0.0</hyperscan.version>
+ <platform.version>3.5.2</platform.version>
+ <gson.version>2.8.5</gson.version>
+ <java-grok.version>0.1.9</java-grok.version>
+ <jython.version>2.7.0</jython.version>
+ <scala-library.version>2.12.4</scala-library.version>
+ <logback-core.version>1.2.2</logback-core.version>
+ <minio.version>3.0.10</minio.version>
+ </properties>
+
+
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-surefire-plugin</artifactId>
+ <version>2.18.1</version>
+ <configuration>
+ <skipTests>true</skipTests>
+ </configuration>
+ </plugin>
+
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-compiler-plugin</artifactId>
+ <version>2.5.1</version>
+ <configuration>
+ <source>${java.version}</source>
+ <target>${java.version}</target>
+ <encoding>UTF-8</encoding>
+ <skip>true</skip>
+ </configuration>
+ </plugin>
+ <plugin>
+ <groupId>org.codehaus.mojo</groupId>
+ <artifactId>versions-maven-plugin</artifactId>
+ <version>2.2</version>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-source-plugin</artifactId>
+ <version>3.0.1</version>
+ <executions>
+ <execution>
+ <id>attach-sources</id>
+ <goals>
+ <goal>jar</goal>
+ </goals>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+
+ <dependencyManagement>
+ <dependencies>
+ <!-- ================================================= -->
+ <!-- rocketmq streams library -->
+ <!-- ================================================= -->
+ <dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams-commons</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams-configurable</artifactId>
+ <version>${project.version}</version>
+ <exclusions>
+ <exclusion>
+ <groupId>ch.qos.logback</groupId>
+ <artifactId>logback-classic</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>ch.qos.logback</groupId>
+ <artifactId>logback-core</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams-db-operator</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams-dim</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams-filter</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams-lease</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams-schedule</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams-script</artifactId>
+ <version>${project.version}</version>
+ <exclusions>
+ <exclusion>
+ <groupId>ch.qos.logback</groupId>
+ <artifactId>logback-classic</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>ch.qos.logback</groupId>
+ <artifactId>logback-core</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams-script-python</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams-serviceloader</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams-transport-minio</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams-window</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams-channel-db</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams-channel-http</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams-channel-rocketmq</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+
+ <!-- ================================================= -->
+ <!-- rocketmq library -->
+ <!-- ================================================= -->
+
+ <dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-tools</artifactId>
+ <version>${rocketmq.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-common</artifactId>
+ <version>${rocketmq.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-client</artifactId>
+ <version>${rocketmq.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-acl</artifactId>
+ <version>${rocketmq.version}</version>
+ </dependency>
+
+ <!-- ================================================= -->
+ <!-- tool library -->
+ <!-- ================================================= -->
+
+ <dependency>
+ <groupId>com.alibaba</groupId>
+ <artifactId>fastjson</artifactId>
+ <version>${fastjson.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>junit</groupId>
+ <artifactId>junit</artifactId>
+ <version>${junit.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>commons-logging</groupId>
+ <artifactId>commons-logging</artifactId>
+ <version>${commons-logging.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>commons</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>commons-io</groupId>
+ <artifactId>commons-io</artifactId>
+ <version>${commons-io.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>log4j</groupId>
+ <artifactId>log4j</artifactId>
+ <version>${log4j.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>com.google.code.gson</groupId>
+ <artifactId>gson</artifactId>
+ <version>${gson.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ <version>${guava.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>com.google.auto.service</groupId>
+ <artifactId>auto-service</artifactId>
+ <version>${auto-service.version}</version>
+ <exclusions>
+ <exclusion>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <dependency>
+ <groupId>com.lmax</groupId>
+ <artifactId>disruptor</artifactId>
+ <version>${disruptor.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>com.gliwka.hyperscan</groupId>
+ <artifactId>hyperscan</artifactId>
+ <version>${hyperscan.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>net.java.dev.jna</groupId>
+ <artifactId>platform</artifactId>
+ <version>${platform.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.springframework</groupId>
+ <artifactId>spring-jdbc</artifactId>
+ <version>${spring.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>mysql</groupId>
+ <artifactId>mysql-connector-java</artifactId>
+ <version>${mysql-connector.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.quartz-scheduler</groupId>
+ <artifactId>quartz</artifactId>
+ <version>${quartz.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.quartz-scheduler</groupId>
+ <artifactId>quartz-jobs</artifactId>
+ <version>${quartz.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>io.krakens</groupId>
+ <artifactId>java-grok</artifactId>
+ <version>${java-grok.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.codehaus.groovy</groupId>
+ <artifactId>groovy-all</artifactId>
+ <version>${groovy.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.python</groupId>
+ <artifactId>jython-standalone</artifactId>
+ <version>${jython.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.httpcomponents</groupId>
+ <artifactId>httpclient</artifactId>
+ <version>${httpclient.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.httpcomponents</groupId>
+ <artifactId>httpmime</artifactId>
+ <version>${httpclient.version}</version>
+ </dependency>
+
+
+ <dependency>
+ <groupId>org.scala-lang</groupId>
+ <artifactId>scala-library</artifactId>
+ <version>${scala-library.version}</version>
+ </dependency>
+
+
+ <dependency>
+ <groupId>ch.qos.logback</groupId>
+ <artifactId>logback-core</artifactId>
+ <version>${logback-core.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>io.minio</groupId>
+ <artifactId>minio</artifactId>
+ <version>${minio.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.rocksdb</groupId>
+ <artifactId>rocksdbjni</artifactId>
+ <version>${rocksdbjni.version}</version>
+ </dependency>
+
+ </dependencies>
+ </dependencyManagement>
+
+</project>
diff --git a/rocketmq-streams-clients/pom.xml b/rocketmq-streams-clients/pom.xml
new file mode 100644
index 0000000..ffc051a
--- /dev/null
+++ b/rocketmq-streams-clients/pom.xml
@@ -0,0 +1,46 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>rocketmq-streams</artifactId>
+ <groupId>org.apache.rocketmq</groupId>
+ <version>2.0.0-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>rocketmq-streams-clients</artifactId>
+ <name>ROCKETMQ STREAMS :: clients</name>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams-commons</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams-channel-rocketmq</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams-channel-db</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams-script</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams-filter</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams-window</artifactId>
+ </dependency>
+ </dependencies>
+
+ <properties>
+ <maven.compiler.source>8</maven.compiler.source>
+ <maven.compiler.target>8</maven.compiler.target>
+ </properties>
+
+</project>
\ No newline at end of file
diff --git a/rocketmq-streams-clients/src/main/java/org/apache/rocketmq/streams/client/DataStreamAction.java b/rocketmq-streams-clients/src/main/java/org/apache/rocketmq/streams/client/DataStreamAction.java
new file mode 100644
index 0000000..105bd8e
--- /dev/null
+++ b/rocketmq-streams-clients/src/main/java/org/apache/rocketmq/streams/client/DataStreamAction.java
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.streams.client;
+
+import com.google.common.collect.Maps;
+import org.apache.rocketmq.streams.client.strategy.Strategy;
+import org.apache.rocketmq.streams.client.transform.DataStream;
+import org.apache.rocketmq.streams.common.component.ComponentCreator;
+import org.apache.rocketmq.streams.common.configure.ConfigureFileKey;
+import org.apache.rocketmq.streams.common.topology.ChainPipeline;
+import org.apache.rocketmq.streams.common.topology.ChainStage;
+import org.apache.rocketmq.streams.common.topology.builder.PipelineBuilder;
+import org.apache.rocketmq.streams.configurable.ConfigurableComponent;
+
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+
+public class DataStreamAction extends DataStream {
+
+ private final Map<String, Object> properties = Maps.newHashMap();
+
+ public DataStreamAction(String namespace, String pipelineName) {
+ super(namespace, pipelineName);
+ }
+
+ public DataStreamAction(PipelineBuilder pipelineBuilder, Set<PipelineBuilder> pipelineBuilders, ChainStage<?> currentChainStage) {
+ super(pipelineBuilder, pipelineBuilders, currentChainStage);
+ }
+
+ public DataStreamAction with(Strategy... strategies) {
+ Properties properties = new Properties();
+ for (Strategy strategy : strategies) {
+ properties.putAll(strategy.getStrategyProperties());
+ }
+ ComponentCreator.createProperties(properties);
+ return this;
+ }
+
+ /**
+ * 启动流任务
+ */
+ public void start() {
+ start(false);
+ }
+
+ /**
+ * 启动流任务
+ */
+ public void asyncStart() {
+ start(true);
+ }
+
+ protected void start(boolean isAsync) {
+ if (this.mainPipelineBuilder == null) {
+ return;
+ }
+ properties.put(ConfigureFileKey.CONNECT_TYPE, "memory");
+ String[] kvs = new String[properties.size()];
+ int i = 0;
+ for (Map.Entry<String, Object> entry : properties.entrySet()) {
+ kvs[i++] = entry.getKey() + ":" + entry.getValue();
+ }
+
+ ConfigurableComponent configurableComponent = ComponentCreator.getComponent(mainPipelineBuilder.getPipelineNameSpace(), ConfigurableComponent.class, kvs);
+ ChainPipeline pipeline = this.mainPipelineBuilder.build(configurableComponent.getService());
+ pipeline.startChannel();
+ if (this.otherPipelineBuilders != null) {
+ for (PipelineBuilder builder : otherPipelineBuilders) {
+ ChainPipeline otherPipeline = builder.build(configurableComponent.getService());
+ otherPipeline.startChannel();
+ }
+ }
+ if (isAsync) {
+ return;
+ }
+ while (true) {
+ try {
+ Thread.sleep(10000);
+ } catch (InterruptedException e) {
+ e.printStackTrace();
+ }
+ }
+ }
+
+}
diff --git a/rocketmq-streams-clients/src/main/java/org/apache/rocketmq/streams/client/StreamBuilder.java b/rocketmq-streams-clients/src/main/java/org/apache/rocketmq/streams/client/StreamBuilder.java
new file mode 100644
index 0000000..f67ee2c
--- /dev/null
+++ b/rocketmq-streams-clients/src/main/java/org/apache/rocketmq/streams/client/StreamBuilder.java
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.streams.client;
+
+import org.apache.rocketmq.streams.client.source.DataStreamSource;
+
+public class StreamBuilder {
+
+ public static DataStreamSource dataStream(String nameSpaceName, String pipelineName) {
+ return DataStreamSource.create(nameSpaceName, pipelineName);
+ }
+
+}
diff --git a/rocketmq-streams-clients/src/main/java/org/apache/rocketmq/streams/client/source/DataStreamSource.java b/rocketmq-streams-clients/src/main/java/org/apache/rocketmq/streams/client/source/DataStreamSource.java
new file mode 100644
index 0000000..8d71c31
--- /dev/null
+++ b/rocketmq-streams-clients/src/main/java/org/apache/rocketmq/streams/client/source/DataStreamSource.java
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.streams.client.source;
+
+import com.google.common.collect.Sets;
+import org.apache.rocketmq.streams.client.transform.DataStream;
+import org.apache.rocketmq.streams.common.channel.impl.file.FileSource;
+import org.apache.rocketmq.streams.common.channel.source.ISource;
+import org.apache.rocketmq.streams.common.topology.builder.PipelineBuilder;
+import org.apache.rocketmq.streams.source.RocketMQSource;
+
+import java.util.Set;
+
+public class DataStreamSource {
+ protected PipelineBuilder mainPipelineBuilder;
+ protected Set<PipelineBuilder> otherPipelineBuilders;
+
+ public DataStreamSource(String namespace, String pipelineName) {
+ this.mainPipelineBuilder = new PipelineBuilder(namespace, pipelineName);
+ this.otherPipelineBuilders = Sets.newHashSet();
+ }
+
+ public static DataStreamSource create(String namespace, String pipelineName) {
+ return new DataStreamSource(namespace, pipelineName);
+ }
+
+ public DataStream fromFile(String filePath) {
+ return fromFile(filePath, true);
+ }
+
+ public DataStream fromFile(String filePath, Boolean isJsonData) {
+ FileSource fileChannel = new FileSource(filePath);
+ fileChannel.setJsonData(isJsonData);
+ this.mainPipelineBuilder.setSource(fileChannel);
+ return new DataStream(this.mainPipelineBuilder, this.otherPipelineBuilders, null);
+ }
+
+ public DataStream fromRocketmq(String topic, String groupName) {
+ return fromRocketmq(topic, groupName, null, false);
+ }
+
+ public DataStream fromRocketmq(String topic, String groupName, boolean isJson) {
+ return fromRocketmq(topic, groupName, null, isJson);
+ }
+
+ public DataStream fromRocketmq(String topic, String groupName, String tags, boolean isJson) {
+ RocketMQSource rocketMQSource = new RocketMQSource();
+ rocketMQSource.setTopic(topic);
+ rocketMQSource.setTags(tags);
+ rocketMQSource.setGroupName(groupName);
+ rocketMQSource.setJsonData(isJson);
+ this.mainPipelineBuilder.setSource(rocketMQSource);
+ return new DataStream(this.mainPipelineBuilder, this.otherPipelineBuilders, null);
+ }
+
+ public DataStream from(ISource<?> source) {
+ this.mainPipelineBuilder.setSource(source);
+ return new DataStream(this.mainPipelineBuilder, this.otherPipelineBuilders, null);
+ }
+
+}
diff --git a/rocketmq-streams-clients/src/main/java/org/apache/rocketmq/streams/client/strategy/CheckpointStrategy.java b/rocketmq-streams-clients/src/main/java/org/apache/rocketmq/streams/client/strategy/CheckpointStrategy.java
new file mode 100644
index 0000000..20494c4
--- /dev/null
+++ b/rocketmq-streams-clients/src/main/java/org/apache/rocketmq/streams/client/strategy/CheckpointStrategy.java
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.client.strategy;
+
+import org.apache.rocketmq.streams.common.component.AbstractComponent;
+import org.apache.rocketmq.streams.common.configurable.IConfigurableService;
+
+import java.util.Properties;
+
+public class CheckpointStrategy implements Strategy {
+
+ private final Properties properties;
+
+ private CheckpointStrategy(Long pollingTime) {
+ properties = new Properties();
+ properties.put(AbstractComponent.CONNECT_TYPE, IConfigurableService.MEMORY_SERVICE_NAME);
+ properties.put(AbstractComponent.POLLING_TIME, pollingTime + "");
+ }
+
+ private CheckpointStrategy(String filePath, Long pollingTime) {
+ properties = new Properties();
+ properties.put(AbstractComponent.CONNECT_TYPE, IConfigurableService.FILE_SERVICE_NAME);
+ properties.put(IConfigurableService.FILE_PATH_NAME, filePath);
+ properties.put(AbstractComponent.POLLING_TIME, pollingTime + "");
+ }
+
+ private CheckpointStrategy(String url, String username, String password, Long pollingTime) {
+ properties = new Properties();
+ properties.put(AbstractComponent.JDBC_DRIVER, AbstractComponent.DEFAULT_JDBC_DRIVER);
+ properties.put(AbstractComponent.JDBC_URL, url);
+ properties.put(AbstractComponent.JDBC_USERNAME, username);
+ properties.put(AbstractComponent.JDBC_PASSWORD, password);
+ properties.put(AbstractComponent.JDBC_TABLE_NAME, AbstractComponent.DEFAULT_JDBC_TABLE_NAME);
+ properties.put(AbstractComponent.POLLING_TIME, pollingTime + "");
+ properties.put(AbstractComponent.CONNECT_TYPE, IConfigurableService.DEFAULT_SERVICE_NAME);
+ }
+
+ @Override
+ public Properties getStrategyProperties() {
+ return this.properties;
+ }
+
+ public static Strategy db(String url, String username, String password, Long pollingTime) {
+ return new CheckpointStrategy(url, username, password, pollingTime);
+ }
+
+ public static Strategy file(String filePath, Long pollingTime) {
+ return new CheckpointStrategy(filePath, pollingTime);
+ }
+
+ public static Strategy mem(Long pollingTime) {
+ return new CheckpointStrategy(pollingTime);
+ }
+
+}
diff --git a/rocketmq-streams-clients/src/main/java/org/apache/rocketmq/streams/client/strategy/StateStrategy.java b/rocketmq-streams-clients/src/main/java/org/apache/rocketmq/streams/client/strategy/StateStrategy.java
new file mode 100644
index 0000000..c647759
--- /dev/null
+++ b/rocketmq-streams-clients/src/main/java/org/apache/rocketmq/streams/client/strategy/StateStrategy.java
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.client.strategy;
+
+import java.util.Properties;
+
+public class StateStrategy implements Strategy {
+
+ private Properties properties;
+
+ private StateStrategy() {
+ }
+
+ @Override
+ public Properties getStrategyProperties() {
+ return this.properties;
+ }
+
+ public static Strategy db() {
+ return new StateStrategy();
+ }
+
+}
diff --git a/rocketmq-streams-clients/src/main/java/org/apache/rocketmq/streams/client/strategy/Strategy.java b/rocketmq-streams-clients/src/main/java/org/apache/rocketmq/streams/client/strategy/Strategy.java
new file mode 100644
index 0000000..727c645
--- /dev/null
+++ b/rocketmq-streams-clients/src/main/java/org/apache/rocketmq/streams/client/strategy/Strategy.java
@@ -0,0 +1,25 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.client.strategy;
+
+import java.util.Properties;
+
+public interface Strategy {
+
+ Properties getStrategyProperties();
+
+}
diff --git a/rocketmq-streams-clients/src/main/java/org/apache/rocketmq/streams/client/transform/DataStream.java b/rocketmq-streams-clients/src/main/java/org/apache/rocketmq/streams/client/transform/DataStream.java
new file mode 100644
index 0000000..41aae96
--- /dev/null
+++ b/rocketmq-streams-clients/src/main/java/org/apache/rocketmq/streams/client/transform/DataStream.java
@@ -0,0 +1,437 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.streams.client.transform;
+
+import com.alibaba.fastjson.JSONObject;
+import com.google.common.collect.Sets;
+import org.apache.rocketmq.streams.client.DataStreamAction;
+import org.apache.rocketmq.streams.client.transform.window.WindowInfo;
+import org.apache.rocketmq.streams.common.channel.impl.OutputPrintChannel;
+import org.apache.rocketmq.streams.common.channel.impl.file.FileSink;
+import org.apache.rocketmq.streams.common.channel.sink.ISink;
+import org.apache.rocketmq.streams.common.component.ComponentCreator;
+import org.apache.rocketmq.streams.common.configure.ConfigureFileKey;
+import org.apache.rocketmq.streams.common.context.AbstractContext;
+import org.apache.rocketmq.streams.common.context.IMessage;
+import org.apache.rocketmq.streams.common.context.MessageHeader;
+import org.apache.rocketmq.streams.common.context.UserDefinedMessage;
+import org.apache.rocketmq.streams.common.functions.*;
+import org.apache.rocketmq.streams.common.topology.ChainPipeline;
+import org.apache.rocketmq.streams.common.topology.ChainStage;
+import org.apache.rocketmq.streams.common.topology.builder.IStageBuilder;
+import org.apache.rocketmq.streams.common.topology.builder.PipelineBuilder;
+import org.apache.rocketmq.streams.common.topology.model.Union;
+import org.apache.rocketmq.streams.common.topology.stages.udf.StageBuilder;
+import org.apache.rocketmq.streams.common.topology.stages.udf.UDFUnionChainStage;
+import org.apache.rocketmq.streams.common.utils.MapKeyUtil;
+import org.apache.rocketmq.streams.configurable.ConfigurableComponent;
+import org.apache.rocketmq.streams.db.sink.DBSink;
+import org.apache.rocketmq.streams.dim.model.DBDim;
+import org.apache.rocketmq.streams.filter.operator.FilterOperator;
+import org.apache.rocketmq.streams.script.operator.impl.ScriptOperator;
+import org.apache.rocketmq.streams.sink.RocketMQSink;
+import org.apache.rocketmq.streams.window.builder.WindowBuilder;
+import org.apache.rocketmq.streams.window.operator.AbstractWindow;
+import org.apache.rocketmq.streams.window.operator.join.JoinWindow;
+
+import java.io.Serializable;
+import java.util.Set;
+
+public class DataStream implements Serializable {
+
+ protected PipelineBuilder mainPipelineBuilder;
+ protected Set<PipelineBuilder> otherPipelineBuilders;
+ protected ChainStage<?> currentChainStage;
+
+ public DataStream(String namespace, String pipelineName) {
+ this.mainPipelineBuilder = new PipelineBuilder(namespace, pipelineName);
+ this.otherPipelineBuilders = Sets.newHashSet();
+ }
+
+ public DataStream(PipelineBuilder pipelineBuilder, Set<PipelineBuilder> pipelineBuilders, ChainStage<?> currentChainStage) {
+ this.mainPipelineBuilder = pipelineBuilder;
+ this.otherPipelineBuilders = pipelineBuilders;
+ this.currentChainStage = currentChainStage;
+ }
+
+ public DataStream script(String script) {
+ ChainStage<?> stage = this.mainPipelineBuilder.createStage(new ScriptOperator(script));
+ this.mainPipelineBuilder.setTopologyStages(currentChainStage, stage);
+ return new DataStream(this.mainPipelineBuilder, this.otherPipelineBuilders, stage);
+ }
+
+ public DataStream filter(String expressions) {
+ ChainStage<?> stage = this.mainPipelineBuilder.createStage(new FilterOperator(expressions));
+ this.mainPipelineBuilder.setTopologyStages(currentChainStage, stage);
+ return new DataStream(this.mainPipelineBuilder, this.otherPipelineBuilders, stage);
+ }
+
+ public <T, O> DataStream map(MapFunction<T, O> mapFunction) {
+ StageBuilder stageBuilder = new StageBuilder() {
+ @Override
+ protected <T> T operate(IMessage message, AbstractContext context) {
+ try {
+ O o = (O)(message.getMessageValue());
+ T result = (T)mapFunction.map(o);
+ if (result != message.getMessageValue()) {
+ if (result instanceof JSONObject) {
+ message.setMessageBody((JSONObject)result);
+ } else {
+ message.setMessageBody(new UserDefinedMessage(result));
+ }
+ }
+ return null;
+ } catch (Exception e) {
+ e.printStackTrace();
+ }
+ return null;
+ }
+ };
+ ChainStage<?> stage = this.mainPipelineBuilder.createStage(stageBuilder);
+ this.mainPipelineBuilder.setTopologyStages(currentChainStage, stage);
+ return new DataStream(this.mainPipelineBuilder, this.otherPipelineBuilders, stage);
+ }
+
+ public <O> DataStream filter(final FilterFunction<O> filterFunction) {
+ StageBuilder mapUDFOperator = new StageBuilder() {
+
+ @Override
+ protected <T> T operate(IMessage message, AbstractContext context) {
+ try {
+ boolean isFilter = filterFunction.filter((O)message.getMessageValue());
+ if (isFilter) {
+ context.breakExecute();
+ }
+ } catch (Exception e) {
+ e.printStackTrace();
+ }
+ return null;
+ }
+ };
+ ChainStage stage = this.mainPipelineBuilder.createStage(mapUDFOperator);
+ this.mainPipelineBuilder.setTopologyStages(currentChainStage, stage);
+ return new DataStream(this.mainPipelineBuilder, this.otherPipelineBuilders, stage);
+ }
+
+ /**
+ * windows streams
+ *
+ * @param windowInfo 通过不同窗口类型动of方法创建，SessionWindow.of(Time.seconds(10))
+ * @return WindowStream
+ */
+ public WindowStream window(WindowInfo windowInfo) {
+ AbstractWindow window = windowInfo.createWindow();
+ ChainStage<?> stage = this.mainPipelineBuilder.createStage(window);
+ this.mainPipelineBuilder.setTopologyStages(currentChainStage, stage);
+ return new WindowStream(window, this.mainPipelineBuilder, this.otherPipelineBuilders, stage);
+ }
+
+ /**
+ * 通用增加stage的方法，低级接口，适合用户自定义stage的场景
+ *
+ * @param stageBuilder 创建stage和变量的接口
+ * @return DataStream
+ */
+ public DataStream addStage(IStageBuilder stageBuilder) {
+ ChainStage<?> stage = this.mainPipelineBuilder.createStage(stageBuilder);
+ this.mainPipelineBuilder.setTopologyStages(currentChainStage, stage);
+ return new DataStream(this.mainPipelineBuilder, this.otherPipelineBuilders, stage);
+ }
+
+ /**
+ * 创建join stream。实现原理，通过共享一个JoinWindow，并打标左右流，在join windown完成缓存，join逻辑
+ *
+ * @param rightStream 通过不同窗口类型动of方法创建，SessionWindow.of(Time.seconds(10))
+ * @return
+ */
+ public JoinStream join(DataStream rightStream) {
+ JoinWindow window = WindowBuilder.createDefaultJoinWindow();
+ //处理左边分支，增加map，主要是增加msg msgRouteFromLable->增加窗口stage
+ ChainStage<?> leftScriptStage = this.mainPipelineBuilder.createStage(new ScriptOperator("setHeader(msgRouteFromLable,'" + MessageHeader.JOIN_LEFT + "')"));
+ this.mainPipelineBuilder.setTopologyStages(currentChainStage, leftScriptStage);
+ this.currentChainStage = leftScriptStage;
+ ChainStage<?> leftWindowStage = this.mainPipelineBuilder.createStage(window);
+ this.mainPipelineBuilder.setTopologyStages(currentChainStage, leftWindowStage);
+
+ //处理右流，右流增加script
+ DataStream dataStream = rightStream.script("setHeader(msgRouteFromLable,'" + MessageHeader.JOIN_RIGHT + "')").addStage(window);
+ //dataStream.addStage(window);
+
+ addOtherDataStream(rightStream);
+ return new JoinStream(window, this.mainPipelineBuilder, this.otherPipelineBuilders, leftWindowStage);
+ }
+
+ /**
+ * 通过共享对象union，完成两个数据汇聚，左流需要设置isMainStream=true
+ *
+ * @param rightStream
+ * @return
+ */
+ public DataStream union(DataStream rightStream) {
+ Union union = new Union();
+
+ //处理左流，做流的isMain设置成true
+ UDFUnionChainStage chainStage = (UDFUnionChainStage)this.mainPipelineBuilder.createStage(union);
+ chainStage.setMainStream(true);
+ this.mainPipelineBuilder.setTopologyStages(currentChainStage, chainStage);
+
+ //处理右流，做流的isMain设置成true
+ rightStream.addStage(union);
+
+ addOtherDataStream(rightStream);
+ return new DataStream(this.mainPipelineBuilder, this.otherPipelineBuilders, chainStage);
+ }
+
+ /**
+ * 把一个流拆分成多个流，通过设置不同流的标签实现
+ *
+ * @param splitFunction 拆分流的具体逻辑
+ * @return
+ */
+ public SplitStream split(SplitFunction splitFunction) {
+ StageBuilder operator = new StageBuilder() {
+ @Override
+ protected <T> T operate(IMessage message, AbstractContext context) {
+ String labelName = splitFunction.split(message.getMessageValue());
+ message.getHeader().addRouteLable(labelName);
+ return null;
+ }
+ };
+ ChainStage<?> stage = this.mainPipelineBuilder.createStage(operator);
+ this.mainPipelineBuilder.setTopologyStages(currentChainStage, stage);
+ return new SplitStream(this.mainPipelineBuilder, this.otherPipelineBuilders, stage);
+ }
+
+ /**
+ * 维表join,mysql场景，不需要指定jdbcdriver
+ *
+ * @param url
+ * @param userName
+ * @param password
+ * @param sqlOrTableName
+ * @return
+ */
+ public JoinStream join(String url, String userName, String password, String sqlOrTableName, long pollingTimeMintue) {
+ return join(url, userName, password, sqlOrTableName, null, pollingTimeMintue);
+ }
+
+ /**
+ * 维表join
+ *
+ * @param url
+ * @param userName
+ * @param password
+ * @param sqlOrTableName
+ * @return
+ */
+ public JoinStream join(String url, String userName, String password, String sqlOrTableName, String jdbcDriver, long pollingTimeMinute) {
+ DBDim dbDim = new DBDim();
+ dbDim.setUrl(url);
+ dbDim.setUserName(userName);
+ dbDim.setPassword(password);
+ dbDim.setSql(sqlOrTableName);
+ dbDim.setPollingTimeMintue(pollingTimeMinute);
+ dbDim.setJdbcdriver(jdbcDriver);
+ this.mainPipelineBuilder.addConfigurables(dbDim);
+ return new JoinStream(dbDim, mainPipelineBuilder, otherPipelineBuilders, currentChainStage);
+ }
+
+ /**
+ * 遍历所有数据
+ *
+ * @param forEachFunction
+ * @param <O>
+ * @return
+ */
+ public <O> DataStream forEach(ForEachFunction<O> forEachFunction) {
+ StageBuilder selfChainStage = new StageBuilder() {
+ @Override
+ protected <T> T operate(IMessage message, AbstractContext context) {
+ forEachFunction.foreach((O)message.getMessageValue());
+ return null;
+ }
+ };
+ ChainStage stage = this.mainPipelineBuilder.createStage(selfChainStage);
+ this.mainPipelineBuilder.setTopologyStages(currentChainStage, stage);
+ return new DataStream(this.mainPipelineBuilder, this.otherPipelineBuilders, stage);
+ }
+
+ /**
+ * 遍历所有数据
+ *
+ * @param forEachFunction
+ * @param <O>
+ * @return
+ */
+ public <O> DataStream forEachMessage(ForEachMessageFunction forEachFunction) {
+ StageBuilder selfChainStage = new StageBuilder() {
+ @Override
+ protected <T> T operate(IMessage message, AbstractContext context) {
+ forEachFunction.foreach(message, context);
+ return null;
+ }
+ };
+ ChainStage stage = this.mainPipelineBuilder.createStage(selfChainStage);
+ this.mainPipelineBuilder.setTopologyStages(currentChainStage, stage);
+ return new DataStream(this.mainPipelineBuilder, this.otherPipelineBuilders, stage);
+ }
+
+ /**
+ * 只保留需要的字段
+ *
+ * @param fieldNames
+ */
+ public DataStream selectFields(String... fieldNames) {
+ ChainStage stage = this.mainPipelineBuilder.createStage(new ScriptOperator("retain(" + MapKeyUtil.createKeyBySign(",", fieldNames) + ")"));
+ this.mainPipelineBuilder.setTopologyStages(currentChainStage, stage);
+ return new DataStream(this.mainPipelineBuilder, this.otherPipelineBuilders, stage);
+ }
+
+ /**
+ * 启动流任务
+ */
+ public void start() {
+ start(false);
+ }
+
+ /**
+ * 启动流任务
+ */
+ public void asynStart() {
+ start(true);
+ }
+
+ protected void start(boolean isAsyn) {
+ if (this.mainPipelineBuilder == null) {
+ return;
+ }
+ ConfigurableComponent configurableComponent = ComponentCreator.getComponent(mainPipelineBuilder.getPipelineNameSpace(), ConfigurableComponent.class, ConfigureFileKey.CONNECT_TYPE + ":memory");
+ ChainPipeline pipeline = this.mainPipelineBuilder.build(configurableComponent.getService());
+ pipeline.startChannel();
+ if (this.otherPipelineBuilders != null) {
+ for (PipelineBuilder builder : otherPipelineBuilders) {
+ ChainPipeline otherPipeline = builder.build(configurableComponent.getService());
+ otherPipeline.startChannel();
+ }
+ }
+ if (isAsyn) {
+ return;
+ }
+ while (true) {
+ try {
+ Thread.sleep(10000);
+ } catch (InterruptedException e) {
+ e.printStackTrace();
+ }
+ }
+ }
+
+ /**
+ * 把其他流的 pipelinebuilder 放到set中
+ *
+ * @param rightSource
+ */
+ protected void addOtherDataStream(DataStream rightSource) {
+ //如果是多流join，需要把把piplinebuider保存下来，在启动时，启动多个pipline
+ if (!rightSource.mainPipelineBuilder.equals(this.mainPipelineBuilder)) {
+ this.otherPipelineBuilders.add(rightSource.mainPipelineBuilder);
+ }
+
+ this.otherPipelineBuilders.addAll(rightSource.otherPipelineBuilders);
+ }
+
+ public DataStreamAction toFile(String filePath) {
+ FileSink fileChannel = new FileSink(filePath);
+ ChainStage<?> output = mainPipelineBuilder.createStage(fileChannel);
+ mainPipelineBuilder.setTopologyStages(currentChainStage, output);
+ return new DataStreamAction(this.mainPipelineBuilder, this.otherPipelineBuilders, output);
+ }
+
+ public DataStreamAction toPrint() {
+ return toPrint(-1);
+ }
+
+ public DataStreamAction toPrint(int batchSize) {
+ OutputPrintChannel outputPrintChannel = new OutputPrintChannel();
+ if (batchSize > 0) {
+ outputPrintChannel.setBatchSize(batchSize);
+ }
+ ChainStage output = this.mainPipelineBuilder.createStage(outputPrintChannel);
+ this.mainPipelineBuilder.setTopologyStages(currentChainStage, output);
+ return new DataStreamAction(this.mainPipelineBuilder, this.otherPipelineBuilders, output);
+ }
+
+ public DataStreamAction toDB(String url, String userName, String password, String tableName) {
+ DBSink dbChannel = new DBSink(url, userName, password);
+ dbChannel.setTableName(tableName);
+ ChainStage<?> output = this.mainPipelineBuilder.createStage(dbChannel);
+ this.mainPipelineBuilder.setTopologyStages(currentChainStage, output);
+ return new DataStreamAction(this.mainPipelineBuilder, this.otherPipelineBuilders, output);
+ }
+
+ public DataStreamAction toMetaq(String topic) {
+ return toMetaq(topic, null);
+ }
+
+ public DataStreamAction toMetaq(String topic, String tags) {
+ return toMetaq(topic, tags, -1);
+ }
+
+ public DataStreamAction toMetaq(String topic, String tags, int batchSize) {
+ RocketMQSink metaqChannel = new RocketMQSink();
+ metaqChannel.setTopic(topic);
+ metaqChannel.setTags(tags);
+ if (batchSize > 0) {
+ metaqChannel.setBatchSize(batchSize);
+ }
+ ChainStage<?> output = this.mainPipelineBuilder.createStage(metaqChannel);
+ this.mainPipelineBuilder.setTopologyStages(currentChainStage, output);
+ return new DataStreamAction(this.mainPipelineBuilder, this.otherPipelineBuilders, output);
+ }
+
+ public DataStreamAction toRocketmq(String topic, String groupName, String endpoint, String namesrvAddr,
+ String accessKey, String secretKey, String instanceId) {
+ return toRocketmq(topic, "*", groupName, namesrvAddr, endpoint, accessKey, secretKey, instanceId);
+ }
+
+ public DataStreamAction toRocketmq(String topic, String tags, String groupName, String endpoint,
+ String namesrvAddr, String accessKey, String secretKey, String instanceId) {
+ return toRocketmq(topic, tags, -1, groupName, namesrvAddr, endpoint, accessKey, secretKey, instanceId);
+ }
+
+ public DataStreamAction toRocketmq(String topic, String tags, int batchSize, String groupName,
+ String endpoint, String namesrvAddr, String accessKey, String secretKey, String instanceId) {
+ RocketMQSink rocketMQSink = new RocketMQSink();
+ rocketMQSink.setTopic(topic);
+ rocketMQSink.setTags(tags);
+ rocketMQSink.setGroupName(groupName);
+ rocketMQSink.setNamesrvAddr(namesrvAddr);
+ if (batchSize > 0) {
+ rocketMQSink.setBatchSize(batchSize);
+ }
+ ChainStage<?> output = this.mainPipelineBuilder.createStage(rocketMQSink);
+ this.mainPipelineBuilder.setTopologyStages(currentChainStage, output);
+ return new DataStreamAction(this.mainPipelineBuilder, this.otherPipelineBuilders, output);
+ }
+
+ public DataStreamAction to(ISink<?> sink) {
+ ChainStage<?> output = this.mainPipelineBuilder.createStage(sink);
+ this.mainPipelineBuilder.setTopologyStages(currentChainStage, output);
+ return new DataStreamAction(this.mainPipelineBuilder, this.otherPipelineBuilders, output);
+ }
+}
diff --git a/rocketmq-streams-clients/src/main/java/org/apache/rocketmq/streams/client/transform/JoinStream.java b/rocketmq-streams-clients/src/main/java/org/apache/rocketmq/streams/client/transform/JoinStream.java
new file mode 100644
index 0000000..26e6420
--- /dev/null
+++ b/rocketmq-streams-clients/src/main/java/org/apache/rocketmq/streams/client/transform/JoinStream.java
@@ -0,0 +1,212 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.streams.client.transform;
+
+import org.apache.rocketmq.streams.client.transform.window.Time;
+import org.apache.rocketmq.streams.common.model.NameCreator;
+import org.apache.rocketmq.streams.common.topology.ChainStage;
+import org.apache.rocketmq.streams.common.topology.builder.PipelineBuilder;
+import org.apache.rocketmq.streams.dim.model.AbstractDim;
+import org.apache.rocketmq.streams.filter.builder.ExpressionBuilder;
+import org.apache.rocketmq.streams.filter.function.expression.Equals;
+import org.apache.rocketmq.streams.filter.operator.expression.Expression;
+import org.apache.rocketmq.streams.filter.operator.expression.RelationExpression;
+import org.apache.rocketmq.streams.script.operator.impl.ScriptOperator;
+import org.apache.rocketmq.streams.window.operator.join.JoinWindow;
+
+import java.util.*;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+public class JoinStream {
+
+ private static final String INNER_VAR_NAME_PREFIX = "___";
+ protected JoinWindow joinWindow;//完成join 条件的添加
+ protected boolean isDimJoin = false;//是否是维表join
+
+ protected AbstractDim dim;//维度表对象
+ protected String onCondition;//条件
+ protected JoinType joinType;//连接类型
+
+ //用于返回DataStream流
+ protected PipelineBuilder pipelineBuilder;
+ protected Set<PipelineBuilder> otherPipelineBuilders;
+ protected ChainStage<?> currentChainStage;
+
+ /**
+ * 双流join 场景
+ *
+ * @param joinWindow
+ * @param pipelineBuilder
+ * @param pipelineBuilders
+ * @param currentChainStage
+ */
+ public JoinStream(JoinWindow joinWindow, PipelineBuilder pipelineBuilder, Set<PipelineBuilder> pipelineBuilders, ChainStage<?> currentChainStage) {
+ this.pipelineBuilder = pipelineBuilder;
+ this.otherPipelineBuilders = pipelineBuilders;
+ this.currentChainStage = currentChainStage;
+ this.joinWindow = joinWindow;
+ }
+
+ /**
+ * 维表join 场景
+ *
+ * @param pipelineBuilder
+ * @param pipelineBuilders
+ * @param currentChainStage
+ */
+ public JoinStream(AbstractDim dim, PipelineBuilder pipelineBuilder, Set<PipelineBuilder> pipelineBuilders, ChainStage<?> currentChainStage) {
+ this.pipelineBuilder = pipelineBuilder;
+ this.otherPipelineBuilders = pipelineBuilders;
+ this.currentChainStage = currentChainStage;
+ this.dim = dim;
+ }
+
+ public JoinStream setJoinType(JoinType joinType) {
+ this.joinType = joinType;
+
+ return this;
+ }
+
+ /**
+ * 指定窗口，如果不指定，默认1个小时
+ *
+ * @param time
+ * @return
+ */
+ public JoinStream window(Time time) {
+
+ //维表join 不需要设置
+ if (isDimJoin) {
+ throw new RuntimeException("can not support this method");
+ }
+ joinWindow.setTimeUnitAdjust(1);
+ joinWindow.setSizeInterval(time.getValue());
+ joinWindow.setSlideInterval(time.getValue());
+ joinWindow.setRetainWindowCount(1);
+ return this;
+ }
+
+ /**
+ * 增加条件,用表达式形式表达(leftFieldName,function,rightFieldName)&&({name,==,otherName}||(age,==,age)) 后续再增加结构化的方法
+ *
+ * @param onCondition (leftFieldName,function,rightFieldName)&&({name,==,otherName}||(age,==,age))
+ * @return
+ */
+ public JoinStream setCondition(String onCondition) {
+ this.onCondition = onCondition;
+ return this;
+ }
+
+ public DataStream toDataSteam() {
+ if (isDimJoin) {
+ return doDimJoin();
+ } else {
+ return doJoin();
+ }
+
+ }
+
+ /**
+ * 维度表join的场景
+ */
+ protected DataStream doDimJoin() {
+ String script = null;
+ if (JoinType.INNER_JOIN == joinType) {
+ String data = createName("inner_join");
+ script = data + "=inner_join('" + dim.getNameSpace() + "','" + dim.getConfigureName() + "','" + onCondition + "'," + null + ",''," + null + ");splitArray('" + data + "');";
+ } else if ((JoinType.LEFT_JOIN == joinType)) {
+ String data = createName("left_join");
+ script = data + "=left_join('" + dim.getNameSpace() + "','" + dim.getConfigureName() + "','" + onCondition + "'," + null + ",'" + null + "'," + null + ");if(!null(" + data + ")){splitArray('" + data + "');};";
+ }
+ ChainStage stage = this.pipelineBuilder.createStage(new ScriptOperator(script));
+ this.pipelineBuilder.setTopologyStages(currentChainStage, stage);
+ return new DataStream(pipelineBuilder, otherPipelineBuilders, stage);
+ }
+
+ /**
+ * 双流join的场景
+ */
+ protected DataStream doJoin() {
+ if (JoinType.INNER_JOIN == joinType) {
+ joinWindow.setJoinType("INNER");
+ } else if (JoinType.LEFT_JOIN == joinType) {
+ joinWindow.setJoinType("LEFT");
+ } else {
+ throw new RuntimeException("can not support this join type, expect INNER,LEFT, real is " + joinType.toString());
+ }
+
+ AtomicBoolean hasNoEqualsExpression = new AtomicBoolean(false);//是否有非等值的join 条件
+ Map<String, String> left2Right = createJoinFieldsFromCondition(onCondition, hasNoEqualsExpression);//把等值条件的左右字段映射成map
+ List<String> leftList = new ArrayList<>();
+ List<String> rightList = new ArrayList<>();
+ leftList.addAll(left2Right.keySet());
+ rightList.addAll(left2Right.values());
+ joinWindow.setLeftJoinFieldNames(leftList);
+ joinWindow.setRightJoinFieldNames(rightList);
+ //如果有非等值，则把这个条件设置进去
+ if (hasNoEqualsExpression.get()) {
+ joinWindow.setExpression(onCondition);
+ }
+ return new DataStream(pipelineBuilder, otherPipelineBuilders, currentChainStage);
+ }
+
+ /**
+ * 支持的连接类型，目前支持inner join和left join
+ */
+ public enum JoinType {
+ INNER_JOIN,
+ LEFT_JOIN
+ }
+
+ /**
+ * 从条件中找到join 左右的字段。如果有非等值，则不包含在内
+ *
+ * @param
+ * @param onCondition
+ * @return
+ */
+ public Map<String, String> createJoinFieldsFromCondition(String onCondition, AtomicBoolean hasNoEqualsExpression) {
+ List<Expression> expressions = new ArrayList<>();
+ List<RelationExpression> relationExpressions = new ArrayList<>();
+ ExpressionBuilder.createOptimizationExpression("tmp", "tmp", onCondition, expressions, relationExpressions);
+ Map<String, String> left2Right = new HashMap<>();
+ for (Expression expression : expressions) {
+ String varName = expression.getVarName();
+ String valueName = expression.getValue().toString();
+ if (!Equals.isEqualFunction(expression.getFunctionName())) {
+ hasNoEqualsExpression.set(true);
+ continue;
+ }
+ left2Right.put(varName, valueName);
+ }
+ return left2Right;
+ }
+
+ public static String createName(String functionName, String... names) {
+ if (names == null || names.length == 0) {
+ return NameCreator.createNewName(INNER_VAR_NAME_PREFIX, functionName);
+ }
+ String[] values = new String[names.length + 2];
+ values[0] = INNER_VAR_NAME_PREFIX;
+ values[1] = functionName;
+ for (int i = 2; i < values.length; i++) {
+ values[i] = names[i - 2];
+ }
+ return NameCreator.createNewName(values);
+ }
+}
diff --git a/rocketmq-streams-clients/src/main/java/org/apache/rocketmq/streams/client/transform/SplitStream.java b/rocketmq-streams-clients/src/main/java/org/apache/rocketmq/streams/client/transform/SplitStream.java
new file mode 100644
index 0000000..49efcd2
--- /dev/null
+++ b/rocketmq-streams-clients/src/main/java/org/apache/rocketmq/streams/client/transform/SplitStream.java
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.streams.client.transform;
+
+import org.apache.rocketmq.streams.common.context.AbstractContext;
+import org.apache.rocketmq.streams.common.context.IMessage;
+import org.apache.rocketmq.streams.common.topology.ChainStage;
+import org.apache.rocketmq.streams.common.topology.builder.PipelineBuilder;
+import org.apache.rocketmq.streams.common.topology.stages.udf.StageBuilder;
+
+import java.util.Set;
+
+public class SplitStream {
+
+
+ /**
+ * 创建datastream时使用
+ */
+ protected PipelineBuilder pipelineBuilder;
+ protected Set<PipelineBuilder> otherPipelineBuilders;
+ protected ChainStage<?> currentChainStage;
+
+
+ public SplitStream(PipelineBuilder pipelineBuilder, Set<PipelineBuilder> pipelineBuilders, ChainStage<?> currentChainStage) {
+ this.pipelineBuilder = pipelineBuilder;
+ this.otherPipelineBuilders = pipelineBuilders;
+ this.currentChainStage = currentChainStage;
+ }
+
+ /**
+ * 选择一个分支
+ * @param lableName
+ * @return
+ */
+ public DataStream select(String lableName){
+ StageBuilder stage = new StageBuilder() {
+ @Override
+ protected <T> T operate(IMessage message, AbstractContext context) {
+ return null;
+ }
+ };
+ stage.setLabel(lableName);
+ this.pipelineBuilder.setTopologyStages(currentChainStage,stage);
+ return new DataStream(pipelineBuilder,otherPipelineBuilders,stage);
+ }
+}
diff --git a/rocketmq-streams-clients/src/main/java/org/apache/rocketmq/streams/client/transform/WindowStream.java b/rocketmq-streams-clients/src/main/java/org/apache/rocketmq/streams/client/transform/WindowStream.java
new file mode 100644
index 0000000..1f6ea24
--- /dev/null
+++ b/rocketmq-streams-clients/src/main/java/org/apache/rocketmq/streams/client/transform/WindowStream.java
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.streams.client.transform;
+
+import com.alibaba.fastjson.JSONObject;
+import org.apache.rocketmq.streams.common.context.UserDefinedMessage;
+import org.apache.rocketmq.streams.common.functions.ReduceFunction;
+import org.apache.rocketmq.streams.common.topology.ChainStage;
+import org.apache.rocketmq.streams.common.topology.builder.PipelineBuilder;
+import org.apache.rocketmq.streams.common.topology.stages.udf.IReducer;
+import org.apache.rocketmq.streams.common.utils.MapKeyUtil;
+import org.apache.rocketmq.streams.window.operator.AbstractWindow;
+
+import java.util.Set;
+
+/**
+ * 做windown 相关操作 可以同时设置多个统计算子，如count，sum，avg 通过toDataSteam/reduce 返回DataSteam
+ */
+public class WindowStream {
+ //window 对象
+ protected AbstractWindow window;
+
+ /**
+ * 创建datastream时使用
+ */
+ protected PipelineBuilder pipelineBuilder;
+ protected Set<PipelineBuilder> otherPipelineBuilders;
+ protected ChainStage<?> currentChainStage;
+
+ public WindowStream(AbstractWindow window, PipelineBuilder pipelineBuilder, Set<PipelineBuilder> pipelineBuilders, ChainStage<?> currentChainStage) {
+ this.pipelineBuilder = pipelineBuilder;
+ this.otherPipelineBuilders = pipelineBuilders;
+ this.currentChainStage = currentChainStage;
+ this.window = window;
+ }
+
+ /**
+ * 做count算子
+ *
+ * @param asName count结果对应的名字，如 sql中count(1) as c 。asName=c
+ * @return
+ */
+ public WindowStream count(String asName) {
+ window.getSelectMap().put(asName, asName + "=count(" + asName + ")");
+ return this;
+ }
+
+ /**
+ * 做min算子
+ *
+ * @param fieldName 算子需要操作的字段名
+ * @return
+ */
+ public WindowStream min(String fieldName) {
+ window.getSelectMap().put(fieldName, fieldName + "=min(" + fieldName + ")");
+ return this;
+ }
+
+ /**
+ * 做max算子
+ *
+ * @param fieldName 算子需要操作的字段名
+ * @return
+ */
+ public WindowStream max(String fieldName) {
+ window.getSelectMap().put(fieldName, fieldName + "=max(" + fieldName + ")");
+ return this;
+ }
+
+ /**
+ * 做avg算子
+ *
+ * @param fieldName 算子需要操作的字段名
+ * @param asName avg结果对应的名字，如 sql中avg(name) as c 。asName=c
+ * @return
+ */
+ public WindowStream avg(String fieldName, String asName) {
+ window.getSelectMap().put(asName, asName + "=avg(" + fieldName + ")");
+ return this;
+ }
+
+ /**
+ * 做sum算子
+ *
+ * @param fieldName 算子需要操作的字段名
+ * @param asName sum结果对应的名字，如 sql中sum(name) as c 。asName=c
+ * @return
+ */
+ public WindowStream sum(String fieldName, String asName) {
+ window.getSelectMap().put(asName, asName + "=sum(" + fieldName + ")");
+ return this;
+ }
+
+ public WindowStream setTimeField(String timeField) {
+ window.setTimeFieldName(timeField);
+ return this;
+ }
+
+ public WindowStream setFireMode(int fireMode) {
+ window.setFireMode(fireMode);
+ return this;
+ }
+
+ public WindowStream setLocalStorageOnly(boolean isLocalStorageOnley) {
+ window.setLocalStorageOnly(isLocalStorageOnley);
+ return this;
+ }
+
+ public WindowStream setMaxMsgGap(Long maxMsgGapSecond) {
+ window.setMsgMaxGapSecond(maxMsgGapSecond);
+ return this;
+ }
+
+ /**
+ * 以哪几个字段做分组，支持多个字段
+ *
+ * @param fieldNames
+ * @return
+ */
+ public WindowStream groupBy(String... fieldNames) {
+ window.setGroupByFieldName(MapKeyUtil.createKeyBySign(";", fieldNames));
+ for (String fieldName : fieldNames) {
+ window.getSelectMap().put(fieldName, fieldName);
+ }
+
+ return this;
+ }
+
+ /**
+ * 以哪几个字段做分组，支持多个字段
+ *
+ * @param fireMode
+ * @return
+ */
+ public WindowStream fireMode(int fireMode) {
+ window.setFireMode(fireMode);
+
+ return this;
+ }
+
+ /**
+ * 以哪几个字段做分组，支持多个字段
+ *
+ * @param waterMarkSecond
+ * @return
+ */
+ public WindowStream waterMark(int waterMarkSecond) {
+ window.setWaterMarkMinute(waterMarkSecond);
+
+ return this;
+ }
+
+ /**
+ * 以哪几个字段做分组，支持多个字段
+ *
+ * @param fieldName
+ * @return
+ */
+ public WindowStream timeField(String fieldName) {
+ window.setTimeFieldName(fieldName);
+
+ return this;
+ }
+
+ /**
+ * 用户自定义reduce逻辑
+ *
+ * @param reduceFunction
+ * @return
+ */
+ public <R, O> DataStream reduce(ReduceFunction<R, O> reduceFunction) {
+ window.setReducer(new IReducer() {
+ @Override
+ public JSONObject reduce(JSONObject accumulator, JSONObject msg) {
+ Object accumulatorValue = accumulator;
+ Object msgValue = msg;
+ if (msg instanceof UserDefinedMessage) {
+ UserDefinedMessage userDefinedMessage = (UserDefinedMessage)msg;
+ msgValue = userDefinedMessage.getMessageValue();
+ }
+ if (accumulator instanceof UserDefinedMessage) {
+ UserDefinedMessage userDefinedMessage = (UserDefinedMessage)accumulator;
+ accumulatorValue = userDefinedMessage.getMessageValue();
+ }
+ R result = reduceFunction.reduce((R)accumulatorValue, (O)msgValue);
+ return new UserDefinedMessage(result);
+ }
+ });
+ return new DataStream(pipelineBuilder, otherPipelineBuilders, currentChainStage);
+ }
+
+ public DataStream toDataSteam() {
+ return new DataStream(pipelineBuilder, otherPipelineBuilders, currentChainStage);
+ }
+}
diff --git a/rocketmq-streams-clients/src/main/java/org/apache/rocketmq/streams/client/transform/window/HoppingWindow.java b/rocketmq-streams-clients/src/main/java/org/apache/rocketmq/streams/client/transform/window/HoppingWindow.java
new file mode 100644
index 0000000..0c945f0
--- /dev/null
+++ b/rocketmq-streams-clients/src/main/java/org/apache/rocketmq/streams/client/transform/window/HoppingWindow.java
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.streams.client.transform.window;
+
+public class HoppingWindow {
+ /**
+ * 滑动窗口信息
+ * @return
+ */
+ public static WindowInfo of(Time windowSize,Time windowSlide){
+ WindowInfo windowInfo=new WindowInfo();
+ windowInfo.setType(WindowInfo.HOPPING_WINDOW);
+ windowInfo.setWindowSize(windowSize);
+ windowInfo.setWindowSlide(windowSlide);
+ return windowInfo;
+ }
+}
diff --git a/rocketmq-streams-clients/src/main/java/org/apache/rocketmq/streams/client/transform/window/SessionWindow.java b/rocketmq-streams-clients/src/main/java/org/apache/rocketmq/streams/client/transform/window/SessionWindow.java
new file mode 100644
index 0000000..67c7f22
--- /dev/null
+++ b/rocketmq-streams-clients/src/main/java/org/apache/rocketmq/streams/client/transform/window/SessionWindow.java
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.streams.client.transform.window;
+
+public class SessionWindow {
+ /**
+ * 滑动窗口信息
+ * @param time
+ * @return
+ */
+ public static WindowInfo of(Time time){
+ WindowInfo windowInfo=new WindowInfo();
+ windowInfo.setType(WindowInfo.SESSION_WINDOW);
+ windowInfo.setWindowSize(time);
+ return windowInfo;
+ }
+}
diff --git a/rocketmq-streams-clients/src/main/java/org/apache/rocketmq/streams/client/transform/window/Time.java b/rocketmq-streams-clients/src/main/java/org/apache/rocketmq/streams/client/transform/window/Time.java
new file mode 100644
index 0000000..2157e01
--- /dev/null
+++ b/rocketmq-streams-clients/src/main/java/org/apache/rocketmq/streams/client/transform/window/Time.java
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.streams.client.transform.window;
+
+/**
+ * 给window指定窗口时间
+ */
+public class Time {
+ protected int value;
+ public Time(int value){
+ this.value=value;
+ }
+ public static Time seconds(int second){
+
+ return new Time(second);
+ }
+ public static Time minutes(int minutes){
+ return new Time(minutes*60);
+ }
+ public static Time hours(int hours){
+ return new Time(60*60*hours);
+ }
+ public static Time days(int days){
+ return new Time(60*60*24*days);
+ }
+
+ public int getValue() {
+ return value;
+ }
+}
diff --git a/rocketmq-streams-clients/src/main/java/org/apache/rocketmq/streams/client/transform/window/TumblingWindow.java b/rocketmq-streams-clients/src/main/java/org/apache/rocketmq/streams/client/transform/window/TumblingWindow.java
new file mode 100644
index 0000000..62c0ce3
--- /dev/null
+++ b/rocketmq-streams-clients/src/main/java/org/apache/rocketmq/streams/client/transform/window/TumblingWindow.java
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.streams.client.transform.window;
+
+public class TumblingWindow {
+ /**
+ * 滑动窗口信息
+ *
+ * @param time
+ * @return
+ */
+ public static WindowInfo of(Time time) {
+ WindowInfo windowInfo = new WindowInfo();
+ windowInfo.setType(WindowInfo.TUMBLING_WINDOW);
+ windowInfo.setWindowSize(time);
+ return windowInfo;
+ }
+}
diff --git a/rocketmq-streams-clients/src/main/java/org/apache/rocketmq/streams/client/transform/window/WindowInfo.java b/rocketmq-streams-clients/src/main/java/org/apache/rocketmq/streams/client/transform/window/WindowInfo.java
new file mode 100644
index 0000000..64be503
--- /dev/null
+++ b/rocketmq-streams-clients/src/main/java/org/apache/rocketmq/streams/client/transform/window/WindowInfo.java
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.client.transform.window;
+
+import org.apache.rocketmq.streams.window.operator.AbstractWindow;
+import org.apache.rocketmq.streams.window.operator.impl.SessionWindow;
+import org.apache.rocketmq.streams.window.operator.impl.WindowOperator;
+
+/**
+ * 保存创建window的信息 主要是窗口类型，窗口大小
+ */
+public class WindowInfo {
+ public static int HOPPING_WINDOW = 1;//滑动窗口
+ public static int TUMBLING_WINDOW = 2;//滚动窗口
+ public static int SESSION_WINDOW = 23;
+ protected int type;//window类型 hopping，Tumbling
+ protected Time windowSize;//窗口大小
+ protected Time windowSlide;//滑动大小
+
+ /**
+ * 创建窗口
+ *
+ * @return
+ */
+ public AbstractWindow createWindow() {
+ AbstractWindow window = null;
+ if (type == HOPPING_WINDOW) {
+ window = new WindowOperator();
+ window.setTimeUnitAdjust(1);
+ window.setSizeInterval(windowSize.getValue());
+ window.setSlideInterval(windowSlide.getValue());
+ } else if (type == TUMBLING_WINDOW) {
+ window = new WindowOperator();
+ window.setTimeUnitAdjust(1);
+ window.setSizeInterval(windowSize.getValue());
+ } else if (type == SESSION_WINDOW) {
+ window = new SessionWindow();
+ window.setTimeUnitAdjust(1);
+ window.setSizeInterval(windowSize.getValue());
+ } else {
+ throw new RuntimeException("can not support the type ,expect 1，2，3。actual is " + type);
+ }
+ return window;
+ }
+
+ public int getType() {
+ return type;
+ }
+
+ public void setType(int type) {
+ this.type = type;
+ }
+
+ public Time getWindowSize() {
+ return windowSize;
+ }
+
+ public void setWindowSize(Time windowSize) {
+ this.windowSize = windowSize;
+ }
+
+ public Time getWindowSlide() {
+ return windowSlide;
+ }
+
+ public void setWindowSlide(Time windowSlide) {
+ this.windowSlide = windowSlide;
+ }
+}
diff --git a/rocketmq-streams-clients/src/test/java/org/apache/rocketmq/streams/client/DBDriverTest.java b/rocketmq-streams-clients/src/test/java/org/apache/rocketmq/streams/client/DBDriverTest.java
new file mode 100644
index 0000000..7e078f8
--- /dev/null
+++ b/rocketmq-streams-clients/src/test/java/org/apache/rocketmq/streams/client/DBDriverTest.java
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.streams.client;
+
+import org.apache.rocketmq.streams.common.component.ComponentCreator;
+import org.apache.rocketmq.streams.common.configure.ConfigureFileKey;
+import org.apache.rocketmq.streams.configuable.ConfigurableComponent;
+import org.apache.rocketmq.streams.configuable.model.Configure;
+import org.apache.rocketmq.streams.db.driver.DriverBuilder;
+import org.junit.Test;
+
+import static junit.framework.TestCase.assertNotNull;
+import static junit.framework.TestCase.assertTrue;
+
+/**
+ * 数据库的存储，需要配置存储的连接参数，请先完成配置，后执行单元用例 如果未建表，可以通过Configure.createTableSQL() 获取建表语句，创建表后，测试
+ */
+public class DBDriverTest {
+ private String URL = "";
+ protected String USER_NAME = "";
+ protected String PASSWORD = "";
+ protected String TABLE_NAME = "rocketmq_streams_configure_source";
+
+ @Test
+ public void testDBConfigurableService() {
+ String namespace = "streams.db.configurable";
+
+ //正式使用时，在配置文件配置
+ ComponentCreator.getProperties().put(ConfigureFileKey.CONNECT_TYPE, "DB");
+ ComponentCreator.getProperties().put(ConfigureFileKey.JDBC_URL, URL);//数据库连接url
+ ComponentCreator.getProperties().put(ConfigureFileKey.JDBC_USERNAME, USER_NAME);//用户名
+ ComponentCreator.getProperties().put(ConfigureFileKey.JDBC_PASSWORD, PASSWORD);//password
+ ComponentCreator.getProperties().put(ConfigureFileKey.JDBC_TABLE_NAME, TABLE_NAME);
+
+ //如果表不存在，创建表
+ String sql = (Configure.createTableSQL(TABLE_NAME));
+ DriverBuilder.createDriver().execute(sql);
+ ConfigurableComponent configurableComponent = ConfigurableComponent.getInstance(namespace);
+ configurableComponent.insert(createPerson(namespace));
+ configurableComponent.refreshConfigurable(namespace);
+ Person person = configurableComponent.queryConfigurable("person", "peronName");
+ assertNotNull(person);
+ }
+
+ /**
+ * 创建configuable对象
+ *
+ * @param namespace
+ * @return
+ */
+ protected Person createPerson(String namespace) {
+ Person person = new Person();
+ person.setName("chris");
+ person.setAge(18);
+ person.setNameSpace(namespace);
+ person.setConfigureName("peronName");
+ person.setType("person");
+ return person;
+ }
+}
diff --git a/rocketmq-streams-clients/src/test/java/org/apache/rocketmq/streams/client/DataStreamTest.java b/rocketmq-streams-clients/src/test/java/org/apache/rocketmq/streams/client/DataStreamTest.java
new file mode 100644
index 0000000..8f54067
--- /dev/null
+++ b/rocketmq-streams-clients/src/test/java/org/apache/rocketmq/streams/client/DataStreamTest.java
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.streams.client;
+
+import org.apache.rocketmq.streams.client.source.DataStreamSource;
+import org.apache.rocketmq.streams.client.strategy.CheckpointStrategy;
+import org.apache.rocketmq.streams.client.strategy.StateStrategy;
+import org.apache.rocketmq.streams.common.utils.DataTypeUtil;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.Serializable;
+import java.sql.*;
+
+public class DataStreamTest implements Serializable {
+
+ DataStreamSource dataStream;
+
+ @Before
+ public void init() {
+ dataStream = StreamBuilder.dataStream("test_namespace", "graph_pipeline");
+ }
+
+ @Test
+ public void testFromFile() {
+ dataStream
+ .fromFile("/Users/junjie.cheng/text.txt", false)
+ .map(message -> message + "--")
+ .toPrint(1)
+ .start();
+ }
+
+ @Test
+ public void testRocketmq() {
+ dataStream
+ .fromRocketmq("TOPIC_EVENT_SAS_SECURITY_EVENT", "111")
+ .map(message -> message + "--")
+ .toPrint(1)
+ .start();
+ }
+
+ @Test
+ public void testDBCheckPoint() {
+ dataStream
+ .fromRocketmq("TSG_META_INFO", "")
+ .map(message -> message + "--")
+ .toPrint(1)
+ .with(CheckpointStrategy.db("", "", "", 0L))
+ .start();
+ }
+
+ @Test
+ public void testFileCheckPoint() {
+ dataStream
+ .fromRocketmq("TSG_META_INFO", "")
+ .map(message -> message + "--")
+ .toPrint(1)
+ .with(CheckpointStrategy.mem(0L))
+ .start();
+ }
+
+ @Test
+ public void testBothStrategy() {
+ dataStream
+ .fromRocketmq("TSG_META_INFO", "")
+ .map(message -> message + "--")
+ .toPrint(1)
+ .with(CheckpointStrategy.mem(0L), StateStrategy.db())
+ .start();
+ }
+
+ @Test
+ public void testMeta() {
+ try {
+ Class.forName("com.mysql.jdbc.Driver");
+ Connection connection = DriverManager.getConnection("", "", "");
+ DatabaseMetaData metaData = connection.getMetaData();
+ ResultSet dataFilter = metaData.getColumns(connection.getCatalog(), "%", "XXX", null);
+ while (dataFilter.next()) {
+ String one = dataFilter.getString("DATA_TYPE");
+ int two = dataFilter.getInt("COLUMN_SIZE");
+ String three = dataFilter.getString("COLUMN_NAME");
+ String four = dataFilter.getString("TYPE_NAME");
+ System.out.println(one + " " + two + " " + three + " " + four + " " + DataTypeUtil.getDataType(dataFilter.getString("TYPE_NAME")));
+ }
+
+ } catch (ClassNotFoundException | SQLException e) {
+ e.printStackTrace();
+ }
+ }
+
+}
diff --git a/rocketmq-streams-clients/src/test/java/org/apache/rocketmq/streams/client/FilterTest.java b/rocketmq-streams-clients/src/test/java/org/apache/rocketmq/streams/client/FilterTest.java
new file mode 100644
index 0000000..19665fd
--- /dev/null
+++ b/rocketmq-streams-clients/src/test/java/org/apache/rocketmq/streams/client/FilterTest.java
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.streams.client;
+
+import com.alibaba.fastjson.JSONObject;
+import java.util.List;
+import org.apache.rocketmq.streams.filter.FilterComponent;
+import org.apache.rocketmq.streams.filter.builder.ExpressionBuilder;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+import org.junit.Test;
+
+import static junit.framework.TestCase.assertTrue;
+
+public class FilterTest {
+ @Test
+ public void testFilter(){
+ JSONObject msg=new JSONObject();
+ msg.put("name","chris");
+ msg.put("age",18);
+ Rule rule= ExpressionBuilder.createRule("tmp","tmp","(name,==,chris)&(age,>=,18)");
+ FilterComponent filterComponent=FilterComponent.getInstance();
+ List<Rule> fireRules=filterComponent.excuteRule(msg,rule);
+ assertTrue(fireRules.size()==1);
+ }
+
+ @Test
+ public void testFilter2(){
+ JSONObject msg=new JSONObject();
+ msg.put("name","chris");
+ msg.put("age",18);
+ boolean result= ExpressionBuilder.executeExecute("tmp","(name,==,chris)&(age,>,int,18)",msg);
+ assertTrue(!result);
+ }
+}
diff --git a/rocketmq-streams-clients/src/test/java/org/apache/rocketmq/streams/client/JoinTest.java b/rocketmq-streams-clients/src/test/java/org/apache/rocketmq/streams/client/JoinTest.java
new file mode 100644
index 0000000..b221e92
--- /dev/null
+++ b/rocketmq-streams-clients/src/test/java/org/apache/rocketmq/streams/client/JoinTest.java
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.streams.client;
+
+import com.alibaba.fastjson.JSONObject;
+import java.io.Serializable;
+import org.apache.rocketmq.streams.client.transform.DataStream;
+import org.apache.rocketmq.streams.client.transform.JoinStream.JoinType;
+import org.apache.rocketmq.streams.common.functions.FilterFunction;
+import org.junit.Test;
+
+public class JoinTest implements Serializable {
+
+ @Test
+ public void testJoin() {
+ DataStream leftStream = (StreamBuilder.dataStream("namespace", "name")
+ .fromFile("/Users/yuanxiaodong/chris/sls_1000.txt")
+ .filter(new FilterFunction<JSONObject>() {
+
+ @Override
+ public boolean filter(JSONObject value) throws Exception {
+ if (value.getString("ProjectName") == null || value.getString("LogStore") == null) {
+ return true;
+ }
+ return false;
+ }
+ }));
+
+ DataStream rightStream = (StreamBuilder.dataStream("namespace", "name2")
+ .fromFile("/Users/yuanxiaodong/chris/sls_1000.txt")
+ .filter(new FilterFunction<JSONObject>() {
+
+ @Override
+ public boolean filter(JSONObject value) throws Exception {
+ if (value.getString("ProjectName") == null || value.getString("LogStore") == null) {
+ return true;
+ }
+ return false;
+ }
+ }));
+
+ leftStream.join(rightStream).setJoinType(JoinType.INNER_JOIN)
+ .setCondition("(ProjectName,==,ProjectName)&(LogStore,==,LogStore)")
+ .toDataSteam()
+ .toPrint()
+ .start();
+
+ }
+
+ @Test
+ public void testDim() {
+ DataStreamAction stream = (StreamBuilder.dataStream("namespace", "name")
+ .fromFile("/Users/yuanxiaodong/chris/sls_1000.txt")
+ .filter(new FilterFunction<JSONObject>() {
+
+ @Override
+ public boolean filter(JSONObject value) throws Exception {
+ if (value.getString("ProjectName") == null || value.getString("LogStore") == null) {
+ return true;
+ }
+ return false;
+ }
+ }))
+ .join("dburl", "dbUserName", "dbPassowrd", "tableNameOrSQL", 5)
+ .setCondition("(name,==,name)")
+ .toDataSteam()
+ .selectFields("name", "age", "address")
+ .toPrint();
+ stream.start();
+ ;
+ ;
+
+ }
+}
diff --git a/rocketmq-streams-clients/src/test/java/org/apache/rocketmq/streams/client/LeaseTest.java b/rocketmq-streams-clients/src/test/java/org/apache/rocketmq/streams/client/LeaseTest.java
new file mode 100644
index 0000000..1d3ec8a
--- /dev/null
+++ b/rocketmq-streams-clients/src/test/java/org/apache/rocketmq/streams/client/LeaseTest.java
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.streams.client;
+
+import java.util.Date;
+import org.apache.rocketmq.streams.common.component.ComponentCreator;
+import org.apache.rocketmq.streams.common.configure.ConfigureFileKey;
+import org.apache.rocketmq.streams.db.driver.DriverBuilder;
+import org.apache.rocketmq.streams.db.driver.JDBCDriver;
+import org.apache.rocketmq.streams.lease.LeaseComponent;
+import org.apache.rocketmq.streams.lease.model.LeaseInfo;
+import org.apache.rocketmq.streams.lease.service.ILeaseGetCallback;
+import org.junit.Test;
+
+import static org.junit.Assert.assertTrue;
+
+public class LeaseTest {
+
+ private String URL="";
+ protected String USER_NAME="";
+ protected String PASSWORD="";
+
+ public LeaseTest(){
+
+ //正式使用时，在配置文件配置
+ ComponentCreator.getProperties().put(ConfigureFileKey.CONNECT_TYPE,"DB");
+ ComponentCreator.getProperties().put(ConfigureFileKey.JDBC_URL,URL);//数据库连接url
+ ComponentCreator.getProperties().put(ConfigureFileKey.JDBC_USERNAME,USER_NAME);//用户名
+ ComponentCreator.getProperties().put(ConfigureFileKey.JDBC_PASSWORD,PASSWORD);//password
+
+ /**
+ * 创建lease info表
+ */
+ JDBCDriver driver= DriverBuilder.createDriver();
+ driver.execute(LeaseInfo.createTableSQL());
+ }
+
+ @Test
+ public void testLease() throws InterruptedException {
+ String leaseName="lease.test";
+ int leaseTime=5;
+ LeaseComponent.getInstance().getService().startLeaseTask(leaseName, leaseTime, new ILeaseGetCallback() {
+ @Override
+ public void callback(Date nextLeaseDate) {
+ System.out.println("I get lease");
+ }
+ });
+ assertTrue(LeaseComponent.getInstance().getService().hasLease(leaseName));
+ Thread.sleep(5000);
+ assertTrue(LeaseComponent.getInstance().getService().hasLease(leaseName));//会一直续约
+ Thread.sleep(5000);
+ assertTrue(LeaseComponent.getInstance().getService().hasLease(leaseName));//会一直续约
+ }
+
+ @Test
+ public void testLock() throws InterruptedException {
+ String name="dipper";
+ String lockName="lease.test";
+ int leaseTime=5;
+ boolean success=LeaseComponent.getInstance().getService().lock(name,lockName,leaseTime);//锁定5秒钟
+ assertTrue(success);//获取锁
+ Thread.sleep(6000);
+ assertTrue(!LeaseComponent.getInstance().getService().hasHoldLock(name,lockName));//超期释放
+ }
+
+ /**
+ * holdlock是一直持有锁，和租约的区别是，当释放锁后，无其他实例抢占
+ *
+ * @throws InterruptedException
+ */
+ @Test
+ public void testHoldLock() throws InterruptedException {
+ String name="dipper";
+ String lockName="lease.test";
+ int leaseTime=6;
+ boolean success=LeaseComponent.getInstance().getService().holdLock(name,lockName,leaseTime);//锁定5秒钟
+ assertTrue(success);//获取锁
+ Thread.sleep(8000);
+ assertTrue(LeaseComponent.getInstance().getService().hasHoldLock(name,lockName));//会自动续约，不会释放，可以手动释放
+ LeaseComponent.getInstance().getService().unlock(name,lockName);
+ assertTrue(!LeaseComponent.getInstance().getService().hasHoldLock(name,lockName));
+ }
+}
diff --git a/rocketmq-streams-clients/src/test/java/org/apache/rocketmq/streams/client/ORMUtilTest.java b/rocketmq-streams-clients/src/test/java/org/apache/rocketmq/streams/client/ORMUtilTest.java
new file mode 100644
index 0000000..a8ba9ab
--- /dev/null
+++ b/rocketmq-streams-clients/src/test/java/org/apache/rocketmq/streams/client/ORMUtilTest.java
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.streams.client;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.rocketmq.streams.common.component.ComponentCreator;
+import org.apache.rocketmq.streams.common.configure.ConfigureFileKey;
+import org.apache.rocketmq.streams.common.configurable.BasedConfigurable;
+import org.apache.rocketmq.streams.common.configurable.annotation.ENVDependence;
+import org.apache.rocketmq.streams.db.driver.orm.ORMUtil;
+import org.junit.Test;
+
+public class ORMUtilTest {
+ private String URL="";
+ protected String USER_NAME="";
+ protected String PASSWORD="";
+
+ public ORMUtilTest(){
+ //正式使用时，在配置文件配置
+ ComponentCreator.getProperties().put(ConfigureFileKey.JDBC_URL,URL);//数据库连接url
+ ComponentCreator.getProperties().put(ConfigureFileKey.JDBC_USERNAME,USER_NAME);//用户名
+ ComponentCreator.getProperties().put(ConfigureFileKey.JDBC_PASSWORD,PASSWORD);//password
+ }
+ @Test
+ public void testInsert(){
+ String namespace="org.apache.configuable.test";
+ List<Person> personList=new ArrayList<>();
+ for(int i=0;i<10;i++){
+ personList.add(createPerson(namespace,"chris"+i));
+ }
+ /**
+ * 不带数据库连接信息（url,userName,Password），使用ConfiguableComponet的连接信息
+ */
+ ORMUtil.batchIgnoreInto(personList);//批量插入，如果有唯一键冲突，替换
+ ORMUtil.batchIgnoreInto(personList);//批量插入，如果有唯一键冲突，忽略
+ ORMUtil.batchInsertInto(personList);////批量插入，如果有唯一键冲突，跑错
+ }
+
+ @Test
+ public void testQueryList(){
+ Map<String,Integer> paras=new HashMap<>();
+ paras.put("age",18);
+ List<Person> personList=ORMUtil.queryForList("select * from person where age >${age} limit 100",paras,Person.class);
+ }
+
+ @Test
+ public void testQueryOneRow(){
+ Person personPara=new Person();
+ personPara.setAge(18);
+ personPara.setName("chris1");
+ Person person=ORMUtil.queryForObject("select * from person where age =${age} and name='${name}' ",personPara,Person.class,URL,USER_NAME,PASSWORD);
+ }
+
+ /**
+ * 创建configuable对象
+ * @param namespace
+ * @return
+ */
+ protected Person createPerson(String namespace,String name){
+ Person person=new Person();
+ person.setName(name);
+ person.setAge(18);
+ person.setNameSpace(namespace);
+ person.setConfigureName("peronName");
+ person.setType("person");
+ return person;
+ }
+}
+
+
+class Person extends BasedConfigurable {
+ @ENVDependence
+ private String name;
+ private int age;
+ private Boolean isMale;
+ private List<String> addresses;
+ private Map<String, Integer> childName2Age;
+
+ public static Person createPerson(String namespace) {
+ Person person = new Person();
+ person.setNameSpace(namespace);
+ person.setType("person");
+ person.setConfigureName("Chris");
+ person.setName("Chris");
+ List<String> addresses = new ArrayList<>();
+ addresses.add("huilongguan");
+ addresses.add("shangdi");
+ person.setAddresses(addresses);
+ Map<String, Integer> childName2Age = new HashMap<>();
+ childName2Age.put("yuanyahan", 8);
+ childName2Age.put("yuanruxi", 4);
+ person.setChildName2Age(childName2Age);
+ person.setMale(true);
+ person.setAge(18);
+ return person;
+ }
+
+ @Override
+ public String toString() {
+ return "org.apache.rocketmq.streams.Person{" + "name='" + name + '\'' + ", age=" + age + ", isMale=" + isMale + ", addresses=" + addresses
+ + ", childName2Age=" + childName2Age + '}';
+ }
+
+ public String getName() {
+ return name;
+ }
+
+ public void setName(String name) {
+ this.name = name;
+ }
+
+ public int getAge() {
+ return age;
+ }
+
+ public void setAge(int age) {
+ this.age = age;
+ }
+
+ public Boolean getMale() {
+ return isMale;
+ }
+
+ public void setMale(Boolean male) {
+ isMale = male;
+ }
+
+ public List<String> getAddresses() {
+ return addresses;
+ }
+
+ public void setAddresses(List<String> addresses) {
+ this.addresses = addresses;
+ }
+
+ public Map<String, Integer> getChildName2Age() {
+ return childName2Age;
+ }
+
+ public void setChildName2Age(Map<String, Integer> childName2Age) {
+ this.childName2Age = childName2Age;
+ }
+
+ @Override
+ public Object clone() {
+ Person person = null;
+ try {
+ person = (Person)super.clone();
+ } catch (CloneNotSupportedException e) {
+ System.out.println("clone error " + e);
+ }
+ return person;
+ }
+}
diff --git a/rocketmq-streams-clients/src/test/java/org/apache/rocketmq/streams/client/SplitTest.java b/rocketmq-streams-clients/src/test/java/org/apache/rocketmq/streams/client/SplitTest.java
new file mode 100644
index 0000000..56ab5ae
--- /dev/null
+++ b/rocketmq-streams-clients/src/test/java/org/apache/rocketmq/streams/client/SplitTest.java
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.streams.client;
+
+import com.alibaba.fastjson.JSONObject;
+import java.io.Serializable;
+import org.apache.rocketmq.streams.client.transform.DataStream;
+import org.apache.rocketmq.streams.client.transform.SplitStream;
+import org.apache.rocketmq.streams.common.functions.FilterFunction;
+import org.apache.rocketmq.streams.common.functions.SplitFunction;
+import org.junit.Test;
+
+public class SplitTest implements Serializable {
+
+ @Test
+ public void testUnion() {
+ DataStream stream = (StreamBuilder.dataStream("namespace", "name")
+ .fromFile("/Users/yuanxiaodong/chris/sls_1000.txt")
+ .filter(new FilterFunction<JSONObject>() {
+
+ @Override
+ public boolean filter(JSONObject value) throws Exception {
+ if (value.getString("ProjectName") == null || value.getString("LogStore") == null) {
+ return true;
+ }
+ return false;
+ }
+ }));
+
+ SplitStream splitStream = stream.split(new SplitFunction<JSONObject>() {
+ @Override
+ public String split(JSONObject o) {
+ if (o.getInteger("age") < 18) {
+ return "children";
+ } else if (o.getInteger("age") >= 18) {
+ return "adult";
+ }
+ return null;
+ }
+ });
+
+ DataStream children = splitStream.select("children");
+ DataStream adult = splitStream.select("adult");
+ children.union(adult)
+ .toPrint()
+ .start();
+
+ }
+
+ @Test
+ public void testDim() {
+ DataStreamAction stream = (StreamBuilder.dataStream("namespace", "name")
+ .fromFile("/Users/yuanxiaodong/chris/sls_1000.txt")
+ .filter(new FilterFunction<JSONObject>() {
+
+ @Override
+ public boolean filter(JSONObject value) throws Exception {
+ if (value.getString("ProjectName") == null || value.getString("LogStore") == null) {
+ return true;
+ }
+ return false;
+ }
+ }))
+ .join("dburl", "dbUserName", "dbPassowrd", "tableNameOrSQL", 5)
+ .setCondition("(name,==,name)")
+ .toDataSteam()
+ .selectFields("name", "age", "address")
+ .toPrint();
+
+ }
+}
diff --git a/rocketmq-streams-clients/src/test/java/org/apache/rocketmq/streams/client/UnionTest.java b/rocketmq-streams-clients/src/test/java/org/apache/rocketmq/streams/client/UnionTest.java
new file mode 100644
index 0000000..1b49185
--- /dev/null
+++ b/rocketmq-streams-clients/src/test/java/org/apache/rocketmq/streams/client/UnionTest.java
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.streams.client;
+
+import com.alibaba.fastjson.JSONObject;
+import java.io.Serializable;
+import org.apache.rocketmq.streams.client.transform.DataStream;
+import org.apache.rocketmq.streams.common.functions.FilterFunction;
+import org.junit.Test;
+
+public class UnionTest implements Serializable {
+
+ @Test
+ public void testUnion() {
+ DataStream leftStream = (StreamBuilder.dataStream("namespace", "name")
+ .fromFile("/Users/yuanxiaodong/chris/sls_1000.txt")
+ .filter(new FilterFunction<JSONObject>() {
+
+ @Override
+ public boolean filter(JSONObject value) throws Exception {
+ if (value.getString("ProjectName") == null || value.getString("LogStore") == null) {
+ return true;
+ }
+ return false;
+ }
+ }));
+
+ DataStream rightStream = (StreamBuilder.dataStream("namespace", "name2")
+ .fromFile("/Users/yuanxiaodong/chris/sls_1000.txt")
+ .filter(new FilterFunction<JSONObject>() {
+
+ @Override
+ public boolean filter(JSONObject value) throws Exception {
+ if (value.getString("ProjectName") == null || value.getString("LogStore") == null) {
+ return true;
+ }
+ return false;
+ }
+ }));
+
+ leftStream.union(rightStream).toPrint().start();
+ }
+
+ @Test
+ public void testDim() {
+ DataStreamAction stream = (StreamBuilder.dataStream("namespace", "name")
+ .fromFile("/Users/yuanxiaodong/chris/sls_1000.txt")
+ .filter(new FilterFunction<JSONObject>() {
+
+ @Override
+ public boolean filter(JSONObject value) throws Exception {
+ if (value.getString("ProjectName") == null || value.getString("LogStore") == null) {
+ return true;
+ }
+ return false;
+ }
+ }))
+ .join("dburl", "dbUserName", "dbPassowrd", "tableNameOrSQL", 5)
+ .setCondition("(name,==,name)")
+ .toDataSteam()
+ .selectFields("name", "age", "address")
+ .toPrint();
+
+ stream.start();
+
+ }
+}
diff --git a/rocketmq-streams-clients/src/test/java/org/apache/rocketmq/streams/client/WindowTest.java b/rocketmq-streams-clients/src/test/java/org/apache/rocketmq/streams/client/WindowTest.java
new file mode 100644
index 0000000..69d98b2
--- /dev/null
+++ b/rocketmq-streams-clients/src/test/java/org/apache/rocketmq/streams/client/WindowTest.java
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.streams.client;
+
+import com.alibaba.fastjson.JSONObject;
+
+import java.io.Serializable;
+
+import org.apache.rocketmq.streams.client.transform.window.Time;
+import org.apache.rocketmq.streams.client.transform.window.TumblingWindow;
+import org.apache.rocketmq.streams.common.functions.ForEachFunction;
+import org.apache.rocketmq.streams.common.functions.MapFunction;
+import org.junit.Test;
+
+public class WindowTest implements Serializable {
+
+ @Test
+ public void testWindow() {
+ StreamBuilder.dataStream("namespace", "name")
+ .fromFile("/Users/duheng/project/opensource/sls_100.txt", false)
+ .map((MapFunction<JSONObject, String>)message -> JSONObject.parseObject(message))
+ .window(TumblingWindow.of(Time.seconds(5)))
+ .groupBy("ProjectName", "LogStore")
+ .setLocalStorageOnly(true)
+ .count("total")
+ .sum("OutFlow", "OutFlow")
+ .sum("InFlow", "InFlow")
+ .toDataSteam()
+ .forEach(new ForEachFunction<JSONObject>() {
+ protected int sum = 0;
+
+ @Override
+ public void foreach(JSONObject o) {
+ int total = o.getInteger("total");
+ sum = sum + total;
+ o.put("sum(total)", sum);
+ }
+ }).toPrint().start();
+
+ }
+
+ // @Test
+ // public void testWindowFromMetaq() throws InterruptedException {
+ // String topic = "TOPIC_DIPPER_SYSTEM_MSG_4";
+ // StreamBuilder.dataStream("namespace", "name")
+ // .fromFile("/Users/yuanxiaodong/chris/sls_100.txt", true)
+ // .toRocketmq(topic)
+ // .asyncStart();
+ //
+ // StreamBuilder.dataStream("namespace", "name1")
+ // .fromRocketmq(topic, "chris", true)
+ // .window(TumblingWindow.of(Time.seconds(5)))
+ // .groupby("ProjectName", "LogStore")
+ // .setLocalStorageOnly(true)
+ // .count("total")
+ // .sum("OutFlow", "OutFlow")
+ // .sum("InFlow", "inflow")
+ // .toDataSteam()
+ // .forEach(new ForEachFunction<JSONObject>() {
+ // protected int sum = 0;
+ //
+ // @Override
+ // public void foreach(JSONObject o) {
+ // int total = o.getInteger("total");
+ // sum = sum + total;
+ // o.put("sum(total)", sum);
+ // }
+ // }).toPrint().start();
+ // }
+
+}
diff --git a/rocketmq-streams-clients/src/test/java/org/apache/rocketmq/streams/client/windows/AbstractWindowFireModeTest.java b/rocketmq-streams-clients/src/test/java/org/apache/rocketmq/streams/client/windows/AbstractWindowFireModeTest.java
new file mode 100644
index 0000000..c102f1f
--- /dev/null
+++ b/rocketmq-streams-clients/src/test/java/org/apache/rocketmq/streams/client/windows/AbstractWindowFireModeTest.java
@@ -0,0 +1,189 @@
+package org.apache.rocketmq.streams.client.windows;
+
+import java.io.Serializable;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import com.alibaba.fastjson.JSONObject;
+
+import org.apache.rocketmq.streams.client.transform.DataStream;
+import org.apache.rocketmq.streams.client.transform.window.Time;
+import org.apache.rocketmq.streams.client.transform.window.TumblingWindow;
+import org.apache.rocketmq.streams.common.functions.ForEachFunction;
+import org.apache.rocketmq.streams.common.functions.MapFunction;
+import org.apache.rocketmq.streams.common.utils.DateUtil;
+
+
+public abstract class AbstractWindowFireModeTest implements Serializable {
+ protected Date date=new Date();
+ public AbstractWindowFireModeTest(){
+
+ date.setYear(2021-1900);
+ date.setMonth(6);
+ date.setDate(14);
+ date.setHours(12);
+ date.setMinutes(1);
+ date.setSeconds(0);
+
+ }
+
+ public void testWindowFireMode0(boolean isLocalOnly) throws InterruptedException {
+ testWindowFireMode1(isLocalOnly,5);
+ }
+
+ public void testWindowFireMode0(boolean isLocalOnly,int windowSize) throws InterruptedException {
+
+ createSourceDataStream().map(new MapFunction<JSONObject, String>() {
+ int count=0;
+ Long time=null;
+ @Override
+ public JSONObject map(String message) throws Exception {
+
+ if(time==null){
+ time=date.getTime();
+ }else {
+ time+=count;
+ }
+ count++;
+ JSONObject msg=JSONObject.parseObject(message);
+
+ msg.put("logTime",time);
+
+ return msg;
+ }
+ })
+ .window(TumblingWindow.of(Time.seconds(windowSize)))
+ .groupBy("ProjectName", "LogStore")
+ .setLocalStorageOnly(isLocalOnly)
+ .setMaxMsgGap(isLocalOnly?10L:20L)
+ .setTimeField("logTime")
+ .count("total")
+ .sum("OutFlow", "OutFlow")
+ .sum("InFlow", "inflow")
+ .toDataSteam()
+ .forEach(new ForEachFunction<JSONObject>() {
+ AtomicInteger sum = new AtomicInteger(0) ;
+ @Override
+ public synchronized void foreach(JSONObject o) {
+ int total = o.getInteger("total");
+ o.put("sum(total)", sum.addAndGet(total));
+ }
+ }).toPrint().start();
+ }
+ public void testWindowFireMode1(boolean isLocalOnly) throws InterruptedException {
+ testWindowFireMode1(isLocalOnly,5);
+ }
+ public void testWindowFireMode1(boolean isLocalOnly,int windowSize) throws InterruptedException {
+ AtomicInteger sum = new AtomicInteger(0) ;
+ createSourceDataStream()
+ //.map(new MapFunction<JSONObject, String>() {
+ // AtomicInteger COUNT=new AtomicInteger(0);
+ // Long time;
+ // @Override
+ // public JSONObject map(String message) throws Exception {
+ //
+ // if(time==null){
+ // time=date.getTime();
+ // }else {
+ // int count=COUNT.incrementAndGet();
+ // time+=count;
+ // }
+ // JSONObject msg=JSONObject.parseObject(message);
+ //
+ // msg.put("logTime",time);
+ // return msg;
+ // }
+ //})
+ .window(TumblingWindow.of(Time.seconds(windowSize)))
+ .setTimeField("logTime")
+ .fireMode(1)
+ .setMaxMsgGap(isLocalOnly?20L:20L)
+ .waterMark(100000000)
+ .groupBy("ProjectName", "LogStore")
+ .setLocalStorageOnly(isLocalOnly)
+ .count("total")
+ .sum("OutFlow", "OutFlow")
+ .sum("InFlow", "InFlow")
+ .toDataSteam()
+ .forEach(new ForEachFunction<JSONObject>() {
+
+
+ @Override
+ public synchronized void foreach(JSONObject o) {
+ int total = o.getInteger("total");
+ o.put("sum(total)", sum.addAndGet(total));
+ }
+ }).toPrint().start();
+ }
+
+ public void testWindowFireMode2(boolean isLocalOnly){
+ long time=new Date().getTime();
+ System.out.println(DateUtil.getCurrentTimeString());
+ createSourceDataStream()
+ .map(new MapFunction<JSONObject, String>() {
+ int count=0;
+ @Override
+ public JSONObject map(String message) throws Exception {
+
+ JSONObject msg=JSONObject.parseObject(message);
+ long time= msg.getLong("logTime");
+ Date date=new Date(time);
+ date.setYear(2021-1900);
+ date.setMonth(6);
+ date.setDate(14);
+ msg.put("logTime",date.getTime()+count++);
+ return msg;
+ }
+ })
+ .window(TumblingWindow.of(Time.seconds(5)))
+ .setTimeField("logTime")
+ .setMaxMsgGap(isLocalOnly?5L:20L)
+ .fireMode(1)
+ .waterMark(100000000)
+ .groupBy("ProjectName", "LogStore")
+ .setLocalStorageOnly(isLocalOnly)
+ .count("total")
+ .sum("OutFlow", "OutFlow")
+ .sum("InFlow", "InFlow")
+ .toDataSteam()
+ .map(new MapFunction<JSONObject, JSONObject>() {
+ long time=new Date().getTime();
+ @Override
+ public JSONObject map(JSONObject message) throws Exception {
+ message.put("name","chris");
+ message.put("time",time++);
+ return message;
+ }
+ })
+ .window(TumblingWindow.of(Time.seconds(5)))
+ .fireMode(2).waterMark(100000000)
+ .setMaxMsgGap(80L)
+ .groupBy("name")
+ .setTimeField("time")
+ .sum("total","sum_total")
+ .setLocalStorageOnly(true)
+ .toDataSteam()
+ .forEach(new ForEachFunction<JSONObject>() {
+ AtomicInteger sum = new AtomicInteger(0) ;
+ Map<String,Integer> map=new HashMap<>();
+ @Override
+ public synchronized void foreach(JSONObject o) {
+ String windowInstanceId=o.getString("windowInstanceId");
+ Integer oldValue=map.get(windowInstanceId);
+ int total = o.getInteger("sum_total");
+ if(oldValue!=null){
+ total=total-oldValue;
+ }
+ int nowValue=sum.addAndGet(total);
+ map.put(windowInstanceId,total);
+ o.put("sum(total)", nowValue);
+ }
+ }).toPrint().start();
+
+ }
+
+
+ protected abstract DataStream createSourceDataStream();
+}
diff --git a/rocketmq-streams-clients/src/test/java/org/apache/rocketmq/streams/client/windows/WindowFromFileTest.java b/rocketmq-streams-clients/src/test/java/org/apache/rocketmq/streams/client/windows/WindowFromFileTest.java
new file mode 100644
index 0000000..71a2593
--- /dev/null
+++ b/rocketmq-streams-clients/src/test/java/org/apache/rocketmq/streams/client/windows/WindowFromFileTest.java
@@ -0,0 +1,158 @@
+package org.apache.rocketmq.streams.client.windows;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import com.alibaba.fastjson.JSONObject;
+
+import org.apache.rocketmq.streams.client.StreamBuilder;
+import org.apache.rocketmq.streams.client.transform.DataStream;
+import org.apache.rocketmq.streams.common.utils.FileUtil;
+import org.apache.rocketmq.streams.common.utils.MapKeyUtil;
+import org.apache.rocketmq.streams.common.utils.PrintUtil;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+import org.apache.rocketmq.streams.db.driver.batchloader.IRowOperator;
+import org.apache.rocketmq.streams.db.driver.orm.ORMUtil;
+import org.apache.rocketmq.streams.window.model.WindowInstance;
+import org.apache.rocketmq.streams.window.operator.AbstractWindow;
+import org.apache.rocketmq.streams.window.operator.impl.WindowOperator;
+import org.apache.rocketmq.streams.window.state.impl.WindowValue;
+import org.apache.rocketmq.streams.window.storage.WindowStorage;
+import org.junit.Test;
+
+public class WindowFromFileTest extends AbstractWindowFireModeTest {
+ protected DataStream createSourceDataStream(){
+ return StreamBuilder.dataStream("namespace", "name1")
+ .fromFile("/Users/yuanxiaodong/chris/sls_100000.txt",false);
+ }
+
+ /**
+ *
+ * @throws InterruptedException
+ */
+ @Test
+ public void testWindowFireMode0AndNoTimeField() throws InterruptedException {
+ super.testWindowFireMode0(false);
+ }
+
+
+
+ @Test
+ public void testWindowFireMode1() throws InterruptedException {
+ super.testWindowFireMode1(false);
+ }
+
+
+
+ @Test
+ public void testWindowFireMode2() {
+ super.testWindowFireMode2(false);
+ }
+
+
+
+ @Test
+ public void testSLS10(){
+ Map<String,Integer> map=create();
+
+ System.out.println(map.size());
+ PrintUtil.print(map);
+ }
+
+
+ @Test
+ public void testLoadData() throws InterruptedException {
+ Map<String,Integer> map=create();
+ WindowOperator windowOperator=new WindowOperator();
+ windowOperator.setNameSpace("namespace");
+ windowOperator.setConfigureName("name1");
+ WindowInstance windowInstance=windowOperator.createWindowInstance("2021-07-14 02:00:00","2021-07-14 03:00:00","2021-07-14 03:00:00","1");
+ WindowStorage storage= new WindowStorage();
+ storage.loadSplitData2Local("1", windowInstance.createWindowInstanceId(), WindowValue.class, new IRowOperator() {
+ @Override
+ public void doProcess(Map<String, Object> row) {
+ WindowValue windowValue = ORMUtil.convert(row, WindowValue.class);
+ String groupBy=windowValue.getGroupBy();
+ if(!map.containsKey(groupBy)){
+
+ System.out.println(groupBy+" ======");
+ }else {
+ Integer count=(Integer)windowValue.getComputedColumnResultByKey("total");
+ int mapCount=map.get(groupBy);
+ if(count!=mapCount){
+ System.out.println(groupBy+" ======");
+ }
+ }
+ }
+ });
+
+
+ Thread.sleep(10000000l);
+ }
+
+
+ protected Map<String,Integer> create(){
+ List<String> lines= FileUtil.loadFileLine("/Users/yuanxiaodong/chris/sls_10000.txt");
+ Map<String,Integer> map=new HashMap<>();
+ for(String line:lines){
+ JSONObject msg=JSONObject.parseObject(line);
+ String projectName=msg.getString("ProjectName");
+ String logstore=msg.getString("LogStore");
+
+ String key= MapKeyUtil.createKey(projectName,logstore);
+
+ if (StringUtil.isEmpty(key)) {
+ key="<null>";
+ }
+ Integer count=map.get(key);
+ if(count==null){
+ map.put(key,1);
+ }else {
+ count++;
+ map.put(key,count);
+ }
+ }
+ return map;
+ }
+
+
+ @Test
+ public void testFileResult(){
+ createFile(3);
+ }
+
+
+ protected void createFile(int count){
+
+ date.setYear(2021-1900);
+ date.setMonth(6);
+ date.setDate(27);
+ date.setHours(12);
+ date.setMinutes(1);
+ date.setSeconds(0);
+ Long time=null;
+ for(int i=0;i<count;i++){
+ List<String> lines= FileUtil.loadFileLine("/Users/yuanxiaodong/chris/sls_100000.txt");
+ List<String> msgs=new ArrayList<>();
+ for(String line:lines){
+ JSONObject msg=JSONObject.parseObject(line);
+ if(time==null){
+ time=date.getTime();
+ }else {
+ time=time+1;
+ }
+ msg.put("logTime",time);
+ AbstractWindow window=new WindowOperator();
+ window.setSizeInterval(5);
+ window.setTimeUnitAdjust(1);
+ window.setTimeFieldName("logTime");
+ //WindowInstance windowInstance= window.queryOrCreateWindowInstance(new Message(msg),"1").get(0);
+ msgs.add(msg.toJSONString());
+ }
+ FileUtil.write("/Users/yuanxiaodong/chris/sls_100000_time_"+i+".txt",msgs);
+ }
+ }
+
+}
diff --git a/rocketmq-streams-clients/src/test/java/org/apache/rocketmq/streams/client/windows/WindowFromMetaq.java b/rocketmq-streams-clients/src/test/java/org/apache/rocketmq/streams/client/windows/WindowFromMetaq.java
new file mode 100644
index 0000000..235ce3d
--- /dev/null
+++ b/rocketmq-streams-clients/src/test/java/org/apache/rocketmq/streams/client/windows/WindowFromMetaq.java
@@ -0,0 +1,47 @@
+package org.apache.rocketmq.streams.client.windows;
+
+import org.apache.rocketmq.streams.client.StreamBuilder;
+import org.apache.rocketmq.streams.client.transform.DataStream;
+import org.junit.Test;
+
+public class WindowFromMetaq extends AbstractWindowFireModeTest {
+
+ String topic = "TOPIC_DIPPER_SYSTEM_MSG_5";
+ @Test
+ public void testWindowFireMode0() throws InterruptedException {
+ super.testWindowFireMode0(true);
+ }
+
+
+
+ @Test
+ public void testWindowFireMode1() throws InterruptedException {
+ super.testWindowFireMode1(true);
+ }
+
+
+
+ @Test
+ public void testWindowFireMode2() {
+ super.testWindowFireMode2(true);
+ }
+
+
+ @Test
+ public void testWindowToMetaq() throws InterruptedException {
+
+ long start=System.currentTimeMillis();
+ StreamBuilder.dataStream("namespace", "name")
+ .fromFile("/Users/yuanxiaodong/chris/sls_10.txt", true)
+ .toMetaq(topic)
+ .start();
+ }
+
+
+ protected DataStream createSourceDataStream(){
+ return StreamBuilder.dataStream("namespace", "name1")
+ .fromRocketmq(topic,"chris1",true);
+ }
+
+
+}
diff --git a/rocketmq-streams-clients/src/test/java/org/apache/rocketmq/streams/client/windows/WindowHighAvailabilityTest.java b/rocketmq-streams-clients/src/test/java/org/apache/rocketmq/streams/client/windows/WindowHighAvailabilityTest.java
new file mode 100644
index 0000000..cbca7bb
--- /dev/null
+++ b/rocketmq-streams-clients/src/test/java/org/apache/rocketmq/streams/client/windows/WindowHighAvailabilityTest.java
@@ -0,0 +1,131 @@
+package org.apache.rocketmq.streams.client.windows;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import com.alibaba.fastjson.JSONObject;
+
+import org.apache.rocketmq.streams.client.StreamBuilder;
+import org.apache.rocketmq.streams.client.transform.DataStream;
+import org.apache.rocketmq.streams.common.utils.FileUtil;
+import org.apache.rocketmq.streams.common.utils.MapKeyUtil;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+import org.apache.rocketmq.streams.db.driver.batchloader.BatchRowLoader;
+import org.apache.rocketmq.streams.db.driver.batchloader.IRowOperator;
+import org.apache.rocketmq.streams.db.driver.orm.ORMUtil;
+import org.apache.rocketmq.streams.window.model.WindowInstance;
+import org.apache.rocketmq.streams.window.operator.impl.WindowOperator;
+import org.apache.rocketmq.streams.window.state.impl.WindowValue;
+import org.apache.rocketmq.streams.window.storage.WindowStorage;
+import org.junit.Test;
+
+public class WindowHighAvailabilityTest extends AbstractWindowFireModeTest {
+ protected DataStream createSourceDataStream(){
+ return StreamBuilder.dataStream("namespace", "name1")
+ .fromFile("/Users/yuanxiaodong/chris/sls_100000.txt",false);
+ }
+
+ /**
+ *
+ * @throws InterruptedException
+ */
+ @Test
+ public void testWindowFireMode0AndNoTimeField() throws InterruptedException {
+ super.testWindowFireMode0(false,60*60);
+ }
+
+
+
+ @Test
+ public void testWindowFireMode1() throws InterruptedException {
+ super.testWindowFireMode1(false,60*5);
+ }
+
+
+
+ @Test
+ public void testWindowFireMode2() {
+ super.testWindowFireMode2(false);
+ }
+
+ @Test
+ public void testLoadData() throws InterruptedException {
+ Map<String,Integer> map=create();
+ WindowOperator windowOperator=new WindowOperator();
+ windowOperator.setNameSpace("namespace");
+ windowOperator.setConfigureName("name1_window_10001");
+ WindowInstance windowInstance=windowOperator.createWindowInstance("2021-07-14 02:00:00","2021-07-14 03:00:00","2021-07-14 03:00:00","1");
+
+ WindowStorage storage= new WindowStorage();
+ System.out.println("1;namespace;name1_window_10001;name1_window_10001;2021-07-14 02:00:00;2021-07-14 03:00:00");
+ System.out.println(windowInstance.createWindowInstanceId());
+ System.out.println(StringUtil.createMD5Str(windowInstance.createWindowInstanceId()));
+ Map<String, WindowValue> windowValueMap=new HashMap<>();
+ AtomicInteger sum=new AtomicInteger(0);
+ BatchRowLoader batchRowLoader = new BatchRowLoader("partition_num",
+ "select * from " + ORMUtil.getTableName(WindowValue.class) + " where window_instance_partition_id ='"
+ + StringUtil.createMD5Str(windowInstance.createWindowInstanceId() )+ "'", new IRowOperator(){
+
+ @Override
+ public void doProcess(Map<String, Object> row) {
+ WindowValue windowValue = ORMUtil.convert(row, WindowValue.class);
+
+ String groupBy=windowValue.getGroupBy();
+ windowValueMap.put(groupBy,windowValue);
+ if(!map.containsKey(groupBy)){
+
+ System.out.println(groupBy+" ======");
+ }else {
+ Integer count=(Integer)windowValue.getComputedColumnResultByKey("total");
+ int mapCount=map.get(groupBy);
+ if(count!=mapCount){
+ System.out.println(groupBy+" ======");
+ }
+ //System.out.println(groupBy+" "+count);
+ sum.addAndGet(count);
+
+ }
+ }
+ });
+ batchRowLoader.startLoadData();
+
+ System.out.println("sum "+sum.get());
+ for(String key:map.keySet()){
+ if(windowValueMap.containsKey(key)==false){
+ System.out.println(key+" "+map.get(key));
+ }
+ }
+
+ Thread.sleep(10000000l);
+ }
+
+
+ protected Map<String,Integer> create(){
+ List<String> lines= FileUtil.loadFileLine("/Users/yuanxiaodong/chris/sls_10000.txt");
+ Map<String,Integer> map=new HashMap<>();
+ for(String line:lines){
+ JSONObject msg=JSONObject.parseObject(line);
+ String projectName=msg.getString("ProjectName");
+ String logstore=msg.getString("LogStore");
+
+ String key= MapKeyUtil.createKey(projectName,logstore);
+
+ if (StringUtil.isEmpty(key)) {
+ key="<null>";
+ }
+ Integer count=map.get(key);
+ if(count==null){
+ map.put(key,1);
+ }else {
+ count++;
+ map.put(key,count);
+ }
+ }
+ return map;
+ }
+
+
+
+}
diff --git a/rocketmq-streams-clients/src/test/resources/log4j.xml b/rocketmq-streams-clients/src/test/resources/log4j.xml
new file mode 100755
index 0000000..a1afd37
--- /dev/null
+++ b/rocketmq-streams-clients/src/test/resources/log4j.xml
@@ -0,0 +1,36 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+<!DOCTYPE log4j:configuration SYSTEM "http://toolkit.alibaba-inc.com/dtd/log4j/log4j.dtd">
+<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
+
+ <appender name="Console" class="org.apache.log4j.ConsoleAppender">
+ <layout class="org.apache.log4j.PatternLayout">
+ <param name="ConversionPattern" value="%d{ISO8601} %l [%t] %-5p - %m%n%n"/>
+ </layout>
+ <filter class="org.apache.log4j.varia.LevelRangeFilter">
+ <param name="LevelMin" value="INFO"/>
+ <param name="LevelMax" value="ERROR"/>
+ </filter>
+ </appender>
+
+ <root>
+ <priority value="INFO"/>
+ <appender-ref ref="Console"/>
+ </root>
+
+</log4j:configuration>
\ No newline at end of file
diff --git a/rocketmq-streams-dim/pom.xml b/rocketmq-streams-dim/pom.xml
new file mode 100644
index 0000000..1175370
--- /dev/null
+++ b/rocketmq-streams-dim/pom.xml
@@ -0,0 +1,47 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams</artifactId>
+ <version>2.0.0-SNAPSHOT</version>
+ </parent>
+ <artifactId>rocketmq-streams-dim</artifactId>
+ <name>ROCKETMQ STREAMS :: dim</name>
+ <packaging>jar</packaging>
+ <properties>
+ <file_encoding>UTF-8</file_encoding>
+ <project.build.sourceEncoding>${file_encoding}</project.build.sourceEncoding>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams-filter</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams-channel-http</artifactId>
+ </dependency>
+ </dependencies>
+
+
+</project>
diff --git a/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/DimComponent.java b/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/DimComponent.java
new file mode 100644
index 0000000..8602b5d
--- /dev/null
+++ b/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/DimComponent.java
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.dim;
+
+import java.util.Properties;
+
+import org.apache.rocketmq.streams.common.component.AbstractComponent;
+import org.apache.rocketmq.streams.common.component.ComponentCreator;
+import org.apache.rocketmq.streams.dim.service.IDimService;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.rocketmq.streams.configurable.ConfigurableComponent;
+import org.apache.rocketmq.streams.dim.service.impl.DimServiceImpl;
+
+public class DimComponent extends AbstractComponent<IDimService> {
+
+ private static final Log LOG = LogFactory.getLog(DimComponent.class);
+
+ // private transient Map<String, DBDim> nameListMap = new HashMap<>();
+
+ protected transient ConfigurableComponent configurableComponent;
+ private transient IDimService dimService;
+
+ public static DimComponent getInstance(String namespace) {
+ return ComponentCreator.getComponent(namespace, ComponentCreator.class);
+ }
+
+ @Override
+ protected boolean startComponent(String namespace) {
+ configurableComponent = ComponentCreator.getComponent(namespace, ConfigurableComponent.class);
+ dimService = new DimServiceImpl(configurableComponent);
+ return true;
+ }
+
+ @Override
+ protected boolean initProperties(Properties properties) {
+ return true;
+ }
+
+ @Override
+ public boolean stop() {
+ return true;
+ }
+
+ @Override
+ public IDimService getService() {
+ return dimService;
+ }
+}
diff --git a/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/builder/DimBuilder.java b/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/builder/DimBuilder.java
new file mode 100644
index 0000000..508bb19
--- /dev/null
+++ b/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/builder/DimBuilder.java
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.dim.builder;
+
+import org.apache.rocketmq.streams.dim.model.DBDim;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+
+public class DimBuilder {
+
+ private String url;
+ private String password;
+ private String userName;
+ protected Long pollingTime = 60L; // 同步数据的事件间隔
+ private String jdbcDriver = "com.mysql.jdbc.Driver";
+
+ public DimBuilder(String url, String userName, String password) {
+ this.url = url;
+ this.password = password;
+ this.userName = userName;
+ }
+
+ public DBDim createDim(String namespace, String name, String sqlOrTableName) {
+ DBDim nameList = new DBDim();
+ nameList.setNameSpace(namespace);
+ if (StringUtil.isNotEmpty(name)) {
+ nameList.setConfigureName(name);
+ }
+ String sql = sqlOrTableName;
+ if (sqlOrTableName.split(" ").length == 1) {
+ sql = "select * from " + sqlOrTableName + " limit 500000";
+ }
+ nameList.setSql(sql);
+ nameList.setJdbcdriver(jdbcDriver);
+ nameList.setPollingTimeMintue(pollingTime);
+ nameList.setUrl(url);
+ nameList.setUserName(userName);
+ nameList.setPassword(password);
+ return nameList;
+ }
+
+ public String getUrl() {
+ return url;
+ }
+
+ public void setUrl(String url) {
+ this.url = url;
+ }
+
+ public String getPassword() {
+ return password;
+ }
+
+ public void setPassword(String password) {
+ this.password = password;
+ }
+
+ public String getUserName() {
+ return userName;
+ }
+
+ public void setUserName(String userName) {
+ this.userName = userName;
+ }
+
+ public Long getPollingTime() {
+ return pollingTime;
+ }
+
+ public void setPollingTime(Long pollingTime) {
+ this.pollingTime = pollingTime;
+ }
+
+ public String getJdbcDriver() {
+ return jdbcDriver;
+ }
+
+ public void setJdbcDriver(String jdbcDriver) {
+ this.jdbcDriver = jdbcDriver;
+ }
+}
diff --git a/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/function/expression/InExpressionResource.java b/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/function/expression/InExpressionResource.java
new file mode 100644
index 0000000..c8a13a7
--- /dev/null
+++ b/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/function/expression/InExpressionResource.java
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.dim.function.expression;
+
+import org.apache.rocketmq.streams.dim.model.DBDim;
+import org.apache.rocketmq.streams.filter.context.RuleContext;
+import org.apache.rocketmq.streams.filter.function.expression.AbstractExpressionFunction;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+import org.apache.rocketmq.streams.filter.operator.expression.Expression;
+import org.apache.rocketmq.streams.script.annotation.Function;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethod;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethodAilas;
+
+import java.util.Map;
+
+@Function
+public class InExpressionResource extends AbstractExpressionFunction {
+
+ /**
+ * value格式 :resourceName.fieldName。如果只有单列，可以省略.fieldname
+ *
+ * @param expression
+ * @param context
+ * @param rule
+ * @return
+ */
+ @FunctionMethod(value = "in_expression_resouce", alias = "in_resouce")
+ @FunctionMethodAilas("in_expression_resouce(resourceName->(varName,functionName,value)&((varName,functionName,"
+ + "value)|(varName,functionName,value)))")
+ @Override
+ public Boolean doExpressionFunction(Expression expression, RuleContext context, Rule rule) {
+ return match(expression, context, rule, false);
+ }
+
+ protected Boolean match(Expression expression, RuleContext context, Rule rule, boolean supportRegex) {
+ Object value = expression.getValue();
+ if (value == null) {
+ return null;
+ }
+ String valueStr = String.valueOf(value);
+ String[] valueArray = valueStr.split("->");
+ String dataResourceNamespace = rule.getNameSpace();
+ String dataResourceName = null;
+ String expressionStr = null;
+ if (valueArray.length == 2) {
+ dataResourceName = valueArray[0];
+ expressionStr = valueArray[1];
+ }
+ if (valueArray.length > 2) {
+ dataResourceNamespace = valueArray[0];
+ dataResourceName = valueArray[1];
+ expressionStr = valueArray[2];
+ }
+
+ DBDim dataResource =
+ (DBDim)context.getConfigurableService().queryConfigurableByIdent(DBDim.TYPE, dataResourceName);
+ if (dataResource == null) {
+ return null;
+ }
+ Map<String, Object> row = dataResource.matchExpression(expressionStr, context.getMessage().getMessageBody());
+ if (row != null && row.size() > 0) {
+ return true;
+ }
+ return false;
+ }
+}
diff --git a/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/function/expression/NotInExpressionResource.java b/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/function/expression/NotInExpressionResource.java
new file mode 100644
index 0000000..752276c
--- /dev/null
+++ b/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/function/expression/NotInExpressionResource.java
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.dim.function.expression;
+
+import org.apache.rocketmq.streams.filter.context.RuleContext;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+import org.apache.rocketmq.streams.filter.operator.expression.Expression;
+import org.apache.rocketmq.streams.script.annotation.Function;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethod;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethodAilas;
+
+@Function
+public class NotInExpressionResource extends InExpressionResource {
+
+ /**
+ * value格式 :resourceName.fieldName。如果只有单列，可以省略.fieldname
+ *
+ * @param expression
+ * @param context
+ * @param rule
+ * @return
+ */
+ @FunctionMethod("not_in_expression_resouce")
+ @FunctionMethodAilas("not_in_expression_resouce(resourceName->(varName,functionName,value)&((varName,"
+ + "functionName,value)|(varName,functionName,value)))")
+ @Override
+ public Boolean doExpressionFunction(Expression expression, RuleContext context, Rule rule) {
+ return !match(expression, context, rule, false);
+ }
+
+}
diff --git a/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/function/script/IntelligenceFunction.java b/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/function/script/IntelligenceFunction.java
new file mode 100644
index 0000000..c2ffb88
--- /dev/null
+++ b/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/function/script/IntelligenceFunction.java
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.dim.function.script;
+
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Map.Entry;
+
+import org.apache.rocketmq.streams.common.component.ComponentCreator;
+import org.apache.rocketmq.streams.configurable.ConfigurableComponent;
+import org.apache.rocketmq.streams.dim.DimComponent;
+import org.apache.rocketmq.streams.dim.intelligence.AbstractIntelligenceCache;
+import org.apache.rocketmq.streams.script.context.FunctionContext;
+import org.apache.rocketmq.streams.script.annotation.Function;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethod;
+import org.apache.rocketmq.streams.common.context.IMessage;
+import org.apache.rocketmq.streams.script.utils.FunctionUtils;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+@Function
+public class IntelligenceFunction {
+
+ private static final Log LOG = LogFactory.getLog(IntelligenceFunction.class);
+
+ private DimComponent nameListComponent;
+
+ @FunctionMethod(value = "intelligence", alias = "qingbao")
+ public void intelligence(IMessage message, FunctionContext context, String namespace, String nameListName, String intelligenceFieldName, String asName) {
+ intelligenceInner(message, context, namespace, nameListName, intelligenceFieldName, asName, true);
+ }
+
+ @FunctionMethod(value = "left_join_intelligence", alias = "left_join_qingbao")
+ public void intelligenceLeftJoin(IMessage message, FunctionContext context, String namespace, String nameListName, String intelligenceFieldName, String asName) {
+ intelligenceInner(message, context, namespace, nameListName, intelligenceFieldName, asName, false);
+ }
+
+ public void intelligenceInner(IMessage message, FunctionContext context, String namespace, String nameListName, String intelligenceFieldName, String asName, boolean isInner) {
+ String key = FunctionUtils.getValueString(message, context, intelligenceFieldName);
+ namespace = FunctionUtils.getValueString(message, context, namespace);
+ nameListName = FunctionUtils.getValueString(message, context, nameListName);
+ ConfigurableComponent configurableComponent = ComponentCreator.getComponent(namespace, ConfigurableComponent.class);
+ AbstractIntelligenceCache intelligenceCache = configurableComponent.queryConfigurable(AbstractIntelligenceCache.TYPE, nameListName);
+ if (intelligenceCache == null) {
+ throw new RuntimeException("can not query intelligence. the namespace is " + namespace + ", the name is " + nameListName);
+ }
+ Map<String, Object> row = intelligenceCache.getRow(key);
+ if (row != null) {
+ asName = FunctionUtils.getValueString(message, context, asName);
+ if (StringUtil.isNotEmpty(asName)) {
+ asName = asName + ".";
+ } else {
+ asName = "";
+ }
+ for (Entry<String, Object> entry : row.entrySet()) {
+ String elementKey = asName + entry.getKey();
+ message.getMessageBody().put(elementKey, entry.getValue());
+ }
+ } else {
+ if (isInner) {
+ context.breakExecute();
+ }
+ }
+ }
+
+}
\ No newline at end of file
diff --git a/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/function/script/IntelligenceNameListFunction.java b/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/function/script/IntelligenceNameListFunction.java
new file mode 100644
index 0000000..02d67fb
--- /dev/null
+++ b/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/function/script/IntelligenceNameListFunction.java
@@ -0,0 +1,24 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.dim.function.script;
+
+import org.apache.rocketmq.streams.script.annotation.Function;
+
+@Function
+public class IntelligenceNameListFunction {
+
+}
diff --git a/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/function/script/NameListFunction.java b/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/function/script/NameListFunction.java
new file mode 100644
index 0000000..3b03a4e
--- /dev/null
+++ b/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/function/script/NameListFunction.java
@@ -0,0 +1,203 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.dim.function.script;
+
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+
+import com.alibaba.fastjson.JSONArray;
+import com.alibaba.fastjson.JSONObject;
+
+import org.apache.rocketmq.streams.common.component.ComponentCreator;
+import org.apache.rocketmq.streams.dim.DimComponent;
+import org.apache.rocketmq.streams.script.context.FunctionContext;
+import org.apache.rocketmq.streams.script.annotation.Function;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethod;
+import org.apache.rocketmq.streams.common.context.IMessage;
+import org.apache.rocketmq.streams.script.utils.FunctionUtils;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+@Function
+public class NameListFunction {
+
+ private static final Log LOG = LogFactory.getLog(NameListFunction.class);
+
+ private DimComponent nameListComponent;
+
+ @FunctionMethod(value = "innerJoin", alias = "inner_join")
+ public JSONArray innerJoin(IMessage message, FunctionContext context, String namespace, String nameListName,
+ String expressionStr, String alias, String script, String... fieldNames) {
+ JSONArray rows = getRows(message, context, namespace, nameListName, expressionStr, alias, script, fieldNames);
+ if (rows == null || rows.size() == 0) {
+ context.breakExecute();
+ return null;
+ }
+ return rows;
+ }
+
+ @FunctionMethod(value = "leftJoin", alias = "left_join")
+ public JSONArray leftJoin(IMessage message, FunctionContext context, String namespace, String nameListName,
+ String expressionStr, String alias, String script, String... fieldNames) {
+ JSONArray rows = getRows(message, context, namespace, nameListName, expressionStr, alias, script, fieldNames);
+ if (rows == null || rows.size() == 0) {
+ return null;
+ }
+ return rows;
+ }
+
+ @FunctionMethod(value = "mark_rows", alias = "namelist_rows")
+ public String namelist(IMessage message, FunctionContext context, String namespace, String nameListName,
+ String expressionStr, String... fieldNames) {
+ JSONArray rows = getRows(message, context, namespace, nameListName, expressionStr, null, null, fieldNames);
+ if (rows == null || rows.size() == 0) {
+ return null;
+ }
+ return rows.toJSONString();
+ }
+
+ @FunctionMethod(value = "mark", alias = "namelist")
+ public String namelist(IMessage message, FunctionContext context, String namespace, String nameListName,
+ String expressionStr, String fieldName) {
+ String tmp = fieldName;
+ nameListName = FunctionUtils.getValueString(message, context, nameListName);
+ namespace = FunctionUtils.getValueString(message, context, namespace);
+ expressionStr = FunctionUtils.getValueString(message, context, expressionStr);
+ fieldName = FunctionUtils.getValueString(message, context, fieldName);
+ if (fieldName == null) {
+ fieldName = tmp;
+ }
+ nameListComponent = ComponentCreator.getComponent(namespace, DimComponent.class);
+ Map<String, Object> row =
+ nameListComponent.getService().match(nameListName, expressionStr, message.getMessageBody());
+ if (row != null) {
+ Object value = row.get(fieldName);
+ if (value == null) {
+ return null;
+ }
+ return value.toString();
+ }
+ return null;
+ }
+
+ @FunctionMethod(value = "mark", alias = "namelist")
+ public String namelist(IMessage message, FunctionContext context, String namespace, String nameListName,
+ String expressionStr, String fieldNames, String joinMark) {
+ nameListName = FunctionUtils.getValueString(message, context, nameListName);
+ namespace = FunctionUtils.getValueString(message, context, namespace);
+ expressionStr = FunctionUtils.getValueString(message, context, expressionStr);
+ fieldNames = FunctionUtils.getValueString(message, context, fieldNames);
+ joinMark = FunctionUtils.getValueString(message, context, joinMark);
+ nameListComponent = ComponentCreator.getComponent(namespace, DimComponent.class);
+ Map<String, Object> row =
+ nameListComponent.getService().match(nameListName, expressionStr, message.getMessageBody());
+ if (row != null) {
+ String[] fieldNameTem = fieldNames.split(",");
+ StringBuilder result = new StringBuilder();
+ for (int i = 0; i < fieldNameTem.length; i++) {
+ Object tem = row.get(fieldNameTem[i]);
+ if (tem != null) {
+ if (result.length() == 0) {
+ if (StringUtil.isNotEmpty(tem.toString()) && !("null".equalsIgnoreCase(tem.toString()))) {
+ result.append(tem);
+ }
+ } else {
+ if (StringUtil.isNotEmpty(tem.toString()) && !("null".equalsIgnoreCase(tem.toString()))) {
+ result.append(joinMark + tem);
+ }
+ }
+ }
+
+ }
+ return result.toString();
+ }
+ return null;
+ }
+
+ @FunctionMethod(value = "in_namelist", alias = "in_namelist")
+ public Boolean inNameList(IMessage message, FunctionContext context, String namespace, String nameListName,
+ String expressionStr) {
+ nameListComponent = ComponentCreator.getComponent(namespace, DimComponent.class);
+ Map<String, Object> row =
+ nameListComponent.getService().match(nameListName, expressionStr, message.getMessageBody());
+ if (row != null && row.size() > 0) {
+ return true;
+ }
+ return false;
+ }
+
+ /**
+ * 根据表达式，从namelist中获取符合条件的数据
+ *
+ * @param message
+ * @param context
+ * @param namespace
+ * @param nameListName
+ * @param expressionStr （varname,functionName,value)&(varname,functionName,value)
+ * @param fieldNames 需要返回的字段名
+ * @return
+ */
+ protected JSONArray getRows(IMessage message, FunctionContext context, String namespace, String nameListName, String expressionStr, String alias, String script, String... fieldNames) {
+ nameListName = FunctionUtils.getValueString(message, context, nameListName);
+ namespace = FunctionUtils.getValueString(message, context, namespace);
+ expressionStr = FunctionUtils.getValueString(message, context, expressionStr);
+ script = FunctionUtils.getValueString(message, context, script);
+ if (StringUtil.isEmpty(script)) {
+ script = null;
+ }
+ nameListComponent = ComponentCreator.getComponent(namespace, DimComponent.class);
+ List<Map<String, Object>> rows =
+ nameListComponent.getService().matchSupportMultiRow(nameListName, expressionStr, message.getMessageBody(), script);
+ if (rows == null || rows.size() == 0) {
+ return null;
+ }
+ JSONArray jsonArray = new JSONArray();
+ for (Map<String, Object> row : rows) {
+ JSONObject jsonObject = new JSONObject();
+ if (fieldNames == null || fieldNames.length == 0) {
+ if (alias == null) {
+ jsonObject.putAll(row);
+ } else {
+ Iterator<Entry<String, Object>> it = row.entrySet().iterator();
+ while (it.hasNext()) {
+ Entry<String, Object> entry = it.next();
+ String fieldName = entry.getKey();
+ if (alias != null) {
+ fieldName = alias + "." + fieldName;
+ }
+ jsonObject.put(fieldName, entry.getValue());
+ }
+ }
+
+ } else {
+ for (String fieldName : fieldNames) {
+ String tmp = fieldName;
+ if (alias != null) {
+ fieldName = alias + "." + fieldName;
+ }
+ jsonObject.put(fieldName, row.get(tmp));
+ }
+ }
+ jsonArray.add(jsonObject);
+ }
+ return jsonArray;
+ }
+
+}
\ No newline at end of file
diff --git a/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/index/DimIndex.java b/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/index/DimIndex.java
new file mode 100644
index 0000000..6c6aeef
--- /dev/null
+++ b/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/index/DimIndex.java
@@ -0,0 +1,319 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.dim.index;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.rocketmq.streams.common.datatype.IntDataType;
+import org.apache.rocketmq.streams.common.cache.compress.impl.IntValueKV;
+import org.apache.rocketmq.streams.common.cache.CompressTable;
+import org.apache.rocketmq.streams.common.utils.MapKeyUtil;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+public class DimIndex {
+
+ private static final Log LOG = LogFactory.getLog(DimIndex.class);
+
+ /**
+ * 索引字段名，支持多个索引，每个索引一行，支持组合索引，多个字段用；拼接 name 单索引 name;age 组合索引
+ */
+ protected List<String> indexs = new ArrayList<>();
+
+ /**
+ * 如果是唯一索引，用压缩值存储 每个索引一行
+ */
+ protected Map<String, IntValueKV> uniqueIndex;
+
+ /**
+ * 如果是非唯一索引，用这个结构存储 每个索引一行，后面的map：key：索引值；value：row id 列表，rowid用字节表示
+ */
+ protected Map<String, Map<String, List<Integer>>> mutilIndex = new HashMap<>();
+
+ protected boolean isUnique = false;//如果是唯一索引，值为true
+
+ public DimIndex(List<String> indexs) {
+ this.indexs = formatIndexs(indexs);
+ }
+
+ public DimIndex(String index, String... indexs) {
+ if (indexs == null) {
+ return;
+ }
+ List<String> indexList = new ArrayList<>();
+ for (String idx : indexs) {
+ indexList.add(idx);
+ }
+ this.indexs = formatIndexs(indexList);
+ }
+
+ /**
+ * 组合索引，多个字段要以名称顺序排列，完成索引名称的标注化处理
+ *
+ * @param indexs
+ * @return
+ */
+ protected List<String> formatIndexs(List<String> indexs) {
+ List<String> allIndex = new ArrayList<>();
+ for (String indexName : indexs) {
+ String[] values = indexName.split(";");
+ List<String> indexList = new ArrayList<>();
+ for (String value : values) {
+ indexList.add(value);
+ }
+ Collections.sort(indexList);
+ String indexKey = MapKeyUtil.createKey(indexList);
+ allIndex.add(indexKey);
+ }
+ return allIndex;
+ }
+
+ /**
+ * 加载一行数据，如果是唯一索引，在uniqueIndex中查找，否则在mutilIndex查找
+ *
+ * @param indexName 索引名，如name
+ * @param indexValue 索引值，如chris
+ * @return
+ */
+ public List<Integer> getRowIds(String indexName, String indexValue) {
+ if (isUnique) {
+ IntValueKV index = this.uniqueIndex.get(indexName);
+ if (index == null) {
+ return null;
+ }
+ Integer rowId = index.get(indexValue);
+ if (rowId == null) {
+ return null;
+ }
+ List<Integer> rowIds = new ArrayList<>();
+ rowIds.add(rowId);
+ return rowIds;
+ } else {
+ Map<String, List<Integer>> indexs = this.mutilIndex.get(indexName);
+ if (indexs == null) {
+ return null;
+ }
+ return indexs.get(indexValue);
+ }
+ }
+
+ /**
+ * 构建索引，如果是唯一索引，构建在uniqueIndex数据结构中，否则构建在mutilIndex这个数据结构中
+ *
+ * @param tableCompress 表数据
+ */
+ public void buildIndex(CompressTable tableCompress) {
+
+ if (isUnique) {
+ Map<String, IntValueKV> fieldIndex2RowIndex = new HashMap<>();
+ buildUniqueIndex(tableCompress, fieldIndex2RowIndex);
+ this.uniqueIndex = fieldIndex2RowIndex;
+ } else {
+ Map<String, Map<String, List<Integer>>> fieldIndex2RowIndex = new HashMap<>();
+ buildIndex(tableCompress, fieldIndex2RowIndex);
+ this.mutilIndex = fieldIndex2RowIndex;
+ }
+ }
+
+ /**
+ * 构建唯一索引
+ *
+ * @param tableCompress
+ * @param fieldIndex2RowIndex
+ */
+ protected void buildUniqueIndex(CompressTable tableCompress, Map<String, IntValueKV> fieldIndex2RowIndex) {
+ if (indexs == null || indexs.size() == 0) {
+ return;
+ }
+
+ int i = 0;
+ Iterator<Map<String, Object>> it = tableCompress.newIterator();
+ while (it.hasNext()) {
+ /**
+ * 为每个索引做构建
+ */
+ for (String indexName : indexs) {
+ IntValueKV index = fieldIndex2RowIndex.get(indexName);
+ if (index == null) {
+ synchronized (this) {
+ index = fieldIndex2RowIndex.get(indexName);
+ if (index == null) {
+ index = new IntValueKV(tableCompress.getRowCount());
+ fieldIndex2RowIndex.put(indexName, index);
+ }
+ }
+
+ }
+ String[] nameIndexs = indexName.split(";");
+ Arrays.sort(nameIndexs);
+ Map<String, String> cacheValues = createRow(it.next());
+ String indexValue = createIndexValue(cacheValues, nameIndexs);
+ index.put(indexValue, i);
+ }
+ i++;
+ }
+ LOG.info(" finish poll data , the row count is " + i + ". byte is " + tableCompress
+ .getByteCount());
+ }
+
+ /**
+ * 从table compress 中取出所有的行，构建索引。把
+ *
+ * @param dataCacheVar
+ * @param fieldIndex2RowIndex
+ */
+ protected void buildIndex(CompressTable dataCacheVar, Map<String, Map<String, List<Integer>>> fieldIndex2RowIndex) {
+ Iterator<Map<String, Object>> it = dataCacheVar.newIterator();
+ int i = 0;
+ while (it.hasNext()) {
+ Map<String, Object> row = it.next();
+ Map<String, String> cacheValues = createRow(row);
+ putValue2Index(cacheValues, i, fieldIndex2RowIndex);
+ i++;
+ }
+
+ LOG.info(" finish poll data , the row count is " + i + ". byte is " + dataCacheVar
+ .getByteCount());
+
+ }
+
+ /**
+ * 创建索引结构
+ *
+ * @param value
+ * @param rowIndex
+ * @param fieldIndex2RowIndex
+ */
+ protected void putValue2Index(Map<String, String> value, Integer rowIndex,
+ Map<String, Map<String, List<Integer>>> fieldIndex2RowIndex) {
+ if (indexs == null || indexs.size() == 0) {
+ return;
+ }
+ for (String indexName : indexs) {
+ Map<String, List<Integer>> name2RowIndexs = fieldIndex2RowIndex.get(indexName);
+ if (name2RowIndexs == null) {
+ synchronized (this) {
+ name2RowIndexs = fieldIndex2RowIndex.get(indexName);
+ if (name2RowIndexs == null) {
+ name2RowIndexs = new HashMap<>();
+ fieldIndex2RowIndex.put(indexName, name2RowIndexs);
+ }
+ }
+
+ }
+ String[] nameIndexs = indexName.split(";");
+ Arrays.sort(nameIndexs);
+ addValue2Index(indexName, name2RowIndexs, rowIndex, value, nameIndexs);
+ }
+ }
+
+ /**
+ * 根据索引名称，把不同的索引值创建key，value放入索引缓存
+ *
+ * @param indexName 索引名称，多字段索引用；分割
+ * @param name2RowIndexs 索引值，list<id>
+ * @param rowIndex 行号
+ * @param row
+ */
+ protected void addValue2Index(String indexName, Map<String, List<Integer>> name2RowIndexs, Integer rowIndex,
+ Map<String, String> row, String[] nameIndexs) {
+ String indexValue = createIndexValue(row, nameIndexs);
+ addValue2Index(name2RowIndexs, rowIndex, indexValue);
+ }
+
+ /**
+ * 对于组合索引，把各个字段的值取出来
+ *
+ * @param row
+ * @param nameIndexs
+ * @return
+ */
+ protected String createIndexValue(Map<String, String> row, String[] nameIndexs) {
+ String[] indexValues = new String[nameIndexs.length];
+ for (int i = 0; i < nameIndexs.length; i++) {
+ indexValues[i] = row.get(nameIndexs[i]);
+ }
+ if (indexValues != null && indexValues.length > 0) {
+ String indexValue = MapKeyUtil.createKey(indexValues);
+ return indexValue;
+ }
+ return null;
+ }
+
+ /**
+ * 把row 中非string的值转化成string
+ *
+ * @param row
+ * @return
+ */
+ protected Map<String, String> createRow(Map<String, Object> row) {
+ Map<String, String> cacheValues = new HashMap<String, String>();//一行数据
+ Iterator<Map.Entry<String, Object>> iterator = row.entrySet().iterator();
+ //把数据value从object转化成string
+ while (iterator.hasNext()) {
+ Map.Entry<String, Object> entry = iterator.next();
+ if (entry != null && entry.getValue() != null && entry.getKey() != null) {
+ cacheValues.put(entry.getKey(), entry.getValue().toString());
+ }
+ }
+ return cacheValues;
+ }
+
+ public static IntDataType INTDATATYPE = new IntDataType();
+
+ /**
+ * 把确定多索引值，和行号放入索引缓存
+ *
+ * @param name2RowIndexs
+ * @param rowIndex
+ * @param indexValue
+ */
+ protected void addValue2Index(Map<String, List<Integer>> name2RowIndexs, Integer rowIndex,
+ String indexValue) {
+
+ List<Integer> rowIndexs = name2RowIndexs.get(indexValue);
+ if (rowIndexs == null) {
+ rowIndexs = new ArrayList<>();
+ name2RowIndexs.put(indexValue, rowIndexs);
+ }
+ rowIndexs.add(rowIndex);
+
+ }
+
+ public boolean isUnique() {
+ return isUnique;
+ }
+
+ public void setUnique(boolean unique) {
+ isUnique = unique;
+ }
+
+ public List<String> getIndexs() {
+ return indexs;
+ }
+
+ public void setIndexs(List<String> indexs) {
+ this.indexs = indexs;
+ }
+
+}
diff --git a/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/index/IndexExecutor.java b/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/index/IndexExecutor.java
new file mode 100644
index 0000000..ceb4f52
--- /dev/null
+++ b/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/index/IndexExecutor.java
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.dim.index;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import com.alibaba.fastjson.JSONObject;
+
+import org.apache.rocketmq.streams.dim.model.AbstractDim;
+import org.apache.rocketmq.streams.filter.builder.ExpressionBuilder;
+import org.apache.rocketmq.streams.filter.function.expression.Equals;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+import org.apache.rocketmq.streams.filter.operator.expression.Expression;
+import org.apache.rocketmq.streams.filter.operator.expression.RelationExpression;
+import org.apache.rocketmq.streams.script.ScriptComponent;
+import org.apache.rocketmq.streams.common.datatype.IntDataType;
+import org.apache.rocketmq.streams.common.utils.DataTypeUtil;
+import org.apache.rocketmq.streams.common.utils.MapKeyUtil;
+
+/**
+ * 执行索引的查询和构建。主要是完成表达式的解析，对于等值的表达式字段，如果有索引，根据索引查询，然后执行非等值部分的判断
+ */
+public class IndexExecutor {
+
+ private String expressionStr;//表达式
+
+ private boolean isSupport = false;//是否支持索引，比如表达式等值部分无所以，则不能走索引逻辑
+
+ private String namespace;
+
+ private String indexNameKey;//索引的名字，多个字段";"分隔
+
+ private List<String> msgNames;//
+
+ private Rule rule;//表达式会被编译成rule
+
+ private List<String> index; //标准化后的索引name
+
+ private Set<String> indexNames = new HashSet<>();
+
+ public IndexExecutor(String expressionStr, String namespace, List<String> index) {
+ this.expressionStr = expressionStr;
+ this.namespace = namespace;
+
+ List<String> allIndex = new ArrayList<>();
+ for (String indexName : index) {
+ String[] values = indexName.split(";");
+ List<String> indexList = new ArrayList<>();
+ for (String value : values) {
+ indexNames.add(value);
+ indexList.add(value);
+ }
+ Collections.sort(indexList);
+ String indexKey = MapKeyUtil.createKey(indexList);
+ allIndex.add(indexKey);
+ }
+ this.index = allIndex;
+ parse();
+ }
+
+ /**
+ * 解析表达式，找出等值字段和非等值字段 如果有索引走索引，否则走全表扫描
+ */
+ protected void parse() {
+ List<Expression> expressions = new ArrayList<>();
+ List<RelationExpression> relationExpressions = new ArrayList<>();
+ Expression expression = ExpressionBuilder.createOptimizationExpression(namespace, "tmp", expressionStr,
+ expressions, relationExpressions);
+
+ RelationExpression relationExpression = null;
+ if (RelationExpression.class.isInstance(expression)) {
+ relationExpression = (RelationExpression)expression;
+ if (!"and".equals(relationExpression.getRelation())) {
+ isSupport = false;
+ return;
+ }
+ }
+
+ this.isSupport = true;
+ List<Expression> indexExpressions = new ArrayList<>();
+ List<Expression> otherExpressions = new ArrayList<>();
+ if (relationExpression != null) {
+ Map<String, Expression> map = new HashMap<>();
+ for (Expression tmp : expressions) {
+ map.put(tmp.getConfigureName(), tmp);
+ }
+ for (Expression tmp : relationExpressions) {
+ map.put(tmp.getConfigureName(), tmp);
+ }
+ List<String> expressionNames = relationExpression.getValue();
+ relationExpression.setValue(new ArrayList<>());
+ for (String expressionName : expressionNames) {
+ Expression subExpression = map.get(expressionName);
+ if (subExpression != null && !RelationExpression.class.isInstance(subExpression)) {
+ indexExpressions.add(subExpression);
+ } else {
+ otherExpressions.add(subExpression);
+ relationExpression.getValue().add(subExpression.getConfigureName());
+ }
+ }
+
+ } else {
+ indexExpressions.add(expression);
+ }
+
+ List<String> fieldNames = new ArrayList<>();
+ Map<String, String> msgNames = new HashMap<>();
+
+ for (Expression expre : indexExpressions) {
+ if (RelationExpression.class.isInstance(expre)) {
+ continue;
+ }
+ String indexName = expre.getValue().toString();
+ if (Equals.isEqualFunction(expre.getFunctionName()) && indexNames.contains(indexName)) {
+
+ fieldNames.add(indexName);
+ msgNames.put(indexName, expre.getVarName());
+ }
+ }
+ Collections.sort(fieldNames);
+ indexNameKey = MapKeyUtil.createKey(fieldNames);
+ if (!this.index.contains(indexNameKey)) {
+ this.isSupport = false;
+ return;
+ }
+ this.msgNames = createMsgNames(fieldNames, msgNames);
+ if (otherExpressions.size() == 0) {
+ return;
+ }
+ Rule rule = null;
+ if (relationExpression == null) {
+ rule = ExpressionBuilder.createRule(namespace, "tmpRule", expression);
+ } else {
+ rule = ExpressionBuilder.createRule(namespace, "tmpRule", expression, expressions, relationExpressions);
+ }
+
+ this.rule = rule;
+
+ }
+
+ /**
+ * 创建索引字段的索引值
+ *
+ * @param fieldNames
+ * @param msgNames
+ * @return
+ */
+ protected List<String> createMsgNames(List<String> fieldNames, Map<String, String> msgNames) {
+ List<String> msgNameList = new ArrayList<>();
+ for (String fieldName : fieldNames) {
+ msgNameList.add(msgNames.get(fieldName));
+ }
+ return msgNameList;
+ }
+
+ public boolean isSupport() {
+ return isSupport;
+ }
+
+ private static IntDataType intDataType = new IntDataType();
+
+ public List<Map<String, Object>> match(JSONObject msg, AbstractDim nameList, boolean needAll) {
+ return match(msg, nameList, needAll);
+ }
+
+ public List<Map<String, Object>> match(JSONObject msg, AbstractDim nameList, boolean needAll, String script) {
+ List<Map<String, Object>> rows = new ArrayList<>();
+ String msgValue = createValue(msg);
+ List<Integer> rowIds = nameList.findRowIdByIndex(indexNameKey, msgValue);
+ if (rowIds == null) {
+ return null;
+ }
+ for (Integer rowId : rowIds) {
+ Map<String, Object> oldRow = nameList.getDataCache().getRow(rowId);
+ Map<String, Object> newRow = executeScript(oldRow, script);
+ if (rule == null) {
+ rows.add(newRow);
+ if (needAll == false) {
+ return rows;
+ }
+ continue;
+ }
+ Rule ruleTemplete = this.rule;
+ Rule rule = ruleTemplete.copy();
+ Map<String, Expression> expressionMap = new HashMap<>();
+ for (Expression expression : rule.getExpressionMap().values()) {
+ expressionMap.put(expression.getConfigureName(), expression);
+ if (RelationExpression.class.isInstance(expression)) {
+ continue;
+ }
+ Object object = expression.getValue();
+ if (object != null && DataTypeUtil.isString(object.getClass())) {
+ String fieldName = (String)object;
+ Object o = newRow.get(fieldName);
+ if (o != null) {
+ Expression e = expression.copy();
+ e.setValue(o.toString());
+ expressionMap.put(e.getConfigureName(), e);
+ }
+ }
+ }
+ rule.setExpressionMap(expressionMap);
+ boolean matched = rule.execute(msg);
+ if (matched) {
+ rows.add(newRow);
+ if (needAll == false) {
+ return rows;
+ }
+ }
+ }
+ return rows;
+ }
+
+ protected Map<String, Object> executeScript(Map<String, Object> oldRow, String script) {
+ if (script == null) {
+ return oldRow;
+ }
+ ScriptComponent scriptComponent = ScriptComponent.getInstance();
+ JSONObject msg = new JSONObject();
+ msg.putAll(oldRow);
+ scriptComponent.getService().executeScript(msg, script);
+ return msg;
+ }
+
+ /**
+ * 按顺序创建msg的key
+ *
+ * @param msg
+ * @return
+ */
+ private String createValue(JSONObject msg) {
+ List<String> value = new ArrayList<>();
+ for (String msgName : msgNames) {
+ value.add(msg.getString(msgName));
+ }
+ return MapKeyUtil.createKey(value);
+ }
+}
diff --git a/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/intelligence/AbstractIntelligenceCache.java b/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/intelligence/AbstractIntelligenceCache.java
new file mode 100644
index 0000000..65b27f7
--- /dev/null
+++ b/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/intelligence/AbstractIntelligenceCache.java
@@ -0,0 +1,395 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.dim.intelligence;
+
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import com.alibaba.fastjson.JSON;
+import com.alibaba.fastjson.JSONObject;
+
+import org.apache.rocketmq.streams.http.source.util.HttpUtil;
+import org.apache.rocketmq.streams.common.component.ComponentCreator;
+import org.apache.rocketmq.streams.common.configurable.BasedConfigurable;
+import org.apache.rocketmq.streams.common.configurable.IAfterConfiguableRefreshListerner;
+import org.apache.rocketmq.streams.common.configurable.IConfigurableService;
+import org.apache.rocketmq.streams.common.configurable.annotation.ENVDependence;
+import org.apache.rocketmq.streams.common.configure.ConfigureFileKey;
+import org.apache.rocketmq.streams.common.cache.compress.impl.IntValueKV;
+import org.apache.rocketmq.streams.common.channel.sink.ISink;
+import org.apache.rocketmq.streams.common.utils.NumberUtils;
+import org.apache.rocketmq.streams.common.utils.SQLUtil;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.rocketmq.streams.common.dboperator.IDBDriver;
+import org.apache.rocketmq.streams.db.driver.JDBCDriver;
+
+public abstract class AbstractIntelligenceCache extends BasedConfigurable implements
+ IAfterConfiguableRefreshListerner {
+
+ private static final Log LOG = LogFactory.getLog(AbstractIntelligenceCache.class);
+
+ public static final String TYPE = "intelligence";
+
+ protected static final int FILE_MAX_LINE = 50000;//每个文件最大行数
+
+ protected transient IntValueKV intValueKV = new IntValueKV(0) {
+ @Override
+ public Integer get(String key) {
+ return null;
+ }
+
+ @Override
+ public void put(String key, Integer value) {
+
+ }
+ };
+
+ protected String idFieldName = "id";//必须有id字段
+
+ protected int batchSize = 3000;
+
+ @ENVDependence
+ protected Long pollingTimeMintue = 30L;
+
+ protected String datasourceName;//情报对应的存储
+
+ protected transient IDBDriver outputDataSource;
+
+ protected static ExecutorService executorService;
+
+ protected transient ScheduledExecutorService scheduledExecutorService;
+
+ public AbstractIntelligenceCache() {
+ setType(TYPE);
+ executorService = new ThreadPoolExecutor(20, 20,
+ 0L, TimeUnit.MILLISECONDS,
+ new LinkedBlockingQueue<Runnable>(1000));
+ scheduledExecutorService = new ScheduledThreadPoolExecutor(3);
+ }
+
+ public IntValueKV startLoadData(String sql, IDBDriver resource) {
+ try {
+ String statisticalSQL = sql;
+ int startIndex = sql.toLowerCase().indexOf("from");
+ statisticalSQL = "select count(1) as c, min(" + idFieldName + ") as min, max(" + idFieldName + ") as max "
+ + sql.substring(startIndex);
+ List<Map<String, Object>> rows = resource.queryForList(statisticalSQL);
+ Map<String, Object> row = rows.get(0);
+ int count = Integer.valueOf(row.get("c").toString());
+ IntValueKV intValueKV = new IntValueKV(count);
+ //int maxBatch=count/maxSyncCount;//每1w条数据，一个并发。如果数据量比较大，为了提高性能，并行执行
+ if (count == 0) {
+ return new IntValueKV(0) {
+ @Override
+ public Integer get(String key) {
+ return null;
+ }
+
+ @Override
+ public void put(String key, Integer value) {
+
+ }
+ };
+ }
+ long min = Long.valueOf(row.get("min").toString());
+ long max = Long.valueOf(row.get("max").toString());
+ int maxSyncCount = count / FILE_MAX_LINE + 1;
+ long step = (max - min + 1) / maxSyncCount;
+ CountDownLatch countDownLatch = new CountDownLatch(maxSyncCount + 1);
+ AtomicInteger finishedCount = new AtomicInteger(0);
+ String taskSQL = null;
+ if (sql.indexOf(" where ") != -1) {
+ taskSQL = sql + " and " + idFieldName + ">#{startIndex} and " + idFieldName + "<=#{endIndex} order by "
+ + idFieldName + " limit " + batchSize;
+ } else {
+ taskSQL = sql + " where " + idFieldName + ">#{startIndex} and " + idFieldName
+ + "<=#{endIndex} order by " + idFieldName + " limit " + batchSize;
+ }
+
+ int i = 0;
+ for (; i < maxSyncCount; i++) {
+ FetchDataTask fetchDataTask = new FetchDataTask(taskSQL, (min - 1) + step * i,
+ (min - 1) + step * (i + 1), countDownLatch, finishedCount, resource, i, intValueKV, this, count);
+ executorService.execute(fetchDataTask);
+ }
+ FetchDataTask fetchDataTask = new FetchDataTask(taskSQL, (min - 1) + step * i, (min - 1) + step * (i + 1),
+ countDownLatch, finishedCount, resource, i, intValueKV, this, count);
+ executorService.execute(fetchDataTask);
+
+ countDownLatch.await();
+
+ LOG.info(getClass().getSimpleName() + " load data finish, load data line size is " + intValueKV.getSize());
+ return intValueKV;
+ } catch (Exception e) {
+ LOG.error("failed loading intelligence data!", e);
+ return new IntValueKV(0) {
+ @Override
+ public Integer get(String key) {
+ return null;
+ }
+
+ @Override
+ public void put(String key, Integer value) {
+
+ }
+ };
+ }
+ }
+
+ protected transient AtomicBoolean hasInit = new AtomicBoolean(false);
+
+ @Override
+ public void doProcessAfterRefreshConfigurable(IConfigurableService configurableService) {
+ this.outputDataSource = configurableService.queryConfigurable(ISink.TYPE, datasourceName);
+ }
+
+ public void startIntelligence() {
+ boolean success = dbInit();
+ if (success) {
+ startIntelligenceInner();
+ } else {
+ Thread thread = new Thread(new Runnable() {
+ @Override
+ public void run() {
+ boolean success = false;
+ while (!success) {
+ success = dbInit();
+ try {
+ Thread.sleep(60 * 1000);
+ } catch (InterruptedException e) {
+ e.printStackTrace();
+ }
+ }
+ startIntelligenceInner();
+ }
+ });
+ thread.start();
+ }
+ }
+
+ public void startIntelligenceInner() {
+ String sql = getSQL();
+ if (hasInit.compareAndSet(false, true)) {
+ this.intValueKV = startLoadData(sql, outputDataSource);
+ scheduledExecutorService.scheduleWithFixedDelay(new Runnable() {
+ @Override
+ public void run() {
+ intValueKV = startLoadData(sql, outputDataSource);
+ }
+ }, pollingTimeMintue, pollingTimeMintue, TimeUnit.MINUTES);
+ }
+ }
+
+ public abstract Map<String, Object> getRow(String key);
+
+ /**
+ * 查询情报需要的sql
+ *
+ * @return
+ */
+ protected abstract String getSQL();
+
+ /**
+ * 情报中的 情报字段名
+ *
+ * @return
+ */
+ public abstract String getKeyName();
+
+ /**
+ * 情报对应的表名
+ *
+ * @return
+ */
+ public abstract String getTableName();
+
+ protected class FetchDataTask implements Runnable {
+ IntValueKV intValueKV;
+ long startIndex;
+ long endIndex;
+ String sql;
+ CountDownLatch countDownLatch;
+ int index;
+ IDBDriver resource;
+ AtomicInteger finishedCount;//完成了多少条
+ AbstractIntelligenceCache cache;
+ int totalSize;//一共有多少条数据
+
+ public FetchDataTask(String sql, long startIndex, long endIndex, CountDownLatch countDownLatch,
+ AtomicInteger finishedCount, IDBDriver resource, int i, IntValueKV intValueKV, AbstractIntelligenceCache cache, int totalSize) {
+ this.startIndex = startIndex;
+ this.endIndex = endIndex;
+ this.countDownLatch = countDownLatch;
+ this.sql = sql;
+ this.finishedCount = finishedCount;
+ this.resource = resource;
+ this.index = i;
+ this.intValueKV = intValueKV;
+ this.cache = cache;
+ this.totalSize = totalSize;
+ }
+
+ @Override
+ public void run() {
+ long currentIndex = startIndex;
+ JSONObject msg = new JSONObject();
+ msg.put("endIndex", endIndex);
+ while (true) {
+ try {
+
+ msg.put("startIndex", currentIndex);
+
+ String sql = SQLUtil.parseIbatisSQL(msg, this.sql);
+ List<Map<String, Object>> rows = resource.queryForList(sql);
+ if (rows == null || rows.size() == 0) {
+ break;
+ }
+ currentIndex = Long.valueOf(rows.get(rows.size() - 1).get(idFieldName).toString());
+
+ int size = rows.size();
+ int count = finishedCount.addAndGet(size);
+ double progress = (double)count / (double)totalSize;
+ progress = progress * 100;
+ System.out.println(cache.getClass().getSimpleName() + ", finished count is " + count + " the total count is " + totalSize + ", the progress is " + String.format("%.2f", progress) + "%");
+ if (size < batchSize) {
+ if (size > 0) {
+ doProccRows(intValueKV, rows, index);
+ }
+ break;
+ }
+ doProccRows(intValueKV, rows, index);
+ } catch (Exception e) {
+ throw new RuntimeException("put data error ", e);
+ }
+ }
+
+ countDownLatch.countDown();
+ }
+ }
+
+ public boolean dbInit() {
+ String uri = "/api/rest/rds/getDataSource";
+ String httpPrefix = "http";
+ String httpProtocol = "http://";
+ try {
+ String dbEndpoint = ComponentCreator.getProperties().getProperty(
+ ConfigureFileKey.INTELLIGENCE_TIP_DB_ENDPOINT);
+ if (StringUtils.isNotBlank(dbEndpoint)) {
+ if (!dbEndpoint.startsWith(httpPrefix)) {
+ dbEndpoint = httpProtocol + dbEndpoint;
+ }
+ String content = HttpUtil.getContent(dbEndpoint + uri);
+ if (StringUtils.isNotBlank(content)) {
+ JSONObject obj = JSON.parseObject(content);
+ String statusCode = "code";
+ int successCode = 200;
+ if (obj.getInteger(statusCode) == successCode) {
+ JSONObject data = obj.getJSONObject("data");
+ JSONObject dbInfo = data.getJSONObject("dBInfo");
+ if (dbInfo != null) {
+ String dbUrl = "jdbc:mysql://" + dbInfo.getString("dbConnection") + ":" + dbInfo.getInteger(
+ "port") + "/" + dbInfo.getString("dBName");
+ String dbUserName = dbInfo.getString("userName");
+ String dbPassword = dbInfo.getString("passWord");
+ JDBCDriver dataSource = (JDBCDriver)this.outputDataSource;
+ dataSource.setUrl(dbUrl);
+ dataSource.setPassword(dbPassword);
+ dataSource.setUserName(dbUserName);
+ dataSource.setHasInit(false);
+ dataSource.init();
+ return true;
+ }
+ }
+ }
+ }
+ return false;
+ } catch (Exception e) {
+ e.printStackTrace();
+ return false;
+ }
+ }
+
+ /**
+ * 把存储0/1字符串的值，转化成bit
+ *
+ * @param values
+ * @return
+ */
+ protected int createInt(List<String> values) {
+ return NumberUtils.createBitMapInt(values);
+ }
+
+ /**
+ * 获取某位的值，如果是1，返回字符串1，否则返回null
+ *
+ * @param num
+ * @param i
+ * @return
+ */
+ protected String getNumBitValue(int num, int i) {
+ boolean exist = NumberUtils.getNumFromBitMapInt(num, i);
+ if (exist) {
+ return "1";
+ }
+ return null;
+ }
+
+ protected abstract void doProccRows(IntValueKV intValueKV, List<Map<String, Object>> rows, int index);
+
+ public String getIdFieldName() {
+ return idFieldName;
+ }
+
+ public void setIdFieldName(String idFieldName) {
+ this.idFieldName = idFieldName;
+ }
+
+ public int getBatchSize() {
+ return batchSize;
+ }
+
+ public void setBatchSize(int batchSize) {
+ this.batchSize = batchSize;
+ }
+
+ public Long getPollingTimeMintue() {
+ return pollingTimeMintue;
+ }
+
+ public void setPollingTimeMintue(Long pollingTimeMintue) {
+ this.pollingTimeMintue = pollingTimeMintue;
+ }
+
+ public String getDatasourceName() {
+ return datasourceName;
+ }
+
+ public void setDatasourceName(String datasourceName) {
+ this.datasourceName = datasourceName;
+ }
+}
diff --git a/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/intelligence/AccountIntelligenceCache.java b/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/intelligence/AccountIntelligenceCache.java
new file mode 100644
index 0000000..9466f3e
--- /dev/null
+++ b/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/intelligence/AccountIntelligenceCache.java
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.dim.intelligence;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.rocketmq.streams.common.configurable.IAfterConfiguableRefreshListerner;
+import org.apache.rocketmq.streams.common.cache.compress.impl.IntValueKV;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+/**
+ * table: ads_yunsec_abnormal_account
+ */
+public class AccountIntelligenceCache extends AbstractIntelligenceCache implements IAfterConfiguableRefreshListerner {
+
+ private static final Log LOG = LogFactory.getLog(AccountIntelligenceCache.class);
+
+ /**
+ * 情报域名
+ */
+ protected transient String keyName = "account";
+
+ @Override
+ public Map<String, Object> getRow(String account) {
+ Integer value = intValueKV.get(account);
+ if (value == null) {
+ return null;
+ }
+ Map<String, Object> row = new HashMap<String, Object>() {{
+ put("account", account);
+ }};
+ return row;
+ }
+
+ @Override
+ protected String getSQL() {
+ return "SELECT id, `account` FROM `ads_yunsec_abnormal_account`";
+ }
+
+ @Override
+ public String getKeyName() {
+ return this.keyName;
+ }
+
+ @Override
+ public String getTableName() {
+ return "ads_yunsec_abnormal_account";
+ }
+
+ @Override
+ protected void doProccRows(IntValueKV intValueKV, List<Map<String, Object>> rows, int index) {
+ rows.forEach(row -> {
+ String account = (String)row.get(keyName);
+ if (account != null) {
+ intValueKV.put(account, 1);
+ }
+ });
+ }
+
+}
diff --git a/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/intelligence/DomainIntelligenceCache.java b/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/intelligence/DomainIntelligenceCache.java
new file mode 100644
index 0000000..72a88bf
--- /dev/null
+++ b/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/intelligence/DomainIntelligenceCache.java
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.dim.intelligence;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.rocketmq.streams.common.configurable.IAfterConfiguableRefreshListerner;
+import org.apache.rocketmq.streams.common.cache.compress.impl.IntValueKV;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+public class DomainIntelligenceCache extends AbstractIntelligenceCache implements IAfterConfiguableRefreshListerner {
+ private static final Log LOG = LogFactory.getLog(DomainIntelligenceCache.class);
+ protected transient String keyName = "domain";
+
+ @Override
+ protected String getSQL() {
+ return "SELECT id, `domain` , `is_malicious_source` , `is_phishing` , `is_c2` , `is_mining_pool` FROM `ads_yunsec_ti_domain_all_df` where curdate() < date_add(modify_time, interval expire_time day) ";
+ }
+
+ @Override
+ public Map<String, Object> getRow(String ip) {
+ Integer value = intValueKV.get(ip);
+ if (value == null) {
+ return null;
+ }
+ Map<String, Object> row = new HashMap<>();
+ row.put("is_malicious_source", getNumBitValue(value, 0));
+ row.put("is_phishing", getNumBitValue(value, 1));
+ row.put("is_c2", getNumBitValue(value, 2));
+ row.put("is_mining_pool", getNumBitValue(value, 3));
+ return row;
+ }
+
+ @Override
+ public String getKeyName() {
+ return this.keyName;
+ }
+
+ @Override
+ public String getTableName() {
+ return "ads_yunsec_ti_domain_all_df";
+ }
+
+ @Override
+ protected void doProccRows(IntValueKV intValueKV, List<Map<String, Object>> rows, int index) {
+ for (Map<String, Object> row : rows) {
+ String ip = (String)row.get(keyName);
+ if (ip == null) {
+ LOG.warn("load Intelligence exception ,the ip is null");
+ continue;
+ }
+ List<String> values = new ArrayList<>();
+ values.add((String)row.get("is_malicious_source"));
+ values.add((String)row.get("is_phishing"));
+ values.add((String)row.get("is_c2"));
+ values.add((String)row.get("is_mining_pool"));
+ int value = createInt(values);
+ synchronized (this) {
+ intValueKV.put(ip, value);
+ }
+
+ }
+ }
+
+}
diff --git a/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/intelligence/IPIntelligenceCache.java b/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/intelligence/IPIntelligenceCache.java
new file mode 100644
index 0000000..6382cd2
--- /dev/null
+++ b/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/intelligence/IPIntelligenceCache.java
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.dim.intelligence;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.rocketmq.streams.common.component.ComponentCreator;
+import org.apache.rocketmq.streams.common.configurable.IAfterConfiguableRefreshListerner;
+import org.apache.rocketmq.streams.common.cache.compress.impl.IntValueKV;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.rocketmq.streams.common.dboperator.IDBDriver;
+import org.apache.rocketmq.streams.db.driver.DriverBuilder;
+
+public class IPIntelligenceCache extends AbstractIntelligenceCache implements IAfterConfiguableRefreshListerner {
+ private static final Log LOG = LogFactory.getLog(IPIntelligenceCache.class);
+ protected transient String keyName = "ip";
+
+
+ @Override
+ protected String getSQL() {
+ return "SELECT id,ip, `is_web_attack` , `is_tor` , `is_proxy` , `is_nat` , `is_mining_pool` , `is_c2` , "
+ + "`is_malicious_source` , `is_3rd` , `is_idc` , `is_malicious_login` FROM `ads_yunsec_ti_ip_all_df` where curdate() < date_add(modify_time, interval expire_time day) ";
+ }
+
+ @Override
+ public String getKeyName() {
+ return this.keyName;
+ }
+
+ @Override
+ public String getTableName() {
+ return "ads_yunsec_ti_ip_all_df";
+ }
+
+ @Override
+ public Map<String, Object> getRow(String ip) {
+
+ Integer value = intValueKV.get(ip);
+ if (value == null) {
+ return null;
+ }
+ Map<String, Object> row = new HashMap<>();
+
+ row.put("is_web_attack", getNumBitValue(value, 0));
+ row.put("is_tor", getNumBitValue(value, 1));
+ row.put("is_proxy", getNumBitValue(value, 2));
+ row.put("is_nat", getNumBitValue(value, 3));
+ row.put("is_mining_pool", getNumBitValue(value, 4));
+ row.put("is_c2", getNumBitValue(value, 5));
+ row.put("is_malicious_source", getNumBitValue(value, 6));
+ row.put("is_3rd", getNumBitValue(value, 7));
+ row.put("is_idc", getNumBitValue(value, 8));
+ row.put("is_malicious_login", getNumBitValue(value, 9));
+ return row;
+ }
+
+ @Override
+ protected void doProccRows(IntValueKV intValueKV, List<Map<String, Object>> rows, int index) {
+ for (Map<String, Object> row : rows) {
+ String ip = (String)row.get(keyName);
+ if (ip == null) {
+ LOG.warn("load Intelligence exception ,the ip is null");
+ continue;
+ }
+ List<String> values = new ArrayList<>();
+ values.add((String)row.get("is_web_attack"));
+ values.add((String)row.get("is_tor"));
+ values.add((String)row.get("is_proxy"));
+ values.add((String)row.get("is_nat"));
+ values.add((String)row.get("is_mining_pool"));
+ values.add((String)row.get("is_c2"));
+ values.add((String)row.get("is_malicious_source"));
+ values.add((String)row.get("is_3rd"));
+ values.add((String)row.get("is_idc"));
+ values.add((String)row.get("is_malicious_login"));
+ int value = createInt(values);
+ synchronized (this) {
+ intValueKV.put(ip, value);
+ }
+ }
+ }
+
+ public static void main(String[] args) {
+ ComponentCreator.setProperties(
+ "/Users/yuanxiaodong/Documents/workdir/档案/阿里安全/专有云/2020/dipper-siem/siem.properties");
+ IPIntelligenceCache ipIntelligenceCache = new IPIntelligenceCache();
+ IDBDriver outputDataSource = DriverBuilder.createDriver();
+ ipIntelligenceCache.startLoadData(ipIntelligenceCache.getSQL(), outputDataSource);
+ }
+}
diff --git a/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/intelligence/URLIntelligenceCache.java b/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/intelligence/URLIntelligenceCache.java
new file mode 100644
index 0000000..8910469
--- /dev/null
+++ b/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/intelligence/URLIntelligenceCache.java
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.dim.intelligence;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.rocketmq.streams.common.configurable.IAfterConfiguableRefreshListerner;
+import org.apache.rocketmq.streams.common.cache.compress.impl.IntValueKV;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+public class URLIntelligenceCache extends AbstractIntelligenceCache implements IAfterConfiguableRefreshListerner {
+
+ private static final Log LOG = LogFactory.getLog(URLIntelligenceCache.class);
+
+ protected transient String keyName = "url";
+
+ @Override
+ protected String getSQL() {
+ return "SELECT id, url, `is_malicious_source` FROM `ads_yunsec_ti_url_all_df` where curdate() < date_add(modify_time, interval expire_time day) ";
+ }
+
+ @Override
+ public Map<String, Object> getRow(String ip) {
+ Integer value = intValueKV.get(ip);
+ if (value == null) {
+ return null;
+ }
+ Map<String, Object> row = new HashMap<>();
+
+ row.put("is_malicious_source", getNumBitValue(value, 0));
+ return row;
+ }
+
+ @Override
+ public String getKeyName() {
+ return this.keyName;
+ }
+
+ @Override
+ public String getTableName() {
+ return "ads_yunsec_ti_url_all_df";
+ }
+
+ @Override
+ protected void doProccRows(IntValueKV intValueKV, List<Map<String, Object>> rows, int index) {
+ for (Map<String, Object> row : rows) {
+ String ip = (String)row.get(keyName);
+ if (ip == null) {
+ LOG.warn("load Intelligence exception ,the ip is null");
+ continue;
+ }
+ List<String> values = new ArrayList<>();
+ values.add((String)row.get("is_malicious_source"));
+ int value = createInt(values);
+ synchronized (this) {
+ intValueKV.put(ip, value);
+ }
+
+ }
+ }
+
+}
diff --git a/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/model/AbstractDim.java b/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/model/AbstractDim.java
new file mode 100644
index 0000000..c51b0e6
--- /dev/null
+++ b/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/model/AbstractDim.java
@@ -0,0 +1,312 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.dim.model;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+import com.alibaba.fastjson.JSONObject;
+
+import org.apache.rocketmq.streams.dim.index.IndexExecutor;
+import org.apache.rocketmq.streams.dim.index.DimIndex;
+import org.apache.rocketmq.streams.filter.builder.ExpressionBuilder;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+import org.apache.rocketmq.streams.filter.operator.expression.Expression;
+import org.apache.rocketmq.streams.filter.operator.expression.RelationExpression;
+import org.apache.rocketmq.streams.common.cache.softreference.ICache;
+import org.apache.rocketmq.streams.common.cache.softreference.impl.SoftReferenceCache;
+import org.apache.rocketmq.streams.common.configurable.BasedConfigurable;
+import org.apache.rocketmq.streams.common.cache.CompressTable;
+import org.apache.rocketmq.streams.common.utils.DataTypeUtil;
+import org.apache.rocketmq.streams.common.utils.MapKeyUtil;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+/**
+ * 这个结构代表一张表 存放表的全部数据和索引
+ */
+public abstract class AbstractDim extends BasedConfigurable {
+
+ private static final Log LOG = LogFactory.getLog(AbstractDim.class);
+
+ public static final String TYPE = "nameList";
+
+ /**
+ * 同步数据的事件间隔，单位是分钟
+ */
+ protected Long pollingTimeMintue = 60L;
+
+ /**
+ * 支持多组索引，如果一个索引是组合索引，需要拼接成一个string，用;分割 建立索引后，会创建索引的数据结构，类似Map<String,List<RowId>，可以快速定位，无索引会全表扫描，不建议使用 如有两组索引：1.name 2. ip;address
+ */
+ protected List<String> indexs = new ArrayList<>();
+
+ /**
+ * 把表数据转化成二进制存储在CompressTable中
+ */
+ protected transient volatile CompressTable dataCache;
+
+ /**
+ * 建立名单的时候，可以指定多组索引，索引的值当作key，row在datacache的index当作value，可以快速匹配索引对应的row key：索引的值 value：row在dataCache的index当作value，可以快速匹配索引对应的row
+ */
+ protected transient DimIndex nameListIndex;
+
+ //是否是唯一索引，唯一索引会用IntValueKV存储，有更高的压缩率
+ protected boolean isUniqueIndex = false;
+
+ //定时加载表数据到内存
+ protected transient ScheduledExecutorService executorService;
+
+ public AbstractDim() {
+ this.setType(TYPE);
+ }
+
+ //protected String index;//只是做标记，为了是简化indexs的赋值
+
+ public String addIndex(String... fieldNames) {
+ return addIndex(this.indexs, fieldNames);
+ }
+
+ @Override
+ protected boolean initConfigurable() {
+ boolean success = super.initConfigurable();
+ loadNameList();
+ executorService = new ScheduledThreadPoolExecutor(3);
+ executorService.scheduleWithFixedDelay(new Runnable() {
+
+ @Override
+ public void run() {
+ loadNameList();
+ }
+ }, pollingTimeMintue, pollingTimeMintue, TimeUnit.MINUTES);
+ return success;
+ }
+
+ /**
+ * 加载维表数据 创建索引
+ */
+ protected void loadNameList() {
+ try {
+ LOG.info(getConfigureName() + " begin polling data");
+ //全表数据
+ CompressTable dataCacheVar = loadData();
+ this.dataCache = dataCacheVar;
+ this.nameListIndex = buildIndex(dataCacheVar);
+ } catch (Exception e) {
+ LOG.error("Load configurables error:" + e.getMessage(), e);
+ }
+ }
+
+ /**
+ * 给维表生成索引数据结构
+ *
+ * @param dataCacheVar 维表
+ * @return
+ */
+ protected DimIndex buildIndex(CompressTable dataCacheVar) {
+ DimIndex dimIndex = new DimIndex(this.indexs);
+ dimIndex.setUnique(isUniqueIndex);
+ dimIndex.buildIndex(dataCacheVar);
+ return dimIndex;
+ }
+
+ /**
+ * 根据索引名和索引值查询匹配的行号
+ *
+ * @param indexName
+ * @param indexValue
+ * @return
+ */
+ public List<Integer> findRowIdByIndex(String indexName, String indexValue) {
+ return nameListIndex == null ? Collections.emptyList() : nameListIndex.getRowIds(indexName, indexValue);
+ }
+
+ /**
+ * 软引用缓存，最大可能保存索引执行器，避免频繁创建，带来额外开销 同时会保护内存不被写爆，当内存不足时自动回收内存
+ */
+ private static ICache<String, IndexExecutor> cache = new SoftReferenceCache<>();
+
+ /**
+ * 先找索引，如果有索引，通过索引匹配。如果没有，全表扫表.
+ *
+ * @param expressionStr 表达式
+ * @param msg 消息
+ * @return 只返回匹配的第一行
+ */
+ public Map<String, Object> matchExpression(String expressionStr, JSONObject msg) {
+ List<Map<String, Object>> rows = matchExpression(expressionStr, msg, true, null);
+ if (rows != null && rows.size() > 0) {
+ return rows.get(0);
+ }
+ return null;
+ }
+
+ /**
+ * 先找索引，如果有索引，通过索引匹配。如果没有，全表扫表
+ *
+ * @param expressionStr 表达式
+ * @param msg 消息
+ * @return 返回全部匹配的行
+ */
+ public List<Map<String, Object>> matchExpression(String expressionStr, JSONObject msg, boolean needAll, String script) {
+ IndexExecutor indexNamelistExecutor = cache.get(expressionStr);
+ if (indexNamelistExecutor == null) {
+ indexNamelistExecutor = new IndexExecutor(expressionStr, getNameSpace(), this.indexs);
+ cache.put(expressionStr, indexNamelistExecutor);
+ }
+ if (indexNamelistExecutor.isSupport()) {
+ return indexNamelistExecutor.match(msg, this, needAll, script);
+ } else {
+ return matchExpressionByLoop(expressionStr, msg, needAll);
+ }
+ }
+
+ /**
+ * 全表扫描，做表达式匹配，返回全部匹配结果
+ *
+ * @param expressionStr
+ * @param msg
+ * @param needAll
+ * @return
+ */
+ protected List<Map<String, Object>> matchExpressionByLoop(String expressionStr, JSONObject msg, boolean needAll) {
+ CompressTable dataCache = this.dataCache;
+ List<Map<String, Object>> rows = matchExpressionByLoop(dataCache.newIterator(), expressionStr, msg, needAll);
+ return rows;
+ }
+
+ /**
+ * 全表扫描，做表达式匹配，返回全部匹配结果。join中有使用
+ *
+ * @param expressionStr
+ * @param msg
+ * @param needAll
+ * @return
+ */
+ public static List<Map<String, Object>> matchExpressionByLoop(Iterator<Map<String, Object>> it, String expressionStr, JSONObject msg, boolean needAll) {
+ List<Map<String, Object>> rows = new ArrayList<>();
+ while (it.hasNext()) {
+ Map<String, Object> values = it.next();
+ Rule ruleTemplete = ExpressionBuilder.createRule("tmp", "tmpRule", expressionStr);
+ Rule rule = ruleTemplete.copy();
+ Map<String, Expression> expressionMap = new HashMap<>();
+ for (Expression expression : rule.getExpressionMap().values()) {
+ expressionMap.put(expression.getConfigureName(), expression);
+ if (RelationExpression.class.isInstance(expression)) {
+ continue;
+ }
+ Object object = expression.getValue();
+ if (object != null && DataTypeUtil.isString(object.getClass())) {
+ String fieldName = (String)object;
+ Object value = values.get(fieldName);
+ if (value != null) {
+ Expression e = expression.copy();
+ e.setValue(value.toString());
+ expressionMap.put(e.getConfigureName(), e);
+ }
+ }
+ }
+ rule.setExpressionMap(expressionMap);
+ boolean matched = rule.execute(msg);
+ if (matched) {
+ rows.add(values);
+ if (needAll == false) {
+ return rows;
+ }
+ }
+ }
+ return rows;
+ }
+
+ protected abstract CompressTable loadData();
+
+ @Override
+ public void destroy() {
+ super.destroy();
+ executorService.shutdown();
+ }
+
+ /**
+ * 设置索引
+ *
+ * @param indexs 字段名称，多个字段";"分隔
+ */
+ public void setIndex(String indexs) {
+ if (StringUtil.isEmpty(indexs)) {
+ return;
+ }
+ List<String> tmp = new ArrayList<>();
+ String[] values = indexs.split(";");
+ this.addIndex(tmp, values);
+ this.indexs = tmp;
+ }
+
+ /**
+ * 建议指定索引，会基于索引建立map，对于等值的判断，可以快速匹配
+ *
+ * @param fieldNames
+ */
+ private String addIndex(List<String> indexs, String... fieldNames) {
+ if (fieldNames == null) {
+ return null;
+ }
+ Arrays.sort(fieldNames);
+ String index = MapKeyUtil.createKey(fieldNames);
+ if (StringUtil.isNotEmpty(index)) {
+ indexs.add(index);
+ }
+ return index;
+ }
+
+ public Long getPollingTimeMintue() {
+ return pollingTimeMintue;
+ }
+
+ public void setPollingTimeMintue(Long pollingTimeMintue) {
+ this.pollingTimeMintue = pollingTimeMintue;
+ }
+
+ public List<String> getIndexs() {
+ return indexs;
+ }
+
+ public void setIndexs(List<String> indexs) {
+ this.indexs = indexs;
+ }
+
+ public CompressTable getDataCache() {
+ return dataCache;
+ }
+
+ public boolean isUniqueIndex() {
+ return isUniqueIndex;
+ }
+
+ public void setUniqueIndex(boolean uniqueIndex) {
+ isUniqueIndex = uniqueIndex;
+ }
+}
diff --git a/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/model/BooleanFieldDBDim.java b/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/model/BooleanFieldDBDim.java
new file mode 100644
index 0000000..80dd2d2
--- /dev/null
+++ b/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/model/BooleanFieldDBDim.java
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.dim.model;
+
+import java.util.List;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.rocketmq.streams.common.metadata.MetaData;
+import org.apache.rocketmq.streams.common.metadata.MetaDataField;
+
+/**
+ * 类似情报表，除了一个核心比对字段和主键，其他都是boolean类型的/或者是只有0/1两个值的字符串和int。可以用这个存储结构
+ */
+public class BooleanFieldDBDim extends DBDim {
+
+ private static final Log LOG = LogFactory.getLog(BooleanFieldDBDim.class);
+
+ /**
+ * 类似情报表，除了一个核心比对字段和主键，其他都是boolean类型的。可以用这个存储结构
+ *
+ * @param metaData
+ * @param notBooleanFieldName
+ * @return
+ */
+ public static boolean support(MetaData metaData, String notBooleanFieldName) {
+ List<MetaDataField> metaDataFields = metaData.getMetaDataFields();
+ for (MetaDataField metaDataField : metaDataFields) {
+ if (metaDataField.getIsPrimary()) {
+ continue;
+ }
+ if (metaDataField.getFieldName().equals(notBooleanFieldName)) {
+ continue;
+ }
+ if (!metaDataField.getDataType().matchClass(Boolean.class)) {
+ return false;
+ }
+ }
+ return true;
+ }
+}
diff --git a/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/model/DBDim.java b/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/model/DBDim.java
new file mode 100644
index 0000000..6180820
--- /dev/null
+++ b/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/model/DBDim.java
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.dim.model;
+
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.apache.rocketmq.streams.common.component.ComponentCreator;
+import org.apache.rocketmq.streams.common.configurable.annotation.ENVDependence;
+import org.apache.rocketmq.streams.common.cache.CompressTable;
+import org.apache.rocketmq.streams.common.utils.IPUtil;
+import org.apache.rocketmq.streams.common.utils.MapKeyUtil;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.rocketmq.streams.db.driver.JDBCDriver;
+import org.apache.rocketmq.streams.db.driver.DriverBuilder;
+
+public class DBDim extends AbstractDim {
+
+ private static final Log LOG = LogFactory.getLog(DBDim.class);
+
+ private String jdbcdriver = "com.mysql.jdbc.Driver";
+
+ @ENVDependence
+ private String url;
+
+ @ENVDependence
+ private String userName;
+
+ @ENVDependence
+ private String password;
+
+ private String sql;//sql 会被定时执行
+
+ private static transient AtomicInteger nameCreator = new AtomicInteger(0);
+
+ /**
+ * 是否支持批量查找
+ */
+ protected transient Boolean supportBatch = false;
+
+ public DBDim() {
+ this.setConfigureName(MapKeyUtil.createKey(IPUtil.getLocalIdentification(), System.currentTimeMillis() + "",
+ nameCreator.incrementAndGet() + ""));
+ this.setType(TYPE);
+ }
+
+ @Override
+ protected CompressTable loadData() {
+ List<Map<String, Object>> rows = executeQuery();
+ CompressTable tableCompress = new CompressTable();
+ for (Map<String, Object> row : rows) {
+ tableCompress.addRow(row);
+ }
+ return tableCompress;
+ }
+
+ protected List<Map<String, Object>> executeQuery() {
+ JDBCDriver resource = createResouce();
+ try {
+ List<Map<String, Object>> result = resource.queryForList(sql);
+ ;
+ LOG.info("load configurable's count is " + result.size());
+ return result;
+ } finally {
+ if (resource != null) {
+ resource.destroy();
+ }
+ }
+
+ }
+
+ protected JDBCDriver createResouce() {
+ return DriverBuilder.createDriver(jdbcdriver, url, userName, password);
+ }
+
+ public void setJdbcdriver(String jdbcdriver) {
+ this.jdbcdriver = jdbcdriver;
+ }
+
+ public void setUrl(String url) {
+ this.url = url;
+ }
+
+ public void setUserName(String userName) {
+ this.userName = userName;
+ }
+
+ public void setPassword(String password) {
+ this.password = password;
+ }
+
+ public void setSql(String sql) {
+ this.sql = sql;
+ }
+
+ public String getJdbcdriver() {
+ return jdbcdriver;
+ }
+
+ public String getUrl() {
+ return url;
+ }
+
+ public String getUserName() {
+ return userName;
+ }
+
+ public String getPassword() {
+ return password;
+ }
+
+ public String getSql() {
+ return sql;
+ }
+
+ public Boolean getSupportBatch() {
+ return supportBatch;
+ }
+
+ public void setSupportBatch(Boolean supportBatch) {
+ this.supportBatch = supportBatch;
+ }
+
+}
diff --git a/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/service/IDimService.java b/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/service/IDimService.java
new file mode 100644
index 0000000..c5c4a2e
--- /dev/null
+++ b/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/service/IDimService.java
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.dim.service;
+
+import java.util.List;
+import java.util.Map;
+
+public interface IDimService {
+
+ /**
+ * 做维表join，关系通过表达式表示，返回所有匹配的行。因为msg没有key，表达式中，以下标表示key，如0，1，2。
+ *
+ * @param dimName 维表的名称
+ * @param expressionStr 表达式（0,functionName,filedName)&（1,functionName,filedName)|（2,functionName,filedName)
+ * @param msgs 流数据
+ * @return 符合匹配条件的所有行
+ */
+ Map<String, Object> match(String dimName, String expressionStr, Object... msgs);
+
+ /**
+ * 做维表join，关系通过表达式表示，返回所有匹配的行。
+ *
+ * @param dimName 维表的名称
+ * @param expressionStr 表达式,varName是msg中的key名称（varName,functionName,filedName)&（varName,functionName,filedName)|（varName,functionName,filedName)
+ * @param msgs 流数据
+ * @return 符合匹配条件的所有行
+ */
+ List<Map<String, Object>> matchSupportMultiRow(String dimName, String expressionStr, Map<String, Object> msgs);
+
+ /**
+ * 做维表join，关系通过表达式表示，返回匹配的一行数据，如果有多行匹配，只返回第一行。
+ *
+ * @param dimName 维表的名称
+ * @param expressionStr 表达式,varName是msg中的key名称（varName,functionName,filedName)&（varName,functionName,filedName)|（varName,functionName,filedName)
+ * @param msgs 流数据
+ * @return 返回匹配的一行数据，如果有多行匹配，只返回第一行。
+ */
+ Map<String, Object> match(String dimName, String expressionStr, Map<String, Object> msgs);
+
+ /**
+ * 做维表join，关系通过表达式表示，返回匹配的全部数据。
+ *
+ * @param dimName 维表的名称
+ * @param expressionStr 表达式,varName是msg中的key名称（varName,functionName,filedName)&（varName,functionName,filedName)|（varName,functionName,filedName)
+ * @param msgs 流数据
+ * @param script 对维表字段做处理的函数,在执行表达式前需要先对维表字段做处理，如fiedlName=trim(fieldName)
+ * @return 返回匹配的一行数据，如果有多行匹配，只返回第一行。
+ */
+ List<Map<String, Object>> matchSupportMultiRow(String dimName,
+ String expressionStr, Map<String, Object> msgs, String script);
+}
diff --git a/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/service/impl/DimServiceImpl.java b/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/service/impl/DimServiceImpl.java
new file mode 100644
index 0000000..20f22ae
--- /dev/null
+++ b/rocketmq-streams-dim/src/main/java/org/apache/rocketmq/streams/dim/service/impl/DimServiceImpl.java
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.dim.service.impl;
+
+import java.util.List;
+import java.util.Map;
+
+import com.alibaba.fastjson.JSONObject;
+
+import org.apache.rocketmq.streams.configurable.ConfigurableComponent;
+import org.apache.rocketmq.streams.dim.model.AbstractDim;
+import org.apache.rocketmq.streams.dim.service.IDimService;
+
+public class DimServiceImpl implements IDimService {
+ protected ConfigurableComponent configurableComponent;
+
+ public DimServiceImpl(ConfigurableComponent configurableComponent) {
+ this.configurableComponent = configurableComponent;
+ }
+
+ /**
+ * 传入要比对的字段，进行规则匹配。字段和名单的比对逻辑，写在规则中
+ *
+ * @param msgs 字段名默认为数组的索引，如1，2，3
+ * @return
+ */
+ @Override
+ public Map<String, Object> match(String dimName, String expressionStr, Object... msgs) {
+ if (msgs == null || msgs.length == 0) {
+ return null;
+ }
+ int i = 0;
+ JSONObject jsonObject = new JSONObject();
+ for (Object o : msgs) {
+ jsonObject.put(i + "", o);
+ i++;
+ }
+ return match(dimName, expressionStr, jsonObject);
+ }
+
+ @Override
+ public List<Map<String, Object>> matchSupportMultiRow(String dimName, String expressionStr, Map<String, Object> msgs) {
+ return matchSupportMultiRow(dimName, expressionStr, msgs, null);
+ }
+
+ @Override
+ public List<Map<String, Object>> matchSupportMultiRow(String dimName, String expressionStr, Map<String, Object> msgs, String script) {
+ JSONObject jsonObject = createJsonable(msgs);
+ AbstractDim nameList = configurableComponent.queryConfigurable(AbstractDim.TYPE, dimName);
+ if (nameList != null) {
+ return nameList.matchExpression(expressionStr, jsonObject, true, script);
+ } else {
+ return null;
+ }
+ }
+
+ @Override
+ public Map<String, Object> match(String nameListName, String expressionStr, Map<String, Object> parameters) {
+ JSONObject jsonObject = createJsonable(parameters);
+ AbstractDim nameList = configurableComponent.queryConfigurable(AbstractDim.TYPE, nameListName);
+ if (nameList != null) {
+ return nameList.matchExpression(expressionStr, jsonObject);
+ } else {
+ return null;
+ }
+ }
+
+ private JSONObject createJsonable(Map<String, Object> parameters) {
+ JSONObject jsonObject = null;
+ if (parameters instanceof JSONObject) {
+ jsonObject = (JSONObject)parameters;
+ } else {
+ jsonObject.putAll(parameters);
+ }
+ return jsonObject;
+ }
+
+}
diff --git a/rocketmq-streams-dim/src/test/java/com/aliyun/service/ConfigureLoaderTest.java b/rocketmq-streams-dim/src/test/java/com/aliyun/service/ConfigureLoaderTest.java
new file mode 100644
index 0000000..3f43a2a
--- /dev/null
+++ b/rocketmq-streams-dim/src/test/java/com/aliyun/service/ConfigureLoaderTest.java
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.aliyun.service;
+
+import org.junit.Test;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStreamReader;
+
+public class ConfigureLoaderTest {
+
+ @Test
+ public void testLoadResource() throws IOException {
+ // BufferedReader br = new BufferedReader(new InputStreamReader(RuleEngineRunner.class.getClassLoader
+ // ().getResourceAsStream(".")));
+ // String line = br.readLine();
+ // while (line != null) {
+ // System.out.println(line);
+ // line = br.readLine();
+ // }
+ }
+}
diff --git a/rocketmq-streams-dim/src/test/java/com/aliyun/service/ExpressionExecutorTest.java b/rocketmq-streams-dim/src/test/java/com/aliyun/service/ExpressionExecutorTest.java
new file mode 100644
index 0000000..380952f
--- /dev/null
+++ b/rocketmq-streams-dim/src/test/java/com/aliyun/service/ExpressionExecutorTest.java
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.aliyun.service;
+
+import com.alibaba.fastjson.JSONObject;
+import org.apache.rocketmq.streams.filter.builder.ExpressionBuilder;
+import org.apache.rocketmq.streams.filter.operator.expression.SimpleExpression;
+import org.apache.rocketmq.streams.filter.FilterComponent;
+import org.junit.Test;
+
+import java.io.File;
+
+public class ExpressionExecutorTest {
+ private static final String CREDIBLE_PROPERTIES = "credible" + File.separator + "credible.properties";
+ private FilterComponent filterComponent;
+
+ private String namespace = "yundun.credible.net.vistor";
+ private String ruleNameSpace = "credible.rule.net.vistor";
+ private String selectorName = "credible.selector.net.vistor";
+ private String selectorExpression = "(host_uuid,=,0a2153e2-e45c-403f-8d5f-d811f400c3fb)";
+ private String procWriteList = "credible.namelist.proc";
+ private String netWriteList = "credible.namelist.net.vistor";
+
+ private String ruleExpression =
+ "(proc_path,in_resouce," + namespace + "->" + procWriteList + ")&(inner_message,not_in_expression_resouce,'"
+ + namespace + "->" + netWriteList
+ + "->(visitor_ip,=,dest_ip)&(visitor_port,=,dest_port)&(proc_path,=,program_path)')";
+
+ // @Test
+ // public void parseExpression() {
+ // List<Expression> expressions = new ArrayList<>();
+ // List<RelationExpression> relationExpressions = new ArrayList<>();
+ // Expression expression = ExpressionBuilder.createExpression("namespace", ruleExpression,
+ // expressions,
+ // relationExpressions);
+ // }
+
+ public ExpressionExecutorTest() {
+ // FilterComponent filterComponent= new FilterComponent();
+ // filterComponent.init(CREDIBLE_PROPERTIES);
+ // filterComponent.start(null);
+ // this.filterComponent=filterComponent;
+ }
+
+ @Test
+ public void testExecutor() {
+ System.out.println("hello wolrd");
+ JSONObject msg = new JSONObject();
+ msg.put("ip", "1.1.1.1");
+ boolean match = ExpressionBuilder.executeExecute(new SimpleExpression("ip", "=", "1.1.1.1"), msg);
+ System.out.println(match);
+ }
+
+ @Test
+ public void testRelationExecutor() {
+ JSONObject jsonObject = new JSONObject();
+ jsonObject.put("ip", "1.2.2.3");
+ jsonObject.put("uid", "1224");
+ jsonObject.put("vmip", "1.1.1.1");
+
+ boolean value =
+ ExpressionBuilder.executeExecute("namespace", "(ip,=,1.2.2.3)&((uid,=,12214)|(vmip,=,1.1.11.1))",
+ jsonObject);
+ System.out.println(value);
+ }
+}
diff --git a/rocketmq-streams-dim/src/test/java/com/aliyun/service/JsonParserTest.java b/rocketmq-streams-dim/src/test/java/com/aliyun/service/JsonParserTest.java
new file mode 100644
index 0000000..8f58a04
--- /dev/null
+++ b/rocketmq-streams-dim/src/test/java/com/aliyun/service/JsonParserTest.java
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.aliyun.service;
+
+import com.alibaba.fastjson.JSON;
+import com.alibaba.fastjson.JSONArray;
+import com.alibaba.fastjson.JSONObject;
+import org.junit.Test;
+
+public class JsonParserTest {
+
+ @Test
+ public void testJson() {
+ String array =
+ "[{\"value\":\"groupname\",\"key\":\"group_name\"},{\"value\":\"username\",\"key\":\"user_name\"},"
+ + "{\"value\":\"seq\",\"key\":\"index\"},{\"value\":\"egroupid\",\"key\":\"egroup_id\"},"
+ + "{\"value\":\"filepath\",\"key\":\"file_path\"},{\"value\":\"groupid\",\"key\":\"group_id\"},"
+ + "{\"value\":\"pfilename\",\"key\":\"pfile_path\"},{\"value\":\"safe_mode\",\"key\":\"perm\"},"
+ + "{\"value\":\"cmdline\",\"key\":\"cmd_line\"}]";
+ String jsonStr =
+ "{\"className\":\"com.aliyun.filter.result.FieldReNameScript\",\"oldField2NewFiled\":" + array + "}";
+ JSONArray jsonObject = JSON.parseArray(array);
+ JSONObject js = JSON.parseObject(jsonStr);
+ System.out.println(js.toJSONString());
+ }
+}
diff --git a/rocketmq-streams-dim/src/test/java/com/aliyun/service/NameListFunctionTest.java b/rocketmq-streams-dim/src/test/java/com/aliyun/service/NameListFunctionTest.java
new file mode 100644
index 0000000..80d044c
--- /dev/null
+++ b/rocketmq-streams-dim/src/test/java/com/aliyun/service/NameListFunctionTest.java
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.aliyun.service;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import com.alibaba.fastjson.JSONObject;
+
+import org.apache.rocketmq.streams.dim.model.AbstractDim;
+import org.apache.rocketmq.streams.dim.model.DBDim;
+import org.junit.Test;
+
+public class NameListFunctionTest {
+
+ @Test
+ public void testNameList() {
+ AbstractDim nameList = create();
+ JSONObject msg = new JSONObject();
+ msg.put("ip", "47.105.77.144");
+ msg.put("vpcId", "1");
+ msg.put("now", "2019-07-18 17:33:29");
+ long start = System.currentTimeMillis();
+ }
+
+ @Test
+ public void testNameList2() {
+ AbstractDim nameList = createMapping();
+ JSONObject msg = new JSONObject();
+ msg.put("levelFile", "aegis-vul_record:level");
+ msg.put("levelValue", "high");
+ msg.put("now", "2019-07-18 17:33:29");
+ long start = System.currentTimeMillis();
+ }
+
+ private AbstractDim create() {
+ DBDim dbNameList = new DBDim();
+ dbNameList.setNameSpace("soc");
+ dbNameList.setConfigureName("isoc_field_mappings");
+ dbNameList.setUrl("");
+ dbNameList.setUserName("");
+ dbNameList.setPassword("");
+ dbNameList.setSql("SELECT * FROM `ecs_info` WHERE STATUS=1 LIMIT 1");
+ List<String> ipFieldNames = new ArrayList<>();
+ ipFieldNames.add("public_ips");
+ ipFieldNames.add("inner_ips");
+ ipFieldNames.add("eip");
+ ipFieldNames.add("private_ips");
+ dbNameList.init();
+ return dbNameList;
+ }
+
+ @Test
+ public void testNameListAllRow() {
+ AbstractDim nameList = createMapping();
+ JSONObject msg = new JSONObject();
+ msg.put("levelFile", "aegis-vul_record:level");
+ msg.put("levelValue", "high");
+ msg.put("now", "2019-07-18 17:33:29");
+ long start = System.currentTimeMillis();
+
+ }
+
+ private AbstractDim createMapping() {
+ DBDim dbNameList = new DBDim();
+ dbNameList.setNameSpace("soc");
+ dbNameList.setConfigureName("isoc_field_mappings");
+ dbNameList.setUrl("");
+ dbNameList.setUserName("");
+ dbNameList.setPassword("");
+ dbNameList.setSql("select * from ads_yunsec_ti_url_all_df limit 100000");
+ dbNameList.init();
+ return dbNameList;
+ }
+
+}
diff --git a/rocketmq-streams-dim/src/test/java/com/aliyun/service/TableCompressTest.java b/rocketmq-streams-dim/src/test/java/com/aliyun/service/TableCompressTest.java
new file mode 100644
index 0000000..3c89fd6
--- /dev/null
+++ b/rocketmq-streams-dim/src/test/java/com/aliyun/service/TableCompressTest.java
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.aliyun.service;
+
+import org.junit.Test;
+
+public class TableCompressTest {
+
+ @Test
+ public void testNameList() throws InterruptedException {
+ }
+}
diff --git a/rocketmq-streams-lease/pom.xml b/rocketmq-streams-lease/pom.xml
new file mode 100755
index 0000000..a0ad067
--- /dev/null
+++ b/rocketmq-streams-lease/pom.xml
@@ -0,0 +1,25 @@
+<?xml version="1.0" encoding="utf-8"?>
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams</artifactId>
+ <version>2.0.0-SNAPSHOT</version>
+ </parent>
+ <artifactId>rocketmq-streams-lease</artifactId>
+ <name>ROCKETMQ STREAMS :: lease</name>
+ <packaging>jar</packaging>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams-channel-db</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams-db-operator</artifactId>
+ </dependency>
+ </dependencies>
+</project>
diff --git a/rocketmq-streams-lease/src/main/java/org/apache/rocketmq/streams/lease/LeaseComponent.java b/rocketmq-streams-lease/src/main/java/org/apache/rocketmq/streams/lease/LeaseComponent.java
new file mode 100644
index 0000000..3a527a3
--- /dev/null
+++ b/rocketmq-streams-lease/src/main/java/org/apache/rocketmq/streams/lease/LeaseComponent.java
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */package org.apache.rocketmq.streams.lease;
+
+import java.util.Properties;
+
+import org.apache.rocketmq.streams.common.component.AbstractComponent;
+import org.apache.rocketmq.streams.common.component.ComponentCreator;
+import org.apache.rocketmq.streams.common.component.ConfigureDescriptor;
+import org.apache.rocketmq.streams.common.configure.ConfigureFileKey;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+import org.apache.rocketmq.streams.configurable.service.ConfigurableServcieType;
+import org.apache.rocketmq.streams.lease.service.ILeaseService;
+import org.apache.rocketmq.streams.lease.service.ILeaseStorage;
+import org.apache.rocketmq.streams.lease.service.impl.LeaseServiceImpl;
+import org.apache.rocketmq.streams.lease.service.impl.MockLeaseImpl;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.rocketmq.streams.lease.service.storages.DBLeaseStorage;
+import org.apache.rocketmq.streams.serviceloader.ServiceLoaderComponent;
+
+/**
+ * 通过db实现租约和锁，可以更轻量级，减少其他中间件的依赖 使用主备场景，只有一个实例运行，当当前实例挂掉，在一定时间内，会被其他实例接手 也可以用于全局锁
+ *
+ * @date 1/9/19
+ */
+public class LeaseComponent extends AbstractComponent<ILeaseService> {
+
+ private static LeaseComponent leaseComponent = null;
+ private static final Log LOG = LogFactory.getLog(LeaseComponent.class);
+ private ILeaseService leaseService;
+
+ public LeaseComponent() {
+ initConfigurableServiceDescriptor();
+ addConfigureDescriptor(
+ new ConfigureDescriptor(CONNECT_TYPE, false, ConfigurableServcieType.DEFAULT_SERVICE_NAME));
+ }
+
+ public static LeaseComponent getInstance() {
+ if(leaseComponent==null){
+ synchronized (LeaseComponent.class){
+ if(leaseComponent==null){
+ leaseComponent =ComponentCreator.getComponent(null,LeaseComponent.class);
+ }
+ }
+ }
+ return leaseComponent;
+ }
+
+ @Override
+ public boolean stop() {
+ return true;
+ }
+
+ @Override
+ public ILeaseService getService() {
+ return leaseService;
+ }
+
+ @Override
+ protected boolean startComponent(String namespace) {
+ return true;
+ }
+
+ @Override
+ protected boolean initProperties(Properties properties) {
+ String connectType = properties.getProperty(JDBC_URL);
+ if (StringUtil.isEmpty(connectType)) {
+ MockLeaseImpl mockLease = new MockLeaseImpl();
+ this.leaseService=mockLease;
+ return true;
+ }
+
+ LeaseServiceImpl leaseService= new LeaseServiceImpl();
+ String storageName=ComponentCreator.getProperties().getProperty(ConfigureFileKey.LEASE_STORAGE_NAME);
+ ILeaseStorage storasge=null;
+ if(StringUtil.isEmpty(storageName)){
+ String jdbc = properties.getProperty(AbstractComponent.JDBC_DRIVER);
+ String url = properties.getProperty(AbstractComponent.JDBC_URL);
+ String userName = properties.getProperty(AbstractComponent.JDBC_USERNAME);
+ String password = properties.getProperty(AbstractComponent.JDBC_PASSWORD);
+ storasge=new DBLeaseStorage(jdbc,url,userName,password);
+ }else {
+ storasge= (ILeaseStorage)ServiceLoaderComponent.getInstance(ILeaseStorage.class).loadService(storageName);
+ }
+ leaseService.setLeaseStorage(storasge);
+ this.leaseService=leaseService;
+ return true;
+ }
+}
diff --git a/rocketmq-streams-lease/src/main/java/org/apache/rocketmq/streams/lease/model/LeaseInfo.java b/rocketmq-streams-lease/src/main/java/org/apache/rocketmq/streams/lease/model/LeaseInfo.java
new file mode 100644
index 0000000..469a711
--- /dev/null
+++ b/rocketmq-streams-lease/src/main/java/org/apache/rocketmq/streams/lease/model/LeaseInfo.java
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.lease.model;
+
+/**
+ * 租约对象，需要创建租约表。
+ */
+
+import java.io.Serializable;
+import java.util.Date;
+
+public class LeaseInfo implements Serializable {
+ private static final long serialVersionUID = 665608838255753618L;
+ private Long id;
+ private Date createTime;
+ private Date updateTime;
+ private String leaseName;//租约名称，多个进程共享一个租约，只要名称相同即可
+ private String leaseUserIp;//区分不同的租约实体，以前默认用ip，但一个机器多个进程的情况下，用ip会区分不开，后续会加上进程号
+ private Date leaseEndDate;//租约到期时间
+ private int status;//租约的有效状态
+ private long version;//版本，通过版本保证更新原子性
+
+ public LeaseInfo() {
+ }
+
+ /**
+ * 建表语句
+ *
+ * @return
+ */
+ public static String createTableSQL() {
+ return "CREATE TABLE IF NOT EXISTS `lease_info` (\n"
+ + " `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT COMMENT '主键',\n"
+ + " `gmt_create` datetime NOT NULL COMMENT '创建时间',\n"
+ + " `gmt_modified` datetime NOT NULL COMMENT '修改时间',\n"
+ + " `lease_name` varchar(255) NOT NULL COMMENT '租约名称',\n"
+ + " `lease_user_ip` varchar(255) NOT NULL COMMENT '租者IP',\n"
+ + " `lease_end_time` varchar(255) NOT NULL COMMENT '租约到期时间',\n"
+ + " `status` int(11) NOT NULL DEFAULT '1' COMMENT '状态',\n"
+ + " `version` bigint(20) NOT NULL COMMENT '版本',\n"
+ + " `candidate_lease_ip` varchar(255) DEFAULT NULL COMMENT '候选租约ip',\n"
+ + " PRIMARY KEY (`id`),\n"
+ + " UNIQUE KEY `uk_name` (`lease_name`)\n"
+ + ") ENGINE=InnoDB AUTO_INCREMENT=8150 DEFAULT CHARSET=utf8 COMMENT='租约信息'\n"
+ + ";";
+ }
+
+ public Long getId() {
+ return this.id;
+ }
+
+ public void setId(Long id) {
+ this.id = id;
+ }
+
+ public Date getCreateTime() {
+ return this.createTime;
+ }
+
+ public void setCreateTime(Date createTime) {
+ this.createTime = createTime;
+ }
+
+ public Date getUpdateTime() {
+ return this.updateTime;
+ }
+
+ public void setUpdateTime(Date updateTime) {
+ this.updateTime = updateTime;
+ }
+
+ public String getLeaseUserIp() {
+ return this.leaseUserIp;
+ }
+
+ public void setLeaseUserIp(String leaseUserIp) {
+ this.leaseUserIp = leaseUserIp;
+ }
+
+ public Date getLeaseEndDate() {
+ return this.leaseEndDate;
+ }
+
+ public void setLeaseEndDate(Date leaseEndDate) {
+ this.leaseEndDate = leaseEndDate;
+ }
+
+ public int getStatus() {
+ return this.status;
+ }
+
+ public void setStatus(int status) {
+ this.status = status;
+ }
+
+ public String getLeaseName() {
+ return this.leaseName;
+ }
+
+ public void setLeaseName(String leaseName) {
+ this.leaseName = leaseName;
+ }
+
+ public long getVersion() {
+ return this.version;
+ }
+
+ public void setVersion(long version) {
+ this.version = version;
+ }
+
+}
+
diff --git a/rocketmq-streams-lease/src/main/java/org/apache/rocketmq/streams/lease/service/ILeaseGetCallback.java b/rocketmq-streams-lease/src/main/java/org/apache/rocketmq/streams/lease/service/ILeaseGetCallback.java
new file mode 100644
index 0000000..ad63dde
--- /dev/null
+++ b/rocketmq-streams-lease/src/main/java/org/apache/rocketmq/streams/lease/service/ILeaseGetCallback.java
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.lease.service;
+
+import java.util.Date;
+
+public interface ILeaseGetCallback {
+
+ /**
+ * 当成功获取租约时，回调接口
+ *
+ * @param nextLeaseDate 租约到期时间
+ */
+ void callback(Date nextLeaseDate);
+
+}
diff --git a/rocketmq-streams-lease/src/main/java/org/apache/rocketmq/streams/lease/service/ILeaseService.java b/rocketmq-streams-lease/src/main/java/org/apache/rocketmq/streams/lease/service/ILeaseService.java
new file mode 100644
index 0000000..7782c52
--- /dev/null
+++ b/rocketmq-streams-lease/src/main/java/org/apache/rocketmq/streams/lease/service/ILeaseService.java
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.lease.service;
+
+import java.util.List;
+import java.util.concurrent.Future;
+
+import org.apache.rocketmq.streams.lease.model.LeaseInfo;
+
+/**
+ * 通过db实现租约和锁，可以更轻量级，减少其他中间件的依赖 使用主备场景，只有一个实例运行，当当前实例挂掉，在一定时间内，会被其他实例接手 也可以用于全局锁
+ */
+public interface ILeaseService {
+
+ /**
+ * 默认锁定时间
+ */
+ static final int DEFALUT_LOCK_TIME = 60 * 5;
+
+ /**
+ * 检查某用户当前时间是否具有租约。这个方法是纯内存操作，无性能开销
+ *
+ * @return true，租约有效；false，租约无效
+ */
+ boolean hasLease(String name);
+
+ /**
+ * 申请租约，会启动一个线程，不停申请租约，直到申请成功。 申请成功后，每 租期/2 续约。 如果目前被其他租户获取租约，只有在对方租约失效，后才允许新的租户获取租约
+ *
+ * @param name 租约名称，无特殊要求，相同名称会竞争租约
+ */
+ void startLeaseTask(String name);
+
+ /**
+ * 申请租约，会启动一个线程，不停申请租约，直到申请成功。 申请成功后，每 租期/2 续约。 如果目前被其他租户获取租约，只有在对方租约失效，后才允许新的租户获取租约
+ *
+ * @param name 租约名称，无特殊要求，相同名称会竞争租约
+ * @param callback 当第一获取租约时，回调此函数
+ */
+ void startLeaseTask(final String name, ILeaseGetCallback callback);
+
+ /**
+ * 申请租约，会启动一个线程，不停申请租约，直到申请成功。 申请成功后，每 租期/2 续约。 如果目前被其他租户获取租约，只有在对方租约失效，后才允许新的租户获取租约
+ *
+ * @param name 租约名称，无特殊要求，相同名称会竞争租约
+ * @param leaseTermSecond 租期，在租期内可以做业务处理，单位是秒
+ * @param callback 当第一获取租约时，回调此函数
+ */
+ void startLeaseTask(final String name, int leaseTermSecond, ILeaseGetCallback callback);
+
+ /**
+ * 申请锁,无论成功与否，立刻返回。如果不释放，最大锁定时间是5分钟
+ *
+ * @param name 业务名称
+ * @param lockerName 锁名称
+ * @return 是否枷锁成功
+ */
+ boolean lock(String name, String lockerName);
+
+ /**
+ * 申请锁,无论成功与否，立刻返回。默认锁定时间是5分钟
+ *
+ * @param name 业务名称
+ * @param lockerName 锁名称
+ * @param lockTimeSecond 如果不释放，锁定的最大时间，单位是秒
+ * @return 是否枷锁成功
+ * @return
+ */
+ boolean lock(String name, String lockerName, int lockTimeSecond);
+
+ /**
+ * 申请锁，如果没有则等待，等待时间可以指定，如果是－1 则无限等待。如果不释放，最大锁定时间是5分钟
+ *
+ * @param name 业务名称
+ * @param lockerName 锁名称
+ * @param waitTime 没获取锁时，最大等待多长时间，如果是－1 则无限等待
+ * @return 是否枷锁成功
+ */
+ boolean tryLocker(String name, String lockerName, long waitTime);
+
+ /**
+ * 申请锁，如果没有则等待，等待时间可以指定，如果是－1 则无限等待。如果不释放，最大锁定时间是lockTimeSecond
+ *
+ * @param name 业务名称
+ * @param lockerName 锁名称
+ * @param waitTime 没获取锁时，最大等待多长时间，如果是－1 则无限等待
+ * @param lockTimeSecond 如果不释放，锁定的最大时间，单位是秒
+ * @return 是否枷锁成功
+ */
+ boolean tryLocker(String name, String lockerName, long waitTime, int lockTimeSecond);
+
+ /**
+ * 释放锁
+ *
+ * @param name
+ * @param lockerName
+ * @return
+ */
+ boolean unlock(String name, String lockerName);
+
+ /**
+ * 对于已经获取锁的，可以通过这个方法，一直持有锁。 和租约的区别是，当释放锁后，无其他实例抢占。无法实现主备模式
+ *
+ * @param name 业务名称
+ * @param lockerName 锁名称
+ * @param lockTimeSecond 租期，这个方法会自动续约，如果不主动释放，会一直持有锁
+ * @return 是否成功获取锁
+ */
+ boolean holdLock(String name, String lockerName, int lockTimeSecond);
+
+ /**
+ * 是否持有锁，不会申请锁。如果以前申请过，且未过期，返回true，否则返回false
+ *
+ * @param name 业务名称
+ * @param lockerName 锁名称
+ * @return
+ */
+ boolean hasHoldLock(String name, String lockerName);
+
+ List<LeaseInfo> queryLockedInstanceByNamePrefix(String name, String lockerNamePrefix);
+
+}
diff --git a/rocketmq-streams-lease/src/main/java/org/apache/rocketmq/streams/lease/service/ILeaseStorage.java b/rocketmq-streams-lease/src/main/java/org/apache/rocketmq/streams/lease/service/ILeaseStorage.java
new file mode 100644
index 0000000..9278bc0
--- /dev/null
+++ b/rocketmq-streams-lease/src/main/java/org/apache/rocketmq/streams/lease/service/ILeaseStorage.java
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.lease.service;
+
+import java.util.List;
+import java.util.Properties;
+
+import org.apache.rocketmq.streams.lease.model.LeaseInfo;
+
+public interface ILeaseStorage {
+
+ /**
+ * 更新lease info，需要是原子操作，存储保障多线程操作的原子性
+ *
+ * @param leaseInfo 租约表数据
+ * @return
+ */
+ boolean updateLeaseInfo(LeaseInfo leaseInfo);
+
+ /**
+ * 统计这个租约名称下，LeaseInfo对象个数
+ *
+ * @param leaseName 租约名称，无特殊要求，相同名称会竞争租约
+ * @return
+ */
+ Integer countLeaseInfo(String leaseName);
+
+ /**
+ * 查询无效的的租约
+ *
+ * @param leaseName 租约名称，无特殊要求，相同名称会竞争租约
+ * @return
+ */
+ LeaseInfo queryInValidateLease(String leaseName);
+
+ /**
+ * 查询有效的的租约
+ *
+ * @param leaseName 租约名称，无特殊要求，相同名称会竞争租约
+ * @return
+ */
+ LeaseInfo queryValidateLease(String leaseName);
+
+ /**
+ * 按前缀查询有效的租约信息
+ *
+ * @param namePrefix
+ * @return
+ */
+ List<LeaseInfo> queryValidateLeaseByNamePrefix(String namePrefix);
+
+ /**
+ * 增加租约
+ *
+ * @param leaseInfo 租约名称，无特殊要求，相同名称会竞争租约
+ */
+ void addLeaseInfo(LeaseInfo leaseInfo);
+
+}
diff --git a/rocketmq-streams-lease/src/main/java/org/apache/rocketmq/streams/lease/service/ILeaseStorasge.java b/rocketmq-streams-lease/src/main/java/org/apache/rocketmq/streams/lease/service/ILeaseStorasge.java
new file mode 100644
index 0000000..cbfe26e
--- /dev/null
+++ b/rocketmq-streams-lease/src/main/java/org/apache/rocketmq/streams/lease/service/ILeaseStorasge.java
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.streams.lease.service;
+
+import org.apache.rocketmq.streams.lease.model.LeaseInfo;
+
+public interface ILeaseStorasge {
+
+
+
+ /**
+ * 更新lease info，需要是原子操作，存储保障多线程操作的原子性
+ * @param leaseInfo 租约表数据
+ * @return
+ */
+ boolean updateLeaseInfo(LeaseInfo leaseInfo);
+
+ /**
+ * 统计这个租约名称下，LeaseInfo对象个数
+ * @param leaseName 租约名称，无特殊要求，相同名称会竞争租约
+ * @return
+ */
+ Integer countLeaseInfo(String leaseName);
+
+ /**
+ * 查询无效的的租约
+ * @param leaseName 租约名称，无特殊要求，相同名称会竞争租约
+ * @return
+ */
+ LeaseInfo queryInValidateLease(String leaseName);
+
+
+ /**
+ * 查询无效的的租约
+ * @param leaseName 租约名称，无特殊要求，相同名称会竞争租约
+ * @return
+ */
+ LeaseInfo queryValidateLease(String leaseName);
+
+ /**
+ * 增加租约
+ * @param leaseInfo 租约名称，无特殊要求，相同名称会竞争租约
+ */
+ void addLeaseInfo(LeaseInfo leaseInfo);
+
+
+
+}
diff --git a/rocketmq-streams-lease/src/main/java/org/apache/rocketmq/streams/lease/service/impl/BasedLesaseImpl.java b/rocketmq-streams-lease/src/main/java/org/apache/rocketmq/streams/lease/service/impl/BasedLesaseImpl.java
new file mode 100644
index 0000000..21db98d
--- /dev/null
+++ b/rocketmq-streams-lease/src/main/java/org/apache/rocketmq/streams/lease/service/impl/BasedLesaseImpl.java
@@ -0,0 +1,404 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.lease.service.impl;
+
+import java.text.SimpleDateFormat;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.rocketmq.streams.common.utils.DateUtil;
+import org.apache.rocketmq.streams.common.utils.IPUtil;
+import org.apache.rocketmq.streams.common.utils.RuntimeUtil;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.rocketmq.streams.lease.model.LeaseInfo;
+import org.apache.rocketmq.streams.lease.service.ILeaseGetCallback;
+import org.apache.rocketmq.streams.lease.service.ILeaseService;
+import org.apache.rocketmq.streams.lease.service.ILeaseStorage;
+
+public abstract class BasedLesaseImpl implements ILeaseService {
+ private static final Log LOG = LogFactory.getLog(BasedLesaseImpl.class);
+
+ private static final String CONSISTENT_HASH_PREFIX = "consistent_hash_";
+ private static AtomicBoolean syncStart = new AtomicBoolean(false);
+ private static final int synTime = 120; // 5分钟的一致性hash同步时间太久了，改为2分钟
+ protected ScheduledExecutorService taskExecutor = null;
+ protected int leaseTerm = 300 * 2; // 租约时间
+
+ // protected transient JDBCDriver jdbcDataSource = null;
+ protected ILeaseStorage leaseStorage;
+ protected volatile Map<String, Date> leaseName2Date = new ConcurrentHashMap<>(); // 每个lease name对应的租约到期时间
+
+ public BasedLesaseImpl() {
+
+ taskExecutor = new ScheduledThreadPoolExecutor(10);
+
+ }
+
+ /**
+ * lease_name: consistent_hash_ip, lease_user_ip: ip,定时刷新lease_info表，检查一致性hash环的节点情况
+ *
+ * @param name
+ * @return
+ */
+ @Override
+ public boolean hasLease(String name) {
+ // 内存中没有租约信息则表示 没有租约
+ Date leaseEndTime = leaseName2Date.get(name);
+ if (leaseEndTime == null) {
+ // LOG.info("内存中根据 " + name + "没有查询到租约信息，表示没有租约");
+ return false;
+ }
+ // LOG.info("查询是否有租约 name:" + name + " ,当前时间：" + new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").format(new Date())
+ // + " 租约到期时间 " + new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").format(leaseEndTime));
+ // 有租约时间，并且租约时间大于当前时间，表示有租约信息
+ if (new Date().before(leaseEndTime)) {
+ return true;
+ }
+
+ return false;
+ }
+
+ private final Map<String, AtomicBoolean> startLeaseMap = new HashMap<>();
+
+ @Override
+ public void startLeaseTask(final String name) {
+ startLeaseTask(name, this.leaseTerm, null);
+ }
+
+ @Override
+ public void startLeaseTask(final String name, ILeaseGetCallback callback) {
+ startLeaseTask(name, this.leaseTerm, callback);
+ }
+
+ @Override
+ public void startLeaseTask(final String name, int leaseTerm, ILeaseGetCallback callback) {
+ ApplyTask applyTask = new ApplyTask(leaseTerm, name, callback);
+ startLeaseTask(name, applyTask, leaseTerm / 2, true);
+ }
+
+ /**
+ * 启动定时器，定时执行任务，确保任务可重入
+ *
+ * @param name
+ * @param runnable 具体任务
+ * @param scheduleTime 调度时间
+ * @param startNow 是否立刻启动一次
+ */
+ protected void startLeaseTask(final String name, Runnable runnable, int scheduleTime, boolean startNow) {
+ AtomicBoolean isStartLease = startLeaseMap.get(name);//多次调用，只启动一次定时任务
+ if (isStartLease == null) {
+ synchronized (this) {
+ isStartLease = startLeaseMap.get(name);
+ if (isStartLease == null) {
+ isStartLease = new AtomicBoolean(false);
+ startLeaseMap.put(name, isStartLease);
+ }
+ }
+ }
+ if (isStartLease.compareAndSet(false, true)) {
+ if (startNow) {
+ runnable.run();
+ }
+ taskExecutor.scheduleWithFixedDelay(runnable, 0, scheduleTime, TimeUnit.SECONDS);
+ }
+ }
+
+ /**
+ * 续约任务
+ */
+ protected class ApplyTask implements Runnable {
+
+ protected String name;
+ protected int leaseTerm;
+ protected ILeaseGetCallback callback;
+
+ public ApplyTask(int leaseTerm, String name) {
+ this(leaseTerm, name, null);
+ }
+
+ public ApplyTask(int leaseTerm, String name, ILeaseGetCallback callback) {
+ this.name = name;
+ this.leaseTerm = leaseTerm;
+ this.callback = callback;
+ }
+
+ @Override
+ public void run() {
+ try {
+ // LOG.info("LeaseServiceImpl name: " + name + "开始获取租约...");
+ AtomicBoolean newApplyLease = new AtomicBoolean(false);
+ Date leaseDate = applyLeaseTask(leaseTerm, name, newApplyLease);
+ if (leaseDate != null) {
+ leaseName2Date.put(name, leaseDate);
+ LOG.info("LeaseServiceImpl, name: " + name + " " + getSelfUser() + " 获取租约成功, 租约到期时间为 "
+ + new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").format(leaseDate));
+ } else {
+ // fix.2020.08.13 这时name对应的租约可能还在有效期内,或者本机还持有租约，需要remove
+ // leaseName2Date.remove(name);
+ LOG.info("LeaseServiceImpl name: " + name + " " + getSelfUser() + " 获取租约失败 ");
+ }
+ if (newApplyLease.get() && callback != null) {
+ callback.callback(leaseDate);
+ }
+ } catch (Exception e) {
+ LOG.error(" LeaseServiceImpl name: " + name + " " + getSelfUser() + " 获取租约出现异常 ", e);
+ }
+
+ }
+ }
+
+ /**
+ * 申请租约，如果当期租约有效，直接更新一个租约周期，如果当前租约无效，先查询是否有有效的租约，如果有申请失败，否则直接申请租约
+ */
+ protected Date applyLeaseTask(int leaseTerm, String name, AtomicBoolean newApplyLease) {
+
+ // 计算下一次租约时间 = 当前时间 + 租约时长
+ Date nextLeaseDate = DateUtil.addSecond(new Date(), leaseTerm);
+
+ // 1 如果已经有租约，则更新租约时间(内存和数据库)即可
+ if (hasLease(name)) {
+ // LOG.info("用户已有租约，更新数据库和内存中的租约信息");
+ // 更新数据库
+ LeaseInfo leaseInfo = queryValidateLease(name);
+ if (leaseInfo == null) {
+ LOG.error("LeaseServiceImpl applyLeaseTask leaseInfo is null");
+ return null;
+ }
+ // fix.2020.08.13,与本机ip相等且满足一致性hash分配策略，才续约，其他情况为null
+ String leaseUserIp = leaseInfo.getLeaseUserIp();
+ if (!leaseUserIp.equals(getSelfUser())) {
+ return null;
+ }
+ leaseInfo.setLeaseEndDate(nextLeaseDate);
+ updateLeaseInfo(leaseInfo);
+ return nextLeaseDate;
+ }
+
+ // 2 没有租约情况 判断是否可以获取租约，只要租约没有被其他人获取，则说明有有效租约
+ boolean success = canGetLease(name);
+ if (!success) { // 表示被其他机器获取到了有效的租约
+ // LOG.info("其他机器获取到了有效的租约");
+ return null;
+ }
+
+ // 3 没有租约而且可以获取租约的情况，则尝试使用数据库原子更新的方式获取租约，保证只有一台机器成功获取租约，而且可以运行
+ boolean flag = tryGetLease(name, nextLeaseDate);
+ if (flag) { // 获取租约成功
+ newApplyLease.set(true);
+ return nextLeaseDate;
+ }
+ return null;
+
+ }
+
+ /**
+ * 查询数据库，自己是否在租期内或没有被其他人租用
+ *
+ * @return
+ */
+ protected boolean canGetLease(String name) {
+ LeaseInfo leaseInfo = queryValidateLease(name);
+ if (leaseInfo == null) {
+ return true;
+ }
+ // fix.2020.08.13,租约ip为本机ip，且与一致性hash分配ip一致，才是有效租约
+ String leaseUserIp = leaseInfo.getLeaseUserIp();
+ if (leaseUserIp.equals(getSelfUser())) {
+ return true;
+ }
+ return false;
+ }
+
+ /**
+ * 更新数据库，占用租期并更新租期时间
+ *
+ * @param time
+ */
+ protected boolean tryGetLease(String name, Date time) {
+ // LOG.info("尝试获取租约 lease name is : " + name + " 下次到期时间: "
+ // + new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").format(time));
+ LeaseInfo validateLeaseInfo = queryValidateLease(name);
+
+ if (validateLeaseInfo == null) {// 这里有两种情况 1 数据库里面没有租约信息 2 数据库里面有租约信息但是已经过期
+ Integer count = countLeaseInfo(name);
+ if (count == null || count == 0) {// 表示现在数据库里面没有任何租约信息，插入租约成功则表示获取成功，失败表示在这一时刻其他机器获取了租约
+ // LOG.info("数据库中暂时没有租约信息，尝试原子插入租约:" + name);
+ // fix.2020.08.13,经过一致性hash计算，该名字的任务不应该在本机执行，直接返回，无需插入。只有分配到hash执行权限的机器才可以插入并获取租约
+ if (!getSelfUser().equals(getConsistentHashHost(name))) {
+ return false;
+ }
+ validateLeaseInfo = new LeaseInfo();
+ validateLeaseInfo.setLeaseName(name);
+ validateLeaseInfo.setLeaseUserIp(getSelfUser());
+ validateLeaseInfo.setLeaseEndDate(time);
+ validateLeaseInfo.setStatus(1);
+ validateLeaseInfo.setVersion(1);
+ if (insert(validateLeaseInfo)) {
+ LOG.info("数据库中暂时没有租约信息，原子插入成功，获取租约成功:" + name);
+ return true;
+ } else {
+ LOG.info("数据库中暂时没有租约信息，原子插入失败，已经被其他机器获取租约:" + name);
+ return false;
+ }
+ } else { // 表示数据库里面有一条但是无效，这里需要两台机器按照version进行原子更新，更新成功的获取租约
+ // LOG.info("数据库中有一条无效的租约信息，尝试根据版本号去原子更新租约信息:" + name);
+ LeaseInfo inValidateLeaseInfo = queryInValidateLease(name);
+ if (inValidateLeaseInfo == null) {// 说明这个时候另外一台机器获取成功了
+ LOG.info("另外一台机器获取成功了租约:" + name);
+ return false;
+ }
+ // fix.2020.08.13,机器重启之后，该名字的任务已经不分配在此机器上执行，直接返回，无需更新数据库
+ if (!getSelfUser().equals(getConsistentHashHost(name))) {
+ return false;
+ }
+ inValidateLeaseInfo.setLeaseName(name);
+ inValidateLeaseInfo.setLeaseUserIp(getSelfUser());
+ inValidateLeaseInfo.setLeaseEndDate(time);
+ inValidateLeaseInfo.setStatus(1);
+ boolean success = updateDBLeaseInfo(inValidateLeaseInfo);
+ if (success) {
+ LOG.info("LeaseServiceImpl 原子更新租约成功，当前机器获取到了租约信息:" + name);
+ } else {
+ LOG.info("LeaseServiceImpl 原子更新租约失败，租约被其他机器获取:" + name);
+ }
+ return success;
+ }
+
+ } else { // 判断是否是自己获取了租约，如果是自己获取了租约则更新时间（内存和数据库），
+ // 这里是为了解决机器重启的情况，机器重启，内存中没有租约信息，但是实际上该用户是有租约权限的
+ // fix.2020.08.13,租约的ip与本机ip相等，且满足一致性hash策略，才会被本机执行
+ String leaseUserIp = validateLeaseInfo.getLeaseUserIp();
+ if (leaseUserIp.equals(getSelfUser())) {
+ // 如果当期用户有租约信息，则更新数据库
+ validateLeaseInfo.setLeaseEndDate(time);
+ boolean hasUpdate = updateLeaseInfo(validateLeaseInfo);
+ if (hasUpdate) {
+ LOG.info(
+ "LeaseServiceImpl机器重启情况，当前用户有租约信息，并且更新数据库成功，租约信息为 name :" + validateLeaseInfo.getLeaseName()
+ + " ip : " + validateLeaseInfo.getLeaseUserIp() + " 到期时间 : " + new SimpleDateFormat(
+ "yyyy-MM-dd HH:mm:ss").format(validateLeaseInfo.getLeaseEndDate()));
+ return true;
+ } else {
+ LOG.info("LeaseServiceImpl 机器重启情况，当前用户有租约信息，并且更新数据库失败，表示失去租约:" + name);
+ return false;
+ }
+ }
+ // LOG.info("LeaseServiceImpl 租约被其他机器获取，租约信息为 name :" + validateLeaseInfo.getLeaseName() + " ip : "
+ // + validateLeaseInfo.getLeaseUserIp() + " 到期时间 : "
+ // + new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").format(validateLeaseInfo.getLeaseEndDate()));
+ return false;
+ }
+
+ }
+
+ protected LeaseInfo queryValidateLease(String name) {
+ //String sql = "SELECT * FROM lease_info WHERE lease_name ='" + name + "' and status=1 and lease_end_time>now()";
+ //// LOG.info("LeaseServiceImpl query validate lease sql:" + sql);
+ //return queryLease(name, sql);
+ return leaseStorage.queryValidateLease(name);
+ }
+
+ protected List<LeaseInfo> queryValidateLeaseByNamePrefix(String namePrefix) {
+ return leaseStorage.queryValidateLeaseByNamePrefix(namePrefix);
+ }
+
+ /**
+ * 如果发生唯一索引冲突返回失败
+ *
+ * @param leaseInfo
+ * @return
+ */
+ private boolean insert(LeaseInfo leaseInfo) {
+ try {
+ addLeaseInfo(leaseInfo);
+ return true;
+ } catch (Exception e) {
+ LOG.error("LeaseServiceImpl insert error", e);
+ return false;
+ }
+ }
+
+ /**
+ * 更新时需要加version＝当前version，如果更新数据条数为0，返回false
+ *
+ * @param leaseInfo
+ * @return
+ */
+ protected boolean updateDBLeaseInfo(LeaseInfo leaseInfo) {
+ return updateLeaseInfo(leaseInfo);
+ }
+
+ protected boolean updateLeaseInfo(LeaseInfo leaseInfo) {
+
+ return leaseStorage.updateLeaseInfo(leaseInfo);
+ }
+
+ protected Integer countLeaseInfo(String name) {
+
+ return leaseStorage.countLeaseInfo(name);
+ }
+
+ protected LeaseInfo queryInValidateLease(String name) {
+
+ return leaseStorage.queryInValidateLease(name);
+ }
+
+ protected void addLeaseInfo(LeaseInfo leaseInfo) {
+
+ leaseStorage.addLeaseInfo(leaseInfo);
+
+ }
+
+ /**
+ * 本地ip地址作为自己的唯一标识
+ *
+ * @return
+ */
+ public static String getLocalName() {
+ return IPUtil.getLocalIdentification() + ":" + Optional.ofNullable(RuntimeUtil.getPid()).orElse("UNKNOWN");
+ }
+
+ /**
+ * 本地ip地址作为自己的唯一标识
+ *
+ * @return
+ */
+ public String getSelfUser() {
+ return getLocalName();
+ }
+
+ private String getConsistentHashHost(String name) {
+ //if (StringUtil.isEmpty(leaseConsistentHashSuffix)) {
+ // return getSelfUser();
+ //}
+ //return consistentHashInstance.getCandidateNode(name);
+ return getSelfUser();
+ }
+
+ public void setLeaseStorage(ILeaseStorage leaseStorage) {
+ this.leaseStorage = leaseStorage;
+ }
+}
diff --git a/rocketmq-streams-lease/src/main/java/org/apache/rocketmq/streams/lease/service/impl/LeaseServiceImpl.java b/rocketmq-streams-lease/src/main/java/org/apache/rocketmq/streams/lease/service/impl/LeaseServiceImpl.java
new file mode 100644
index 0000000..860710a
--- /dev/null
+++ b/rocketmq-streams-lease/src/main/java/org/apache/rocketmq/streams/lease/service/impl/LeaseServiceImpl.java
@@ -0,0 +1,275 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.lease.service.impl;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.rocketmq.streams.common.utils.DateUtil;
+import org.apache.rocketmq.streams.common.utils.MapKeyUtil;
+import org.apache.rocketmq.streams.lease.model.LeaseInfo;
+import org.apache.rocketmq.streams.lease.service.ILeaseService;
+
+import java.text.SimpleDateFormat;
+import java.util.Date;
+import java.util.List;
+import java.util.concurrent.*;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+public class LeaseServiceImpl extends BasedLesaseImpl {
+
+ private static final Log LOG = LogFactory.getLog(LeaseServiceImpl.class);
+
+ private transient ConcurrentHashMap<String, HoldLockTask> holdLockTasks = new ConcurrentHashMap();
+
+ protected ConcurrentHashMap<String, HoldLockFunture> seizeLockingFuntures = new ConcurrentHashMap<>();
+ //如果是抢占锁状态中，则不允许申请锁
+
+ public LeaseServiceImpl() {
+ super();
+ }
+
+ /**
+ * 尝试获取锁，可以等待waitTime，如果到点未返回，则直接返回。如果是-1，则一直等待
+ *
+ * @param name 业务名称
+ * @param lockerName 锁名称
+ * @param waitTime 等待时间，是微秒单位
+ * @return
+ */
+ @Override
+ public boolean tryLocker(String name, String lockerName, long waitTime) {
+ return tryLocker(name, lockerName, waitTime, ILeaseService.DEFALUT_LOCK_TIME);
+ }
+
+ @Override
+ public boolean tryLocker(String name, String lockerName, long waitTime, int lockTimeSecond) {
+ long now = System.currentTimeMillis();
+ boolean success = lock(name, lockerName, lockTimeSecond);
+ while (!success) {
+ if (waitTime > -1 && (System.currentTimeMillis() - now > waitTime)) {
+ break;
+ }
+ success = lock(name, lockerName, lockTimeSecond);
+ if (success) {
+ return success;
+ }
+ try {
+ Thread.sleep(100);
+ } catch (InterruptedException e) {
+ LOG.error("LeaseServiceImpl try locker error", e);
+ }
+ }
+ return success;
+
+ }
+
+ @Override
+ public boolean lock(String name, String lockerName) {
+ return lock(name, lockerName, ILeaseService.DEFALUT_LOCK_TIME);
+ }
+
+ @Override
+ public boolean lock(String name, String lockerName, int leaseSecond) {
+ lockerName = createLockName(name, lockerName);
+ Future future = seizeLockingFuntures.get(lockerName);
+ if (future != null && ((HoldLockFunture)future).isDone == false) {
+ return false;
+ }
+ Date nextLeaseDate =
+ DateUtil.addSecond(new Date(), leaseSecond);// 默认锁定5分钟，用完需要立刻释放.如果时间不同步，可能导致锁失败
+ boolean success = tryGetLease(lockerName, nextLeaseDate);// 申请锁，锁的时间是leaseTerm
+ return success;
+ }
+
+ @Override
+ public boolean unlock(String name, String lockerName) {
+ // LOG.info("LeaseServiceImpl unlock,name:" + name);
+ lockerName = createLockName(name, lockerName);
+ LeaseInfo validateLeaseInfo = queryValidateLease(lockerName);
+ if (validateLeaseInfo == null) {
+ LOG.warn("LeaseServiceImpl unlock,validateLeaseInfo is null,lockerName:" + lockerName);
+ }
+ if (validateLeaseInfo != null && validateLeaseInfo.getLeaseUserIp().equals(getSelfUser())) {
+ validateLeaseInfo.setStatus(0);
+ updateDBLeaseInfo(validateLeaseInfo);
+ }
+ HoldLockTask holdLockTask = holdLockTasks.remove(lockerName);
+ if (holdLockTask != null) {
+ holdLockTask.close();
+ }
+ leaseName2Date.remove(lockerName);
+ return false;
+ }
+
+ /**
+ * 如果有锁，则一直持有，如果不能获取，则结束。和租约不同，租约是没有也会尝试重试，一备对方挂机，自己可以接手工作
+ *
+ * @param name
+ * @param secondeName
+ * @param lockTimeSecond 获取锁的时间
+ * @return
+ */
+ @Override
+ public boolean holdLock(String name, String secondeName, int lockTimeSecond) {
+ if (hasHoldLock(name, secondeName)) {
+ return true;
+ }
+ synchronized (this) {
+ if (hasHoldLock(name, secondeName)) {
+ return true;
+ }
+ String lockerName = createLockName(name, secondeName);
+ Date nextLeaseDate =
+ DateUtil.addSecond(new Date(), lockTimeSecond);
+ boolean success = tryGetLease(lockerName, nextLeaseDate);// 申请锁，锁的时间是leaseTerm
+ if (success == false) {
+ return false;
+ }
+ leaseName2Date.put(lockerName, nextLeaseDate);
+
+ if (!holdLockTasks.containsKey(lockerName)) {
+ HoldLockTask holdLockTask = new HoldLockTask(lockTimeSecond, lockerName, this);
+ holdLockTask.start();
+ holdLockTasks.putIfAbsent(lockerName, holdLockTask);
+ }
+ }
+
+ return true;
+ }
+
+ /**
+ * 是否持有锁，不访问数据库，直接看本地
+ *
+ * @param name
+ * @param secondeName
+ * @return
+ */
+ @Override
+ public boolean hasHoldLock(String name, String secondeName) {
+ String lockerName = createLockName(name, secondeName);
+ return hasLease(lockerName);
+ }
+
+ @Override
+ public List<LeaseInfo> queryLockedInstanceByNamePrefix(String name, String lockerNamePrefix) {
+ String leaseNamePrefix = MapKeyUtil.createKey(name, lockerNamePrefix);
+ return queryValidateLeaseByNamePrefix(leaseNamePrefix);
+ }
+
+ private String createLockName(String name, String lockerName) {
+ return MapKeyUtil.createKey(name, lockerName);
+ }
+
+ private class HoldLockTask extends ApplyTask {
+ protected volatile boolean iscontinue = true;
+ protected LeaseServiceImpl leaseService;
+ protected ScheduledExecutorService scheduledExecutor;
+
+ public HoldLockTask(int leaseTerm, String name, LeaseServiceImpl leaseService) {
+ super(leaseTerm, name);
+ this.leaseService = leaseService;
+ scheduledExecutor = new ScheduledThreadPoolExecutor(1);
+
+ }
+
+ public void start() {
+ scheduledExecutor.scheduleWithFixedDelay(this, leaseTerm / 2, leaseTerm / 2, TimeUnit.SECONDS);
+ }
+
+ public void close() {
+ iscontinue = false;
+ if (scheduledExecutor != null) {
+ scheduledExecutor.shutdown();
+ }
+ }
+
+ public boolean isIscontinue() {
+ return iscontinue;
+ }
+
+ @Override
+ public void run() {
+ try {
+ if (!iscontinue) {
+ return;
+ }
+ Date leaseDate = applyLeaseTask(leaseTerm, name, new AtomicBoolean(false));
+ if (leaseDate != null) {
+ leaseName2Date.put(name, leaseDate);
+ LOG.debug("LeaseServiceImpl, name: " + name + " " + getSelfUser() + " 续约锁成功, 租约到期时间为 "
+ + new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").format(leaseDate));
+ } else {
+ iscontinue = false;
+ synchronized (leaseService) {
+ holdLockTasks.remove(name);
+ }
+ LOG.info("LeaseServiceImpl name: " + name + " " + getSelfUser() + " 续约锁失败，续锁程序会停止");
+ }
+ } catch (Exception e) {
+ iscontinue = false;
+ LOG.error(" LeaseServiceImpl name: " + name + " " + getSelfUser() + " 续约锁出现异常，续锁程序会停止", e);
+ }
+
+ }
+
+ }
+
+ /**
+ * 抢占锁的future，必须等锁超时才能继续获取锁
+ */
+ protected class HoldLockFunture implements Future<Boolean> {
+ private volatile boolean isDone = false;
+ private volatile Date date = null;
+
+ @Override
+ public boolean cancel(boolean mayInterruptIfRunning) {
+ throw new RuntimeException("can not cancel");
+ }
+
+ @Override
+ public boolean isCancelled() {
+ return false;
+ }
+
+ @Override
+ public boolean isDone() {
+ if (date != null && System.currentTimeMillis() - date.getTime() >= 0) {
+ isDone = true;
+ return isDone;
+ }
+ return false;
+ }
+
+ @Override
+ public Boolean get() throws InterruptedException, ExecutionException {
+ while (isDone() == false) {
+ Thread.sleep(1000);
+ }
+ return true;
+ }
+
+ private long startTime = System.currentTimeMillis();
+
+ @Override
+ public Boolean get(long timeout, TimeUnit unit)
+ throws InterruptedException, ExecutionException, TimeoutException {
+
+ throw new RuntimeException("can not support timeout ");
+ }
+
+ }
+}
diff --git a/rocketmq-streams-lease/src/main/java/org/apache/rocketmq/streams/lease/service/impl/MockLeaseImpl.java b/rocketmq-streams-lease/src/main/java/org/apache/rocketmq/streams/lease/service/impl/MockLeaseImpl.java
new file mode 100644
index 0000000..d952581
--- /dev/null
+++ b/rocketmq-streams-lease/src/main/java/org/apache/rocketmq/streams/lease/service/impl/MockLeaseImpl.java
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.lease.service.impl;
+
+import java.util.Date;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+import org.apache.rocketmq.streams.common.utils.DateUtil;
+import org.apache.rocketmq.streams.lease.model.LeaseInfo;
+import org.apache.rocketmq.streams.lease.service.ILeaseGetCallback;
+import org.apache.rocketmq.streams.lease.service.ILeaseService;
+
+/**
+ * 在内存和文件模式下使用，所有的申请都会返回true，主要用来做业务测试
+ */
+public class MockLeaseImpl implements ILeaseService {
+ @Override
+ public boolean hasLease(String name) {
+ return true;
+ }
+
+ @Override
+ public void startLeaseTask(String name) {
+
+ }
+
+ @Override
+ public void startLeaseTask(String name, ILeaseGetCallback callback) {
+ callback.callback(DateUtil.addMinute(new Date(), 1));
+ }
+
+ @Override
+ public void startLeaseTask(String name, int leaseTerm, ILeaseGetCallback callback) {
+
+ }
+
+ @Override
+ public boolean lock(String name, String lockerName) {
+ return true;
+ }
+
+ @Override
+ public boolean lock(String name, String lockerName, int lockTimeSecond) {
+ return true;
+ }
+
+ @Override
+ public boolean tryLocker(String name, String lockerName, long waitTime) {
+ return true;
+ }
+
+ @Override
+ public boolean tryLocker(String name, String lockerName, long waitTime, int lockTimeSecond) {
+ return true;
+ }
+
+ @Override
+ public boolean unlock(String name, String lockerName) {
+ return true;
+ }
+
+ @Override
+ public boolean holdLock(String name, String lockerName, int lockTimeSecond) {
+ return true;
+ }
+
+ @Override
+ public boolean hasHoldLock(String name, String lockerName) {
+ return true;
+ }
+
+ @Override
+ public List<LeaseInfo> queryLockedInstanceByNamePrefix(String name, String lockerNamePrefix) {
+ return null;
+ }
+
+}
diff --git a/rocketmq-streams-lease/src/main/java/org/apache/rocketmq/streams/lease/service/storages/DBLeaseStorage.java b/rocketmq-streams-lease/src/main/java/org/apache/rocketmq/streams/lease/service/storages/DBLeaseStorage.java
new file mode 100644
index 0000000..b99132d
--- /dev/null
+++ b/rocketmq-streams-lease/src/main/java/org/apache/rocketmq/streams/lease/service/storages/DBLeaseStorage.java
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.lease.service.storages;
+
+import java.math.BigInteger;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.rocketmq.streams.common.utils.DateUtil;
+import org.apache.rocketmq.streams.common.utils.SQLUtil;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+import org.apache.rocketmq.streams.db.driver.JDBCDriver;
+import org.apache.rocketmq.streams.db.driver.DriverBuilder;
+import org.apache.rocketmq.streams.lease.model.LeaseInfo;
+import org.apache.rocketmq.streams.lease.service.ILeaseStorage;
+
+public class DBLeaseStorage implements ILeaseStorage {
+ private static final Log LOG = LogFactory.getLog(DBLeaseStorage.class);
+ protected JDBCDriver jdbcDataSource;
+ private String url;
+ protected String userName;
+ protected String password;
+ protected String jdbc;
+
+ public DBLeaseStorage(String jdbc, String url, String userName, String password) {
+ this.jdbc = jdbc;
+ this.url = url;
+ this.userName = userName;
+ this.password = password;
+ jdbcDataSource = DriverBuilder.createDriver(jdbc, url, userName, password);
+ }
+
+ @Override
+ public boolean updateLeaseInfo(LeaseInfo leaseInfo) {
+ String sql = "UPDATE lease_info SET version=version+1,status=#{status},gmt_modified=now()";
+ String whereSQL = " WHERE id=#{id} and version=#{version}";
+
+ if (StringUtil.isNotEmpty(leaseInfo.getLeaseName())) {
+ sql += ",lease_name=#{leaseName}";
+ }
+ if (StringUtil.isNotEmpty(leaseInfo.getLeaseUserIp())) {
+ sql += ",lease_user_ip=#{leaseUserIp}";
+ }
+ if (leaseInfo.getLeaseEndDate() != null) {
+ sql += ",lease_end_time=#{leaseEndDate}";
+ }
+ sql += whereSQL;
+ sql = SQLUtil.parseIbatisSQL(leaseInfo, sql);
+ try {
+ int count = getOrCreateJDBCDataSource().update(sql);
+ boolean success = count > 0;
+ if (success) {
+ synchronized (this) {
+ leaseInfo.setVersion(leaseInfo.getVersion() + 1);
+ }
+ } else {
+ System.out.println(count);
+ }
+ return success;
+ } catch (Exception e) {
+ LOG.error("LeaseServiceImpl updateLeaseInfo excuteUpdate error", e);
+ throw new RuntimeException("execute sql error " + sql, e);
+ }
+ }
+
+ @Override
+ public Integer countLeaseInfo(String leaseName) {
+ String sql = "SELECT count(*) as c FROM lease_info WHERE lease_name = '" + leaseName + "' and status = 1";
+ try {
+
+ List<Map<String, Object>> rows = getOrCreateJDBCDataSource().queryForList(sql);
+ if (rows == null || rows.size() == 0) {
+ return null;
+ }
+ Long value = (Long)rows.get(0).get("c");
+ return value.intValue();
+ } catch (Exception e) {
+ throw new RuntimeException("execute sql error " + sql, e);
+ }
+ }
+
+ @Override
+ public LeaseInfo queryInValidateLease(String leaseName) {
+ String sql = "SELECT * FROM lease_info WHERE lease_name ='" + leaseName + "' and status=1 and lease_end_time<'" + DateUtil.getCurrentTimeString() + "'";
+ LOG.info("LeaseServiceImpl queryInValidateLease builder:" + sql);
+ return queryLease(leaseName, sql);
+ }
+
+ @Override
+ public LeaseInfo queryValidateLease(String leaseName) {
+ String sql = "SELECT * FROM lease_info WHERE lease_name ='" + leaseName + "' and status=1 and lease_end_time>now()";
+ return queryLease(leaseName, sql);
+ }
+
+ @Override
+ public List<LeaseInfo> queryValidateLeaseByNamePrefix(String namePrefix) {
+ String sql = "SELECT * FROM lease_info WHERE lease_name like '" + namePrefix + "%' and status=1 and lease_end_time>now()";
+ try {
+ List<LeaseInfo> leaseInfos = new ArrayList<>();
+ List<Map<String, Object>> rows = getOrCreateJDBCDataSource().queryForList(sql);
+ if (rows == null || rows.size() == 0) {
+ return null;
+ }
+ for (Map<String, Object> row : rows) {
+ LeaseInfo leaseInfo = convert(row);
+ leaseInfos.add(leaseInfo);
+ }
+
+ return leaseInfos;
+ } catch (Exception e) {
+ throw new RuntimeException("execute sql error " + sql, e);
+ }
+ }
+
+ @Override
+ public void addLeaseInfo(LeaseInfo leaseInfo) {
+ String sql =
+ " REPLACE INTO lease_info(lease_name,lease_user_ip,lease_end_time,status,version,gmt_create,gmt_modified)"
+ + " VALUES (#{leaseName},#{leaseUserIp},#{leaseEndDate},#{status},#{version},now(),now())";
+ sql = SQLUtil.parseIbatisSQL(leaseInfo, sql);
+ try {
+
+ getOrCreateJDBCDataSource().execute(sql);
+ } catch (Exception e) {
+ LOG.error("LeaseServiceImpl execute sql error,sql:" + sql, e);
+ throw new RuntimeException("execute sql error " + sql, e);
+ }
+ }
+
+ protected JDBCDriver getOrCreateJDBCDataSource() {
+ if (this.jdbcDataSource == null || !this.jdbcDataSource.isValidate()) {
+ synchronized (this) {
+ if (this.jdbcDataSource == null || !this.jdbcDataSource.isValidate()) {
+ this.jdbcDataSource =
+ DriverBuilder.createDriver(this.jdbc, this.url, this.userName, this.password);
+ }
+ }
+ }
+ return jdbcDataSource;
+ }
+
+ protected LeaseInfo queryLease(String name, String sql) {
+ try {
+ List<Map<String, Object>> rows = getOrCreateJDBCDataSource().queryForList(sql);
+ if (rows == null || rows.size() == 0) {
+ return null;
+ }
+ return convert(rows.get(0));
+ } catch (Exception e) {
+ throw new RuntimeException("execute sql error " + sql, e);
+ }
+ }
+
+ protected LeaseInfo convert(Map<String, Object> map) {
+ LeaseInfo leaseInfo = new LeaseInfo();
+ leaseInfo.setId(getMapLongValue("id", map));
+ leaseInfo.setCreateTime(getMapDateValue("gmt_create", map));
+ leaseInfo.setLeaseEndDate(getMapDateValue("lease_end_time", map));
+ leaseInfo.setLeaseName(getMapValue("lease_name", map, String.class));
+ leaseInfo.setLeaseUserIp(getMapValue("lease_user_ip", map, String.class));
+ Integer stauts = getMapValue("status", map, Integer.class);
+ if (stauts != null) {
+ leaseInfo.setStatus(stauts);
+ }
+ leaseInfo.setUpdateTime(getMapDateValue("gmt_modified", map));
+ Long version = getMapLongValue("version", map);
+ if (version != null) {
+ leaseInfo.setVersion(version);
+ }
+ return leaseInfo;
+ }
+
+ @SuppressWarnings("unchecked")
+ private <T> T getMapValue(String fieldName, Map<String, Object> map, Class<T> integerClass) {
+ Object value = map.get(fieldName);
+ if (value == null) {
+ return null;
+ }
+ return (T)value;
+ }
+
+ private Long getMapLongValue(String fieldName, Map<String, Object> map) {
+ Object value = map.get(fieldName);
+ if (value == null) {
+ return null;
+ }
+ if (value instanceof Long) {
+ return (Long)value;
+ }
+ if (value instanceof BigInteger) {
+ return ((BigInteger)value).longValue();
+ }
+ return null;
+ }
+
+ private Date getMapDateValue(String fieldName, Map<String, Object> map) {
+ Object value = map.get(fieldName);
+ if (value == null) {
+ return null;
+ }
+ if (value instanceof Date) {
+ return (Date)value;
+ }
+ if (value instanceof String) {
+ return DateUtil.parseTime(((String)value));
+ }
+ return null;
+
+ }
+
+}
diff --git a/rocketmq-streams-lease/src/test/java/org/apache/rocketmq/streams/lease/LeaseComponentTest.java b/rocketmq-streams-lease/src/test/java/org/apache/rocketmq/streams/lease/LeaseComponentTest.java
new file mode 100644
index 0000000..6404e9f
--- /dev/null
+++ b/rocketmq-streams-lease/src/test/java/org/apache/rocketmq/streams/lease/LeaseComponentTest.java
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.lease;
+
+import java.util.Date;
+
+import org.apache.rocketmq.streams.common.component.ComponentCreator;
+import org.apache.rocketmq.streams.common.configure.ConfigureFileKey;
+import org.apache.rocketmq.streams.db.driver.DriverBuilder;
+import org.apache.rocketmq.streams.db.driver.JDBCDriver;
+import org.apache.rocketmq.streams.lease.model.LeaseInfo;
+import org.apache.rocketmq.streams.lease.service.ILeaseGetCallback;
+import org.apache.rocketmq.streams.lease.service.ILeaseService;
+import org.junit.Test;
+
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+public class LeaseComponentTest {
+
+ private String URL = "";
+ protected String USER_NAME = "";
+ protected String PASSWORD = "";
+
+ public LeaseComponentTest() {
+
+ //正式使用时，在配置文件配置
+ ComponentCreator.getProperties().put(ConfigureFileKey.CONNECT_TYPE, "DB");
+ ComponentCreator.getProperties().put(ConfigureFileKey.JDBC_URL, URL);//数据库连接url
+ ComponentCreator.getProperties().put(ConfigureFileKey.JDBC_USERNAME, USER_NAME);//用户名
+ ComponentCreator.getProperties().put(ConfigureFileKey.JDBC_PASSWORD, PASSWORD);//password
+
+ JDBCDriver driver = DriverBuilder.createDriver();
+ driver.execute(LeaseInfo.createTableSQL());
+ }
+
+ @Test
+ public void testLease() throws InterruptedException {
+ String leaseName = "lease.test";
+ int leaseTime = 5;
+ LeaseComponent.getInstance().getService().startLeaseTask(leaseName, leaseTime, new ILeaseGetCallback() {
+ @Override
+ public void callback(Date nextLeaseDate) {
+ System.out.println("I get lease");
+ }
+ });
+ assertTrue(LeaseComponent.getInstance().getService().hasLease(leaseName));
+ Thread.sleep(5000);
+ assertTrue(LeaseComponent.getInstance().getService().hasLease(leaseName));//会一直续约
+ Thread.sleep(5000);
+ assertTrue(LeaseComponent.getInstance().getService().hasLease(leaseName));//会一直续约
+ }
+
+ @Test
+ public void testLock() throws InterruptedException {
+ String name = "dipper";
+ String lockName = "lease.test";
+ int leaseTime = 5;
+ boolean success = LeaseComponent.getInstance().getService().lock(name, lockName, leaseTime);//锁定5秒钟
+ assertTrue(success);//获取锁
+ Thread.sleep(6000);
+ assertFalse(LeaseComponent.getInstance().getService().hasHoldLock(name, lockName));//超期释放
+ }
+
+ /**
+ * holdlock是一直持有锁，和租约的区别是，当释放锁后，无其他实例抢占
+ *
+ * @throws InterruptedException
+ */
+ @Test
+ public void testHoldLock() throws InterruptedException {
+ String name = "dipper";
+ String lockName = "lease.test";
+ int leaseTime = 6;
+ boolean success = LeaseComponent.getInstance().getService().holdLock(name, lockName, leaseTime);//锁定5秒钟
+ assertTrue(success);//获取锁
+ Thread.sleep(8000);
+ assertTrue(LeaseComponent.getInstance().getService().hasHoldLock(name, lockName));//会自动续约，不会释放，可以手动释放
+ LeaseComponent.getInstance().getService().unlock(name, lockName);
+ assertFalse(LeaseComponent.getInstance().getService().hasHoldLock(name, lockName));
+ }
+
+ @Test
+ public void testHoldLockContinue() throws InterruptedException {
+ String name = "dipper";
+ String lockName = "lease.test";
+ int leaseTime = 6;
+ boolean success = holdLock(name, lockName, leaseTime);//锁定5秒钟
+ while (true) {
+ Thread.sleep(1000);
+ System.out.println(holdLock(name, lockName, leaseTime));
+ }
+ }
+
+ protected boolean holdLock(String name, String lockName, int leaseTime) {
+ ILeaseService leaseService = LeaseComponent.getInstance().getService();
+ if (leaseService.hasHoldLock(name, lockName)) {
+ return true;
+ }
+
+ boolean success = leaseService.holdLock(name, lockName, leaseTime);
+ return success;
+ }
+
+}
diff --git a/rocketmq-streams-lease/src/test/resources/log4j.xml b/rocketmq-streams-lease/src/test/resources/log4j.xml
new file mode 100755
index 0000000..7812fe7
--- /dev/null
+++ b/rocketmq-streams-lease/src/test/resources/log4j.xml
@@ -0,0 +1,20 @@
+<?xml version="1.0" encoding="UTF-8" ?>
+<!DOCTYPE log4j:configuration SYSTEM "http://toolkit.alibaba-inc.com/dtd/log4j/log4j.dtd">
+<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
+
+ <appender name="Console" class="org.apache.log4j.ConsoleAppender">
+ <layout class="org.apache.log4j.PatternLayout">
+ <param name="ConversionPattern" value="%d{ISO8601} %l [%t] %-5p - %m%n%n"/>
+ </layout>
+ <filter class="org.apache.log4j.varia.LevelRangeFilter">
+ <param name="LevelMin" value="INFO"/>
+ <param name="LevelMax" value="ERROR"/>
+ </filter>
+ </appender>
+
+ <root>
+ <priority value="INFO"/>
+ <appender-ref ref="Console"/>
+ </root>
+
+</log4j:configuration>
\ No newline at end of file
[rocketmq-streams] 12/35: modify the README.md
Posted by du...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
duhengforever pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/rocketmq-streams.git
commit 0783741de4caf531e46cbcaea8126fb99fc7ac41
Author: 刈刀 <je...@gmail.com>
AuthorDate: Mon Aug 2 12:16:19 2021 +0800
modify the README.md
---
README.md | 1 +
1 file changed, 1 insertion(+)
diff --git a/README.md b/README.md
index 649319c..5325cdd 100644
--- a/README.md
+++ b/README.md
@@ -7,6 +7,7 @@
## DataStream Example
+
```java
import org.apache.rocketmq.streams.client.transform.DataStream;
[rocketmq-streams] 15/35: modify the README.md
Posted by du...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
duhengforever pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/rocketmq-streams.git
commit a9450c971b9a4905ec52ac6c379564d929ecdda8
Author: 刈刀 <je...@gmail.com>
AuthorDate: Mon Aug 2 12:25:35 2021 +0800
modify the README.md
---
README.md | 1 +
1 file changed, 1 insertion(+)
diff --git a/README.md b/README.md
index 649319c..3b163db 100644
--- a/README.md
+++ b/README.md
@@ -2,6 +2,7 @@
## Features
+
* 轻量级部署：可以单独部署，也支持集群部署
* 多种类型的数据输入以及输出，source支持 rocketmq ， sink支持db, rocketmq 等
[rocketmq-streams] 21/35: Merge branch 'apache:main' into main
Posted by du...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
duhengforever pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/rocketmq-streams.git
commit 607418a9c9ce2fedd322fee5ead5395d1698abaf
Merge: 2564d59 f349dd9
Author: xstorm1 <xs...@live.cn>
AuthorDate: Mon Aug 2 15:40:00 2021 +0800
Merge branch 'apache:main' into main
.gitignore | 25 ++
README.md | 108 ++++-
pom.xml | 410 +++++++++++++++++
rocketmq-streams-clients/pom.xml | 46 ++
.../rocketmq/streams/client/DataStreamAction.java | 101 +++++
.../rocketmq/streams/client/StreamBuilder.java | 28 ++
.../streams/client/source/DataStreamSource.java | 76 ++++
.../client/strategy/CheckpointStrategy.java | 69 +++
.../streams/client/strategy/StateStrategy.java | 37 ++
.../rocketmq/streams/client/strategy/Strategy.java | 25 ++
.../streams/client/transform/DataStream.java | 437 ++++++++++++++++++
.../streams/client/transform/JoinStream.java | 212 +++++++++
.../streams/client/transform/SplitStream.java | 61 +++
.../streams/client/transform/WindowStream.java | 210 +++++++++
.../client/transform/window/HoppingWindow.java | 32 ++
.../client/transform/window/SessionWindow.java | 32 ++
.../streams/client/transform/window/Time.java | 45 ++
.../client/transform/window/TumblingWindow.java | 33 ++
.../client/transform/window/WindowInfo.java | 83 ++++
.../rocketmq/streams/client/DBDriverTest.java | 75 ++++
.../rocketmq/streams/client/DataStreamTest.java | 107 +++++
.../apache/rocketmq/streams/client/FilterTest.java | 49 +++
.../apache/rocketmq/streams/client/JoinTest.java | 89 ++++
.../apache/rocketmq/streams/client/LeaseTest.java | 98 +++++
.../rocketmq/streams/client/ORMUtilTest.java | 172 ++++++++
.../apache/rocketmq/streams/client/SplitTest.java | 86 ++++
.../apache/rocketmq/streams/client/UnionTest.java | 82 ++++
.../apache/rocketmq/streams/client/WindowTest.java | 86 ++++
.../client/windows/AbstractWindowFireModeTest.java | 189 ++++++++
.../streams/client/windows/WindowFromFileTest.java | 158 +++++++
.../streams/client/windows/WindowFromMetaq.java | 47 ++
.../client/windows/WindowHighAvailabilityTest.java | 131 ++++++
.../src/test/resources/log4j.xml | 36 ++
rocketmq-streams-db-operator/pom.xml | 34 ++
.../rocketmq-streams-db-operator.iml | 16 +
.../streams/db/configuable/DBConfigureService.java | 282 ++++++++++++
.../DBSupportParentConfigureService.java | 37 ++
.../rocketmq/streams/db/driver/DriverBuilder.java | 111 +++++
.../rocketmq/streams/db/driver/IDriverBudiler.java | 36 ++
.../rocketmq/streams/db/driver/JDBCDriver.java | 277 ++++++++++++
.../db/driver/batchloader/BatchRowLoader.java | 179 ++++++++
.../db/driver/batchloader/IRowOperator.java | 33 ++
.../rocketmq/streams/db/driver/orm/ORMUtil.java | 490 +++++++++++++++++++++
.../rocketmq/streams/db/operator/SQLOperator.java | 178 ++++++++
.../org/apache/rocketmq/streams/db/Person.java | 110 +++++
.../DBSupportParentConfigureServiceTest.java | 74 ++++
.../streams/db/driver/orm/ORMUtilTest.java | 86 ++++
rocketmq-streams-dim/pom.xml | 47 ++
.../apache/rocketmq/streams/dim/DimComponent.java | 63 +++
.../rocketmq/streams/dim/builder/DimBuilder.java | 94 ++++
.../function/expression/InExpressionResource.java | 80 ++++
.../expression/NotInExpressionResource.java | 45 ++
.../dim/function/script/IntelligenceFunction.java | 81 ++++
.../script/IntelligenceNameListFunction.java | 24 +
.../dim/function/script/NameListFunction.java | 203 +++++++++
.../rocketmq/streams/dim/index/DimIndex.java | 319 ++++++++++++++
.../rocketmq/streams/dim/index/IndexExecutor.java | 258 +++++++++++
.../intelligence/AbstractIntelligenceCache.java | 395 +++++++++++++++++
.../dim/intelligence/AccountIntelligenceCache.java | 77 ++++
.../dim/intelligence/DomainIntelligenceCache.java | 83 ++++
.../dim/intelligence/IPIntelligenceCache.java | 108 +++++
.../dim/intelligence/URLIntelligenceCache.java | 80 ++++
.../rocketmq/streams/dim/model/AbstractDim.java | 312 +++++++++++++
.../streams/dim/model/BooleanFieldDBDim.java | 55 +++
.../apache/rocketmq/streams/dim/model/DBDim.java | 140 ++++++
.../rocketmq/streams/dim/service/IDimService.java | 65 +++
.../streams/dim/service/impl/DimServiceImpl.java | 92 ++++
.../com/aliyun/service/ConfigureLoaderTest.java | 37 ++
.../com/aliyun/service/ExpressionExecutorTest.java | 80 ++++
.../java/com/aliyun/service/JsonParserTest.java | 40 ++
.../com/aliyun/service/NameListFunctionTest.java | 90 ++++
.../java/com/aliyun/service/TableCompressTest.java | 26 ++
rocketmq-streams-lease/pom.xml | 25 ++
.../rocketmq/streams/lease/LeaseComponent.java | 103 +++++
.../rocketmq/streams/lease/model/LeaseInfo.java | 127 ++++++
.../streams/lease/service/ILeaseGetCallback.java | 30 ++
.../streams/lease/service/ILeaseService.java | 136 ++++++
.../streams/lease/service/ILeaseStorage.java | 73 +++
.../streams/lease/service/ILeaseStorasge.java | 63 +++
.../lease/service/impl/BasedLesaseImpl.java | 404 +++++++++++++++++
.../lease/service/impl/LeaseServiceImpl.java | 275 ++++++++++++
.../streams/lease/service/impl/MockLeaseImpl.java | 95 ++++
.../lease/service/storages/DBLeaseStorage.java | 229 ++++++++++
.../rocketmq/streams/lease/LeaseComponentTest.java | 119 +++++
.../src/test/resources/log4j.xml | 20 +
rocketmq-streams-transport-minio/pom.xml | 25 ++
.../rocketmq-streams-transport-minio.iml | 17 +
.../transport/minio/MinioFileTransport.java | 141 ++++++
.../yundun/dipper/configurable/DataTpyeTest.java | 70 +++
.../streams/configuable/model/DataTpyeTest.java | 68 +++
.../rocketmq/streams/configuable/model/Person.java | 97 ++++
.../streams/configurable/model/Person.java | 97 ++++
.../component/ConfigurableComponent.properties | 7 +
.../src/test/resources/log4j.xml | 20 +
.../src/test/resources/pro-function.txt | 11 +
.../src/test/resources/python_script.py | 22 +
96 files changed, 10585 insertions(+), 1 deletion(-)
[rocketmq-streams] 26/35: Merge pull request #5 from cw68ster/main
Posted by du...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
duhengforever pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/rocketmq-streams.git
commit 5ef7b1d40a99262078430f691fed2afa8c8352c2
Merge: f349dd9 1992a35
Author: rongtong <ji...@163.com>
AuthorDate: Mon Aug 2 18:55:10 2021 +0800
Merge pull request #5 from cw68ster/main
add module channel-db,channel-configurable
rocketmq-streams-channel-db/pom.xml | 21 +
.../streams/db/sink/AbstractMultiTableSink.java | 150 ++++++
.../apache/rocketmq/streams/db/sink/DBSink.java | 239 +++++++++
.../rocketmq/streams/db/sink/DBSinkBuilder.java | 76 +++
.../streams/db/sink/SelfMultiTableSink.java | 53 ++
.../streams/db/sink/SplitBySerialNumber.java | 36 ++
.../streams/db/sink/SplitByTimeMultiTableSink.java | 36 ++
.../streams/db/sink/db/DBWriteOnlyChannelTest.java | 84 +++
rocketmq-streams-configurable/pom.xml | 21 +
.../streams/configuable/ConfigurableComponent.java | 189 +++++++
.../streams/configuable/model/Configure.java | 130 +++++
.../service/AbstractConfigurableService.java | 561 +++++++++++++++++++++
.../AbstractSupportParentConfigureService.java | 263 ++++++++++
.../service/ConfigurableServcieType.java | 31 ++
.../service/ConfigurableServiceFactory.java | 58 +++
.../service/impl/FileConfigureService.java | 250 +++++++++
.../impl/FileSupportParentConfigureService.java | 38 ++
.../service/impl/MemoryConfigureService.java | 122 +++++
.../impl/MemorySupportParentConfigureService.java | 39 ++
.../configurable/ConfigurableComponent.java | 188 +++++++
.../streams/configurable/model/Configure.java | 127 +++++
.../service/AbstractConfigurableService.java | 553 ++++++++++++++++++++
.../AbstractSupportParentConfigureService.java | 255 ++++++++++
.../service/ConfigurableServcieType.java | 29 ++
.../service/ConfigurableServiceFactory.java | 59 +++
.../service/impl/FileConfigureService.java | 249 +++++++++
.../impl/FileSupportParentConfigureService.java | 37 ++
.../service/impl/MemoryConfigureService.java | 121 +++++
.../impl/MemorySupportParentConfigureService.java | 37 ++
.../src/main/resources/log4j.xml | 20 +
.../configuable/ConfiguableComponentTest.java | 112 ++++
.../rocketmq/streams/configuable/model/Person.java | 110 ++++
.../configurable/ConfigurableComponentTest.java | 108 ++++
.../streams/configurable/model/Person.java | 110 ++++
.../src/test/resources/log4j.xml | 20 +
35 files changed, 4532 insertions(+)
[rocketmq-streams] 04/35: add rocketmq-streams,
rocketmq-channel-rocketmq, rocketmq-serviceloader module
Posted by du...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
duhengforever pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/rocketmq-streams.git
commit 2564d592a288c8548a83662e65e5eb1d2ac96944
Author: 赤玄 <yu...@alibaba-inc.com>
AuthorDate: Mon Aug 2 11:58:20 2021 +0800
add rocketmq-streams, rocketmq-channel-rocketmq, rocketmq-serviceloader module
---
rocketmq-streams-channel-rocketmq/pom.xml | 87 +
.../rocketmq-streams-channel-rocketmq.iml | 74 +
.../rocketmq/streams/RocketMQChannelBuilder.java | 90 +
.../apache/rocketmq/streams/RocketMQOffset.java | 89 +
.../streams/queue/RocketMQMessageQueue.java | 107 +
.../apache/rocketmq/streams/sink/RocketMQSink.java | 318 +++
.../rocketmq/streams/source/RocketMQSource.java | 292 ++
.../rocketmq/streams/AbstractChannelTest.java | 58 +
.../rocketmq/streams/RocketMQChannelTest.java | 176 ++
....streams.common.channel.builder.IChannelBuilder | 1 +
.../rocketmq/streams/RocketMQChannelBuilder.class | Bin 0 -> 3971 bytes
.../apache/rocketmq/streams/RocketMQOffset.class | Bin 0 -> 3628 bytes
.../streams/queue/RocketMQMessageQueue.class | Bin 0 -> 3849 bytes
.../rocketmq/streams/sink/RocketMQSink$1.class | Bin 0 -> 2040 bytes
.../rocketmq/streams/sink/RocketMQSink.class | Bin 0 -> 10455 bytes
.../rocketmq/streams/source/RocketMQSource$1.class | Bin 0 -> 3170 bytes
.../rocketmq/streams/source/RocketMQSource.class | Bin 0 -> 12146 bytes
.../target/maven-archiver/pom.properties | 5 +
...ams-channel-rocketmq-2.0.0-SNAPSHOT-sources.jar | Bin 0 -> 13382 bytes
...tmq-streams-channel-rocketmq-2.0.0-SNAPSHOT.jar | Bin 0 -> 20715 bytes
rocketmq-streams-serviceloader/pom.xml | 24 +
.../rocketmq-streams-serviceloader.iml | 61 +
.../serviceloader/IServiceLoaderService.java | 31 +
.../serviceloader/ServiceLoaderComponent.java | 132 +
.../namefinder/IServiceNameGetter.java | 30 +
.../impl/AnnotationServiceNameGetter.java | 39 +
.../streams/serviceloader/utils/FileUtil.java | 73 +
.../component/ServiceLoaderComponent.properties | 1 +
.../serviceloader/ServiceLoaderComponentTest.java | 17 +
.../src/test/resources/log4j.xml | 20 +
...ams.serviceloader.namefinder.IServiceNameGetter | 1 +
.../component/ServiceLoaderComponent.properties | 1 +
.../serviceloader/IServiceLoaderService.class | Bin 0 -> 413 bytes
.../serviceloader/ServiceLoaderComponent.class | Bin 0 -> 6038 bytes
.../namefinder/IServiceNameGetter.class | Bin 0 -> 235 bytes
.../impl/AnnotationServiceNameGetter.class | Bin 0 -> 1138 bytes
.../streams/serviceloader/utils/FileUtil.class | Bin 0 -> 3352 bytes
.../target/maven-archiver/pom.properties | 5 +
...treams-serviceloader-2.0.0-SNAPSHOT-sources.jar | Bin 0 -> 8077 bytes
...cketmq-streams-serviceloader-2.0.0-SNAPSHOT.jar | Bin 0 -> 9883 bytes
.../target/test-classes/log4j.xml | 20 +
.../serviceloader/ServiceLoaderComponentTest.class | Bin 0 -> 1558 bytes
rocketmq-streams-window/pom.xml | 36 +
.../rocketmq-streams-window.iml | 82 +
.../streams/window/builder/WindowBuilder.java | 95 +
.../streams/window/model/FunctionExecutor.java | 51 +
.../rocketmq/streams/window/model/WindowCache.java | 149 +
.../streams/window/model/WindowInstance.java | 506 ++++
.../window/offset/IWindowMaxValueManager.java | 87 +
.../streams/window/offset/WindowMaxValue.java | 89 +
.../window/offset/WindowMaxValueManager.java | 247 ++
.../window/operator/AbstractShuffleWindow.java | 83 +
.../streams/window/operator/AbstractWindow.java | 781 ++++++
.../streams/window/operator/impl/OverWindow.java | 154 ++
.../window/operator/impl/SessionWindow.java | 248 ++
.../window/operator/impl/WindowOperator.java | 451 +++
.../streams/window/operator/join/DBOperator.java | 427 +++
.../streams/window/operator/join/JoinWindow.java | 553 ++++
.../streams/window/operator/join/Operator.java | 76 +
.../window/shuffle/AbstractSystemChannel.java | 321 +++
.../streams/window/shuffle/ShuffleChannel.java | 542 ++++
.../streams/window/source/WindowRireSource.java | 336 +++
.../streams/window/state/WindowBaseValue.java | 188 ++
.../streams/window/state/impl/JoinLeftState.java | 20 +
.../streams/window/state/impl/JoinRightState.java | 20 +
.../streams/window/state/impl/JoinState.java | 94 +
.../streams/window/state/impl/WindowValue.java | 548 ++++
.../window/storage/AbstractWindowStorage.java | 120 +
.../streams/window/storage/ICommonStorage.java | 50 +
.../streams/window/storage/IKeyGenerator.java | 29 +
.../window/storage/IShufflePartitionManager.java | 52 +
.../rocketmq/streams/window/storage/IStorage.java | 79 +
.../streams/window/storage/IWindowStorage.java | 68 +
.../window/storage/ShufflePartitionManager.java | 86 +
.../streams/window/storage/StorageManager.java | 307 +++
.../streams/window/storage/WindowStorage.java | 183 ++
.../streams/window/storage/db/DBStorage.java | 186 ++
.../streams/window/storage/file/FileStorage.java | 196 ++
.../window/storage/rocksdb/RocksdbStorage.java | 293 ++
.../src/main/resources/dipper.cs | 2892 ++++++++++++++++++++
.../src/main/resources/dipper.properties | 21 +
.../streams/configruation/model/Person.java | 84 +
.../streams/configurable/model/Person.java | 97 +
.../streams/window/WindowInstanceTest.java | 122 +
.../streams/window/WindowMaxValueTest.java | 35 +
.../rocketmq/streams/window/WindowValueTest.java | 36 +
.../src/test/resources/log4j.xml | 20 +
rocketmq-streams-window/target/classes/dipper.cs | 2892 ++++++++++++++++++++
.../target/classes/dipper.properties | 21 +
.../streams/window/builder/WindowBuilder.class | Bin 0 -> 3057 bytes
.../streams/window/model/FunctionExecutor.class | Bin 0 -> 1591 bytes
.../streams/window/model/WindowCache.class | Bin 0 -> 6328 bytes
.../streams/window/model/WindowInstance.class | Bin 0 -> 14044 bytes
.../window/offset/IWindowMaxValueManager.class | Bin 0 -> 1137 bytes
.../streams/window/offset/WindowMaxValue.class | Bin 0 -> 2113 bytes
.../window/offset/WindowMaxValueManager.class | Bin 0 -> 9412 bytes
.../window/operator/AbstractShuffleWindow.class | Bin 0 -> 4175 bytes
.../streams/window/operator/AbstractWindow$1.class | Bin 0 -> 1063 bytes
.../streams/window/operator/AbstractWindow.class | Bin 0 -> 29704 bytes
.../streams/window/operator/impl/OverWindow.class | Bin 0 -> 5387 bytes
.../window/operator/impl/SessionWindow.class | Bin 0 -> 13002 bytes
.../window/operator/impl/WindowOperator$1.class | Bin 0 -> 1944 bytes
.../impl/WindowOperator$WindowRowOperator.class | Bin 0 -> 3973 bytes
.../window/operator/impl/WindowOperator.class | Bin 0 -> 17416 bytes
.../streams/window/operator/join/DBOperator.class | Bin 0 -> 16903 bytes
.../window/operator/join/JoinWindow$1.class | Bin 0 -> 2931 bytes
.../window/operator/join/JoinWindow$2.class | Bin 0 -> 2870 bytes
.../streams/window/operator/join/JoinWindow.class | Bin 0 -> 21377 bytes
.../streams/window/operator/join/Operator.class | Bin 0 -> 2912 bytes
.../window/shuffle/AbstractSystemChannel.class | Bin 0 -> 11888 bytes
.../ShuffleChannel$ShuffleOutputDataSource.class | Bin 0 -> 5054 bytes
.../streams/window/shuffle/ShuffleChannel.class | Bin 0 -> 22485 bytes
.../window/source/WindowRireSource$1$1.class | Bin 0 -> 1524 bytes
.../streams/window/source/WindowRireSource$1.class | Bin 0 -> 2346 bytes
.../WindowRireSource$WindowInstanceCache$1$1.class | Bin 0 -> 1833 bytes
.../WindowRireSource$WindowInstanceCache$1.class | Bin 0 -> 2199 bytes
.../WindowRireSource$WindowInstanceCache.class | Bin 0 -> 1592 bytes
.../streams/window/source/WindowRireSource.class | Bin 0 -> 8081 bytes
.../streams/window/state/WindowBaseValue.class | Bin 0 -> 4083 bytes
.../streams/window/state/impl/JoinLeftState.class | Bin 0 -> 395 bytes
.../streams/window/state/impl/JoinRightState.class | Bin 0 -> 398 bytes
.../streams/window/state/impl/JoinState.class | Bin 0 -> 1943 bytes
.../streams/window/state/impl/WindowValue$1.class | Bin 0 -> 930 bytes
.../streams/window/state/impl/WindowValue.class | Bin 0 -> 23728 bytes
.../window/storage/AbstractWindowStorage$1.class | Bin 0 -> 2596 bytes
.../window/storage/AbstractWindowStorage.class | Bin 0 -> 6724 bytes
.../streams/window/storage/ICommonStorage.class | Bin 0 -> 968 bytes
.../streams/window/storage/IKeyGenerator.class | Bin 0 -> 193 bytes
.../window/storage/IShufflePartitionManager.class | Bin 0 -> 642 bytes
.../rocketmq/streams/window/storage/IStorage.class | Bin 0 -> 2875 bytes
.../streams/window/storage/IWindowStorage.class | Bin 0 -> 2282 bytes
.../window/storage/ShufflePartitionManager.class | Bin 0 -> 2851 bytes
.../streams/window/storage/StorageManager.class | Bin 0 -> 353 bytes
.../streams/window/storage/WindowStorage$1.class | Bin 0 -> 1402 bytes
.../WindowStorage$WindowBaseValueIterator.class | Bin 0 -> 980 bytes
.../streams/window/storage/WindowStorage.class | Bin 0 -> 7364 bytes
.../streams/window/storage/db/DBStorage$1.class | Bin 0 -> 1523 bytes
.../window/storage/db/DBStorage$DBIterator.class | Bin 0 -> 4299 bytes
.../streams/window/storage/db/DBStorage.class | Bin 0 -> 7161 bytes
.../window/storage/file/FileStorage$1.class | Bin 0 -> 2620 bytes
.../streams/window/storage/file/FileStorage.class | Bin 0 -> 9198 bytes
.../rocksdb/RocksdbStorage$LocalIterator.class | Bin 0 -> 3346 bytes
.../window/storage/rocksdb/RocksdbStorage.class | Bin 0 -> 11737 bytes
.../target/maven-archiver/pom.properties | 5 +
...ketmq-streams-window-2.0.0-SNAPSHOT-sources.jar | Bin 0 -> 357527 bytes
.../rocketmq-streams-window-2.0.0-SNAPSHOT.jar | Bin 0 -> 403392 bytes
.../target/test-classes/log4j.xml | 20 +
147 files changed, 15830 insertions(+)
diff --git a/rocketmq-streams-channel-rocketmq/pom.xml b/rocketmq-streams-channel-rocketmq/pom.xml
new file mode 100644
index 0000000..835322c
--- /dev/null
+++ b/rocketmq-streams-channel-rocketmq/pom.xml
@@ -0,0 +1,87 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>rocketmq-streams</artifactId>
+ <groupId>org.apache.rocketmq</groupId>
+ <version>2.0.0-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>rocketmq-streams-channel-rocketmq</artifactId>
+ <name>ROCKETMQ STREAMS :: channel-rocketmq</name>
+ <packaging>jar</packaging>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams-commons</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams-serviceloader</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-client</artifactId>
+ <exclusions>
+ <exclusion>
+ <groupId>ch.qos.logback</groupId>
+ <artifactId>logback-classic</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>ch.qos.logback</groupId>
+ <artifactId>logback-core</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-acl</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>junit</groupId>
+ <artifactId>junit</artifactId>
+ <scope>test</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-api</artifactId>
+ <version>1.7.7</version>
+ </dependency>
+ <dependency>
+ <groupId>org.slf4j</groupId>
+ <artifactId>jcl-over-slf4j</artifactId>
+ <version>1.7.7</version>
+ </dependency>
+ <dependency>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-log4j12</artifactId>
+ <version>1.7.7</version>
+ </dependency>
+ <dependency>
+ <groupId>log4j</groupId>
+ <artifactId>log4j</artifactId>
+ <version>1.2.17</version>
+ </dependency>
+ </dependencies>
+
+</project>
\ No newline at end of file
diff --git a/rocketmq-streams-channel-rocketmq/rocketmq-streams-channel-rocketmq.iml b/rocketmq-streams-channel-rocketmq/rocketmq-streams-channel-rocketmq.iml
new file mode 100644
index 0000000..d0b5aa3
--- /dev/null
+++ b/rocketmq-streams-channel-rocketmq/rocketmq-streams-channel-rocketmq.iml
@@ -0,0 +1,74 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<module org.jetbrains.idea.maven.project.MavenProjectsManager.isMavenModule="true" type="JAVA_MODULE" version="4">
+ <component name="NewModuleRootManager" LANGUAGE_LEVEL="JDK_1_8">
+ <output url="file://$MODULE_DIR$/target/classes" />
+ <output-test url="file://$MODULE_DIR$/target/test-classes" />
+ <content url="file://$MODULE_DIR$">
+ <sourceFolder url="file://$MODULE_DIR$/src/main/java" isTestSource="false" />
+ <sourceFolder url="file://$MODULE_DIR$/src/test/java" isTestSource="true" />
+ <excludeFolder url="file://$MODULE_DIR$/target" />
+ </content>
+ <orderEntry type="inheritedJdk" />
+ <orderEntry type="sourceFolder" forTests="false" />
+ <orderEntry type="module" module-name="rocketmq-streams-commons" />
+ <orderEntry type="library" name="Maven: org.apache.commons:commons-lang3:3.11" level="project" />
+ <orderEntry type="library" name="Maven: com.alibaba:fastjson:1.2.27" level="project" />
+ <orderEntry type="library" name="Maven: commons-logging:commons-logging:1.1" level="project" />
+ <orderEntry type="library" name="Maven: logkit:logkit:1.0.1" level="project" />
+ <orderEntry type="library" name="Maven: avalon-framework:avalon-framework:4.1.3" level="project" />
+ <orderEntry type="library" name="Maven: javax.servlet:servlet-api:2.3" level="project" />
+ <orderEntry type="library" name="Maven: commons-io:commons-io:2.5" level="project" />
+ <orderEntry type="library" name="Maven: com.google.code.gson:gson:2.8.5" level="project" />
+ <orderEntry type="library" name="Maven: com.google.auto.service:auto-service:1.0-rc5" level="project" />
+ <orderEntry type="library" name="Maven: com.google.auto.service:auto-service-annotations:1.0-rc5" level="project" />
+ <orderEntry type="library" name="Maven: com.google.auto:auto-common:0.10" level="project" />
+ <orderEntry type="library" name="Maven: com.google.guava:guava:25.1-jre" level="project" />
+ <orderEntry type="library" name="Maven: com.google.code.findbugs:jsr305:3.0.2" level="project" />
+ <orderEntry type="library" name="Maven: org.checkerframework:checker-qual:2.0.0" level="project" />
+ <orderEntry type="library" name="Maven: com.google.errorprone:error_prone_annotations:2.1.3" level="project" />
+ <orderEntry type="library" name="Maven: com.google.j2objc:j2objc-annotations:1.1" level="project" />
+ <orderEntry type="library" name="Maven: org.codehaus.mojo:animal-sniffer-annotations:1.14" level="project" />
+ <orderEntry type="library" name="Maven: com.lmax:disruptor:3.2.0" level="project" />
+ <orderEntry type="library" name="Maven: com.gliwka.hyperscan:hyperscan:5.4.0-2.0.0" level="project" />
+ <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:5.4.0-1.0.0" level="project" />
+ <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:linux-x86_64:5.4.0-1.0.0" level="project" />
+ <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:windows-x86_64:5.4.0-1.0.0" level="project" />
+ <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:macosx-x86_64:5.4.0-1.0.0" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp-platform:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-arm:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-arm64:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-x86:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-x86_64:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:ios-arm64:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:ios-x86_64:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-armhf:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-arm64:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-ppc64le:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-x86:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-x86_64:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:macosx-x86_64:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:windows-x86:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:windows-x86_64:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: net.java.dev.jna:platform:3.5.2" level="project" />
+ <orderEntry type="library" name="Maven: net.java.dev.jna:jna:3.5.2" level="project" />
+ <orderEntry type="module" module-name="rocketmq-streams-serviceloader" />
+ <orderEntry type="library" name="Maven: org.apache.rocketmq:rocketmq-client:4.5.2" level="project" />
+ <orderEntry type="library" name="Maven: org.apache.rocketmq:rocketmq-common:4.5.2" level="project" />
+ <orderEntry type="library" name="Maven: org.apache.rocketmq:rocketmq-acl:4.5.2" level="project" />
+ <orderEntry type="library" name="Maven: org.apache.rocketmq:rocketmq-remoting:4.5.2" level="project" />
+ <orderEntry type="library" name="Maven: io.netty:netty-all:4.0.42.Final" level="project" />
+ <orderEntry type="library" name="Maven: io.netty:netty-tcnative-boringssl-static:1.1.33.Fork26" level="project" />
+ <orderEntry type="library" name="Maven: org.apache.rocketmq:rocketmq-logging:4.5.2" level="project" />
+ <orderEntry type="library" name="Maven: org.apache.rocketmq:rocketmq-srvutil:4.5.2" level="project" />
+ <orderEntry type="library" name="Maven: commons-cli:commons-cli:1.2" level="project" />
+ <orderEntry type="library" name="Maven: org.yaml:snakeyaml:1.19" level="project" />
+ <orderEntry type="library" name="Maven: commons-codec:commons-codec:1.9" level="project" />
+ <orderEntry type="library" scope="TEST" name="Maven: junit:junit:4.12" level="project" />
+ <orderEntry type="library" scope="TEST" name="Maven: org.hamcrest:hamcrest-core:1.3" level="project" />
+ <orderEntry type="library" name="Maven: org.slf4j:slf4j-api:1.7.7" level="project" />
+ <orderEntry type="library" name="Maven: org.slf4j:jcl-over-slf4j:1.7.7" level="project" />
+ <orderEntry type="library" name="Maven: org.slf4j:slf4j-log4j12:1.7.7" level="project" />
+ <orderEntry type="library" name="Maven: log4j:log4j:1.2.17" level="project" />
+ </component>
+</module>
\ No newline at end of file
diff --git a/rocketmq-streams-channel-rocketmq/src/main/java/org/apache/rocketmq/streams/RocketMQChannelBuilder.java b/rocketmq-streams-channel-rocketmq/src/main/java/org/apache/rocketmq/streams/RocketMQChannelBuilder.java
new file mode 100644
index 0000000..e83b1dc
--- /dev/null
+++ b/rocketmq-streams-channel-rocketmq/src/main/java/org/apache/rocketmq/streams/RocketMQChannelBuilder.java
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.streams;
+
+import com.alibaba.fastjson.JSONObject;
+import com.google.auto.service.AutoService;
+import org.apache.rocketmq.streams.common.channel.builder.AbstractSupportShuffleChannelBuilder;
+import org.apache.rocketmq.streams.common.channel.builder.IChannelBuilder;
+import org.apache.rocketmq.streams.common.channel.sink.ISink;
+import org.apache.rocketmq.streams.common.channel.source.ISource;
+import org.apache.rocketmq.streams.common.metadata.MetaData;
+import org.apache.rocketmq.streams.common.model.ServiceName;
+import org.apache.rocketmq.streams.common.utils.ConfigurableUtil;
+import org.apache.rocketmq.streams.sink.RocketMQSink;
+import org.apache.rocketmq.streams.source.RocketMQSource;
+
+import java.util.Properties;
+
+@AutoService(IChannelBuilder.class)
+@ServiceName(value = RocketMQChannelBuilder.TYPE, aliasName = "RocketMQSource")
+public class RocketMQChannelBuilder extends AbstractSupportShuffleChannelBuilder {
+ public static final String TYPE = "rocketmq";
+
+ @Override
+ public ISource createSource(String namespace, String name, Properties properties, MetaData metaData) {
+
+ RocketMQSource rocketMQSource = (RocketMQSource) ConfigurableUtil.create(RocketMQSource.class.getName(),namespace,name,createFormatProperty(properties),null);
+ return rocketMQSource;
+ }
+
+ protected JSONObject createFormatProperty(Properties properties){
+ JSONObject formatProperties=new JSONObject();
+ for(Object object:properties.keySet()){
+ String key=(String)object;
+ if ("type".equals(key)) {
+ continue;
+ }
+ formatProperties.put(key,properties.getProperty(key));
+ }
+ IChannelBuilder.formatPropertiesName(formatProperties,properties,"topic","topic");
+ IChannelBuilder.formatPropertiesName(formatProperties,properties,"tags","tag");
+ IChannelBuilder.formatPropertiesName(formatProperties,properties,"maxThread","thread.max.count");
+ IChannelBuilder.formatPropertiesName(formatProperties,properties,"pullIntervalMs","pullIntervalMs");
+ IChannelBuilder.formatPropertiesName(formatProperties,properties,"offsetTime","offsetTime");
+ IChannelBuilder.formatPropertiesName(formatProperties,properties,"namesrvAddr","namesrvAddr");
+ if (properties.getProperty("group") != null) {
+ String group = properties.getProperty("group");
+ if (group.startsWith("GID_")) {
+ formatProperties.put("groupName", group);
+ } else {
+ formatProperties.put("groupName", "GID_" + group);
+ }
+ }
+
+
+ return formatProperties;
+ }
+
+
+ @Override
+ public String getType() {
+ return TYPE;
+ }
+
+ @Override
+ public ISink createSink(String namespace, String name, Properties properties, MetaData metaData) {
+ RocketMQSink rocketMQSink = (RocketMQSink) ConfigurableUtil.create(RocketMQSink.class.getName(),namespace,name,createFormatProperty(properties),null);
+ return rocketMQSink;
+ }
+
+ @Override
+ public ISink createBySource(ISource piplineSoure) {
+ return null;
+ }
+}
diff --git a/rocketmq-streams-channel-rocketmq/src/main/java/org/apache/rocketmq/streams/RocketMQOffset.java b/rocketmq-streams-channel-rocketmq/src/main/java/org/apache/rocketmq/streams/RocketMQOffset.java
new file mode 100644
index 0000000..ef0c322
--- /dev/null
+++ b/rocketmq-streams-channel-rocketmq/src/main/java/org/apache/rocketmq/streams/RocketMQOffset.java
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.streams;
+
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import org.apache.rocketmq.client.consumer.store.OffsetStore;
+import org.apache.rocketmq.client.consumer.store.ReadOffsetType;
+import org.apache.rocketmq.client.exception.MQBrokerException;
+import org.apache.rocketmq.client.exception.MQClientException;
+import org.apache.rocketmq.common.message.MessageQueue;
+import org.apache.rocketmq.remoting.exception.RemotingException;
+import org.apache.rocketmq.streams.common.channel.source.AbstractSupportOffsetResetSource;
+import org.apache.rocketmq.streams.queue.RocketMQMessageQueue;
+
+public class RocketMQOffset implements OffsetStore {
+ protected OffsetStore offsetStore;
+ protected AbstractSupportOffsetResetSource source;
+ public RocketMQOffset(OffsetStore offsetStore, AbstractSupportOffsetResetSource source){
+ this.offsetStore=offsetStore;
+ this.source=source;
+ }
+ @Override
+ public void load() throws MQClientException {
+ offsetStore.load();
+ }
+
+ @Override
+ public void updateOffset(MessageQueue mq, long offset, boolean increaseOnly) {
+ offsetStore.updateOffset(mq,offset,increaseOnly);
+ }
+
+ @Override
+ public long readOffset(MessageQueue mq, ReadOffsetType type) {
+ return offsetStore.readOffset(mq,type);
+ }
+
+ @Override
+ public void persistAll(Set<MessageQueue> mqs) {
+ Set<String> queueIds=new HashSet<>();
+ for(MessageQueue mq:mqs){
+ queueIds.add(new RocketMQMessageQueue(mq).getQueueId());
+ }
+ source.sendCheckpoint(queueIds);
+ offsetStore.persistAll(mqs);
+ }
+
+ @Override
+ public void persist(MessageQueue mq) {
+ source.sendCheckpoint(new RocketMQMessageQueue(mq).getQueueId());
+ offsetStore.persist(mq);
+ }
+
+ @Override
+ public void removeOffset(MessageQueue mq) {
+ Set<String> splitIds = new HashSet<>();
+ splitIds.add(new RocketMQMessageQueue(mq).getQueueId());
+ source.removeSplit(splitIds);
+ offsetStore.removeOffset(mq);
+ offsetStore.removeOffset(mq);
+ }
+
+ @Override
+ public Map<MessageQueue, Long> cloneOffsetTable(String topic) {
+ return offsetStore.cloneOffsetTable(topic);
+ }
+
+ @Override
+ public void updateConsumeOffsetToBroker(MessageQueue mq, long offset, boolean isOneway)
+ throws RemotingException, MQBrokerException, InterruptedException, MQClientException {
+ offsetStore.updateOffset(mq,offset,isOneway);
+ }
+}
diff --git a/rocketmq-streams-channel-rocketmq/src/main/java/org/apache/rocketmq/streams/queue/RocketMQMessageQueue.java b/rocketmq-streams-channel-rocketmq/src/main/java/org/apache/rocketmq/streams/queue/RocketMQMessageQueue.java
new file mode 100644
index 0000000..46f0ef9
--- /dev/null
+++ b/rocketmq-streams-channel-rocketmq/src/main/java/org/apache/rocketmq/streams/queue/RocketMQMessageQueue.java
@@ -0,0 +1,107 @@
+package org.apache.rocketmq.streams.queue;
+
+import com.alibaba.fastjson.JSONObject;
+import org.apache.rocketmq.common.message.MessageQueue;
+import org.apache.rocketmq.streams.common.channel.split.ISplit;
+import org.apache.rocketmq.streams.common.configurable.BasedConfigurable;
+import org.apache.rocketmq.streams.common.utils.MapKeyUtil;
+
+public class RocketMQMessageQueue extends BasedConfigurable implements ISplit<RocketMQMessageQueue, MessageQueue> {
+ protected transient MessageQueue queue;
+ protected String brokeName;
+ protected String topic;
+ protected int mqQueueId;
+
+
+ @Override
+ protected void getJsonObject(JSONObject jsonObject) {
+ super.getJsonObject(jsonObject);
+ queue=new MessageQueue(topic,brokeName,mqQueueId);
+ }
+
+
+ public RocketMQMessageQueue(MessageQueue queue) {
+ this.queue = queue;
+ this.brokeName=queue.getBrokerName();
+ this.topic=queue.getTopic();
+ this.mqQueueId=queue.getQueueId();
+ }
+
+ public RocketMQMessageQueue() {
+
+ }
+
+ @Override
+ public MessageQueue getQueue() {
+ return queue;
+ }
+
+ @Override
+ public int compareTo(RocketMQMessageQueue o) {
+ return queue.compareTo(o.queue);
+ }
+
+
+
+ @Override
+ public String getQueueId() {
+ return getQueueId(this.queue);
+ }
+
+ @Override
+ public String getPlusQueueId() {
+ return MapKeyUtil.createKeyBySign("_",queue.getTopic(),queue.getBrokerName(),getSplitNumerStr(queue.getQueueId()+1)+"");
+ }
+
+ public static String getQueueId(MessageQueue queue){
+
+ String[] topic = queue.getTopic().split("%");
+ if (topic.length > 1) {
+ return MapKeyUtil.createKeyBySign("_",topic[1],queue.getBrokerName(),getSplitNumerStr(queue.getQueueId())+"");
+ }
+ return MapKeyUtil.createKeyBySign("_",queue.getTopic(),queue.getBrokerName(),getSplitNumerStr(queue.getQueueId())+"");
+ }
+ /**
+ * 获取分片的字符串格式，需要3位对齐
+ * @param splitNumer
+ * @return
+ */
+ private static String getSplitNumerStr(int splitNumer){
+ int len=(splitNumer+"").length();
+ if(len==3){
+ return splitNumer+"";
+ }
+ String splitNumerStr=splitNumer+"";
+ while (len<3){
+ splitNumerStr="0"+splitNumerStr;
+ len=splitNumerStr.length();
+ }
+ return splitNumerStr;
+ }
+
+ public String getBrokeName() {
+ return brokeName;
+ }
+
+ public void setBrokeName(String brokeName) {
+ this.brokeName = brokeName;
+ }
+
+ public String getTopic() {
+ return topic;
+ }
+
+ public void setTopic(String topic) {
+ this.topic = topic;
+ }
+
+ public int getMqQueueId() {
+ return mqQueueId;
+ }
+
+ public void setMqQueueId(int mqQueueId) {
+ this.mqQueueId = mqQueueId;
+ }
+
+
+}
diff --git a/rocketmq-streams-channel-rocketmq/src/main/java/org/apache/rocketmq/streams/sink/RocketMQSink.java b/rocketmq-streams-channel-rocketmq/src/main/java/org/apache/rocketmq/streams/sink/RocketMQSink.java
new file mode 100644
index 0000000..8caf246
--- /dev/null
+++ b/rocketmq-streams-channel-rocketmq/src/main/java/org/apache/rocketmq/streams/sink/RocketMQSink.java
@@ -0,0 +1,318 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.streams.sink;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.rocketmq.client.consumer.DefaultMQPushConsumer;
+import org.apache.rocketmq.client.producer.DefaultMQProducer;
+import org.apache.rocketmq.client.producer.SendCallback;
+import org.apache.rocketmq.client.producer.SendResult;
+import org.apache.rocketmq.common.message.Message;
+import org.apache.rocketmq.common.message.MessageQueue;
+import org.apache.rocketmq.streams.common.channel.sink.AbstractSupportShuffleSink;
+import org.apache.rocketmq.streams.common.channel.split.ISplit;
+import org.apache.rocketmq.streams.common.configurable.annotation.ENVDependence;
+import org.apache.rocketmq.streams.common.context.IMessage;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+import org.apache.rocketmq.streams.queue.RocketMQMessageQueue;
+
+import java.util.*;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+
+public class RocketMQSink extends AbstractSupportShuffleSink {
+
+ protected static final Log LOG = LogFactory.getLog(RocketMQSink.class);
+ @ENVDependence
+ protected String tags = "*";
+
+ protected String topic;
+ protected String groupName;
+
+ private transient List<DefaultMQPushConsumer> consumers=new ArrayList<>();
+ protected transient DefaultMQProducer producer;
+
+ protected Long pullIntervalMs;
+ protected String namesrvAddr;
+
+
+ public RocketMQSink(){}
+
+
+ @Override
+ protected boolean initConfigurable() {
+ super.initConfigurable();
+ return true;
+ }
+
+ protected transient AtomicBoolean isProcessing=new AtomicBoolean(false);
+ @Override
+ protected boolean batchInsert(List<IMessage> messages) {
+ if (messages == null) {
+ return true;
+ }
+ List<IMessage> needOrderProducer=new ArrayList<>();
+ for (int i=0;i<messages.size();i++) {
+ IMessage message =messages.get(i);
+ if(getSPlit(message)!=null){
+ if(i==0){
+ needOrderProducer=messages;
+ break;
+ }
+ needOrderProducer.add(message);
+ continue;
+ }
+ AtomicInteger msgFinishCount=new AtomicInteger(1);
+ putMessage2Mq(message,msgFinishCount,messages.size());
+ }
+ if(needOrderProducer.size()==0){
+ return true;
+ }
+ boolean success= isProcessing.compareAndSet(false,true);
+ if(!success){
+ while (isProcessing.get()){
+ synchronized (this){
+ try {
+ this.wait();
+ success= isProcessing.compareAndSet(false,true);
+ if(success){
+ break;
+ }
+ } catch (InterruptedException e) {
+ e.printStackTrace();
+ }
+
+ }
+ }
+ }
+ AtomicInteger msgFinishCount=new AtomicInteger(0);
+ for(IMessage message:needOrderProducer){
+ putMessage2Mq(message,msgFinishCount,needOrderProducer.size());
+ }
+
+
+ return true;
+ }
+
+ protected boolean putMessage2Mq(IMessage fieldName2Value,AtomicInteger msgFinishCount,int allMsgSize) {
+ MessageQueue targetQueue = null;
+ ISplit<RocketMQMessageQueue,MessageQueue> channelQueue=getSPlit(fieldName2Value);
+
+ if (channelQueue!= null) {
+ targetQueue = channelQueue.getQueue();
+ }
+ sendMessage(fieldName2Value.getMessageValue().toString(), null, this.tags, targetQueue,msgFinishCount,allMsgSize);
+ return true;
+ }
+
+ /**
+ * 发送metaq消息
+ * @param content 消息内容
+ * @param key 消息的Key字段是为了唯一标识消息的，方便运维排查问题。如果不设置Key，则无法定位消息丢失原因。
+ * @param targetQueue
+ */
+ protected void sendMessage(String content, String key, String tags, MessageQueue targetQueue,AtomicInteger msgFinishCount,int allMsgSize) {
+ try {
+ if (StringUtil.isEmpty(topic)) {
+ if (LOG.isErrorEnabled()) {
+ LOG.error("topic is blank:" + content);
+ }
+ return;
+ }
+ initProducer();
+ final RocketMQSink rocketMQSink=this;
+ Message msg = new Message(topic, tags, key, content.getBytes("UTF-8"));
+ if (targetQueue != null) {
+ producer.send(msg, targetQueue,new SendCallback() {
+ @Override
+ public void onSuccess(SendResult sendResult) {
+ int finishCount=msgFinishCount.incrementAndGet();
+ if(finishCount==allMsgSize){
+ synchronized (rocketMQSink){
+ isProcessing.set(false);
+ rocketMQSink.notifyAll();;
+ }
+ }
+ }
+
+ @Override
+ public void onException(Throwable e) {
+ int finishCount=msgFinishCount.incrementAndGet();
+ if(finishCount==allMsgSize){
+ synchronized (rocketMQSink){
+ isProcessing.set(false);
+ rocketMQSink.notifyAll();;
+ }
+ }
+ }
+ });
+ } else {
+ producer.sendOneway(msg );
+ }
+ } catch (Exception e) {
+ LOG.error("send message error:" + content, e);
+ }
+ }
+
+ protected void initProducer() {
+ if(producer==null){
+ synchronized (this){
+ if(producer==null){
+ destroy();
+ producer = new DefaultMQProducer(groupName + "producer", true, null);
+ try {
+ if (this.namesrvAddr != null && !"".equalsIgnoreCase(this.namesrvAddr)) {
+ producer.setNamesrvAddr(this.namesrvAddr);
+ }
+ producer.start();
+ } catch (Exception e) {
+ setInitSuccess(false);
+ throw new RuntimeException("创建队列失败," + topic + ",msg=" + e.getMessage(), e);
+ }
+ }
+ }
+
+ }
+ }
+
+ public void destroyProduce() {
+ if (producer != null) {
+ try {
+ producer.shutdown();
+ producer=null;
+ } catch (Throwable t) {
+ if (LOG.isWarnEnabled()) {
+ LOG.warn(t.getMessage(), t);
+ }
+ }
+ }
+ }
+
+ @Override
+ public void destroy() {
+ super.destroy();
+ destroyProduce();
+ }
+
+ @Override
+ public String getShuffleTopicFieldName() {
+ return "topic";
+ }
+
+ @Override
+ protected void createTopicIfNotExist(int splitNum) {
+
+ }
+
+ @Override
+ public List<ISplit> getSplitList() {
+ initProducer();
+ List<ISplit> messageQueues=new ArrayList<>();
+ try {
+
+ if (messageQueues == null || messageQueues.size() == 0) {
+ List<MessageQueue> metaqQueueSet = producer.fetchPublishMessageQueues(topic);
+ List<ISplit> queueList = new ArrayList<>();
+ for (MessageQueue queue : metaqQueueSet) {
+ RocketMQMessageQueue rocketMQMessageQueue = new RocketMQMessageQueue(queue);
+ queueList.add(rocketMQMessageQueue);
+
+ }
+ Collections.sort(queueList);
+ messageQueues = queueList;
+ }
+ }catch (Exception e){
+ throw new RuntimeException(e);
+ }
+
+ return messageQueues;
+ }
+
+ @Override
+ public int getSplitNum() {
+ List<ISplit> splits=getSplitList();
+ if(splits==null||splits.size()==0){
+ return 0;
+ }
+ Set<Integer> splitNames=new HashSet<>();
+ for(ISplit split:splits){
+ MessageQueue messageQueue= (MessageQueue)split.getQueue();
+ splitNames.add(messageQueue.getQueueId());
+ }
+ return splitNames.size();
+ }
+
+
+ public String getTags() {
+ return tags;
+ }
+
+ public void setTags(String tags) {
+ this.tags = tags;
+ }
+
+ public String getTopic() {
+ return topic;
+ }
+
+ public void setTopic(String topic) {
+ this.topic = topic;
+ }
+
+ public String getGroupName() {
+ return groupName;
+ }
+
+ public void setGroupName(String groupName) {
+ this.groupName = groupName;
+ }
+
+ public List<DefaultMQPushConsumer> getConsumers() {
+ return consumers;
+ }
+
+ public void setConsumers(List<DefaultMQPushConsumer> consumers) {
+ this.consumers = consumers;
+ }
+
+ public DefaultMQProducer getProducer() {
+ return producer;
+ }
+
+ public void setProducer(DefaultMQProducer producer) {
+ this.producer = producer;
+ }
+
+ public Long getPullIntervalMs() {
+ return pullIntervalMs;
+ }
+
+ public void setPullIntervalMs(Long pullIntervalMs) {
+ this.pullIntervalMs = pullIntervalMs;
+ }
+
+ public String getNamesrvAddr() {
+ return namesrvAddr;
+ }
+
+ public void setNamesrvAddr(String namesrvAddr) {
+ this.namesrvAddr = namesrvAddr;
+ }
+
+}
diff --git a/rocketmq-streams-channel-rocketmq/src/main/java/org/apache/rocketmq/streams/source/RocketMQSource.java b/rocketmq-streams-channel-rocketmq/src/main/java/org/apache/rocketmq/streams/source/RocketMQSource.java
new file mode 100644
index 0000000..c2af211
--- /dev/null
+++ b/rocketmq-streams-channel-rocketmq/src/main/java/org/apache/rocketmq/streams/source/RocketMQSource.java
@@ -0,0 +1,292 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.streams.source;
+
+import com.alibaba.fastjson.JSONObject;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.rocketmq.client.AccessChannel;
+import org.apache.rocketmq.client.consumer.DefaultMQPushConsumer;
+import org.apache.rocketmq.client.consumer.listener.ConsumeOrderlyStatus;
+import org.apache.rocketmq.client.consumer.listener.MessageListenerOrderly;
+import org.apache.rocketmq.client.consumer.store.RemoteBrokerOffsetStore;
+import org.apache.rocketmq.client.impl.MQClientManager;
+import org.apache.rocketmq.client.impl.consumer.DefaultMQPushConsumerImpl;
+import org.apache.rocketmq.client.impl.consumer.ProcessQueue;
+import org.apache.rocketmq.client.impl.consumer.RebalancePushImpl;
+import org.apache.rocketmq.client.impl.factory.MQClientInstance;
+import org.apache.rocketmq.common.consumer.ConsumeFromWhere;
+import org.apache.rocketmq.common.message.MessageExt;
+import org.apache.rocketmq.common.message.MessageQueue;
+import org.apache.rocketmq.common.protocol.NamespaceUtil;
+import org.apache.rocketmq.common.protocol.heartbeat.MessageModel;
+import org.apache.rocketmq.streams.common.channel.source.AbstractSupportOffsetResetSource;
+import org.apache.rocketmq.streams.common.configurable.annotation.ENVDependence;
+import org.apache.rocketmq.streams.common.utils.ReflectUtil;
+import org.apache.rocketmq.streams.RocketMQOffset;
+import org.apache.rocketmq.streams.queue.RocketMQMessageQueue;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+public class RocketMQSource extends AbstractSupportOffsetResetSource {
+
+ protected static final Log LOG = LogFactory.getLog(RocketMQSource.class);
+ @ENVDependence
+ protected String tags = "*";
+
+ private transient List<DefaultMQPushConsumer> consumers = new ArrayList<>();
+
+ protected Long pullIntervalMs;
+
+ /**
+ * 消息队列命名空间接入点
+ */
+ protected String namesrvAddr;
+
+ protected transient ConsumeFromWhere consumeFromWhere;//默认从哪里消费,不会被持久化。不设置默认从尾部消费
+ protected transient String consumerOffset;//从哪里开始消费
+
+ public RocketMQSource() {}
+
+ public RocketMQSource(String topic, String tags, String groupName, String endpoint,
+ String namesrvAddr, String accessKey, String secretKey, String instanceId) {
+ this.topic = topic;
+ this.tags = tags;
+ this.groupName = groupName;
+ this.namesrvAddr = namesrvAddr;
+ }
+
+ @Override
+ protected boolean initConfigurable() {
+ return super.initConfigurable();
+ }
+
+ @Override
+ protected boolean startSource() {
+ try {
+ destroyConsumer();
+ consumers.add(startConsumer());
+ return true;
+ } catch (Exception e) {
+ setInitSuccess(false);
+ e.printStackTrace();
+ throw new RuntimeException("start rocketmq channel error " + topic, e);
+ }
+ }
+
+ protected DefaultMQPushConsumer startConsumer() {
+ try {
+ DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(groupName);
+ if (pullIntervalMs != null) {
+ consumer.setPullInterval(pullIntervalMs);
+ }
+ // consumer.setConsumeThreadMax(maxThread);
+ // consumer.setConsumeThreadMin(maxThread);
+
+ consumer.setPersistConsumerOffsetInterval((int)this.checkpointTime);
+ consumer.setConsumeMessageBatchMaxSize(maxFetchLogGroupSize);
+ consumer.setAccessChannel(AccessChannel.CLOUD);
+ consumer.setNamesrvAddr(this.namesrvAddr);
+ if (consumeFromWhere != null) {
+ consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_TIMESTAMP);
+ if (consumerOffset != null) {
+ consumer.setConsumeTimestamp(consumerOffset);
+ }
+ }
+
+ consumer.subscribe(topic, tags);
+ consumer.registerMessageListener((MessageListenerOrderly)(msgs, context) -> {
+ try {
+ int i = 0;
+ for (MessageExt msg : msgs) {
+ String data = new String(msg.getBody(), CHARSET);
+ JSONObject jsonObject = create(data);
+ String queueId = RocketMQMessageQueue.getQueueId(context.getMessageQueue());
+ String offset = msg.getQueueOffset() + "";
+ org.apache.rocketmq.streams.common.context.Message message = createMessage(jsonObject, queueId, offset, false);
+ message.getHeader().setOffsetIsLong(true);
+ // message.getHeader().setQueueId(RocketMQMessageQueue.getQueueId(context.getMessageQueue()));
+ // message.getHeader().setOffset(String.valueOf(msg.getQueueOffset()));
+ // message.getHeader().setMessageQueue(new RocketMQMessageQueue(context.getMessageQueue()));
+ if (i == msgs.size() - 1) {
+ message.getHeader().setNeedFlush(true);
+ }
+ executeMessage(message);
+ i++;
+ }
+ } catch (Exception e) {
+ LOG.error("消费metaq报错：" + e, e);
+ }
+
+ return ConsumeOrderlyStatus.SUCCESS;// 返回消费成功
+ });
+
+ setOffsetStore(consumer);
+ addRebalanceCallback(consumer);
+ consumer.start();
+
+ return consumer;
+ } catch (Exception e) {
+ setInitSuccess(false);
+ e.printStackTrace();
+ throw new RuntimeException("start metaq channel error " + topic, e);
+ }
+ }
+
+ /**
+ * 设置offset存储，包装原有的RemoteBrokerOffsetStore，在保存offset前发送系统消息
+ *
+ * @param consumer
+ */
+ protected void setOffsetStore(DefaultMQPushConsumer consumer) {
+ DefaultMQPushConsumerImpl defaultMQPushConsumer = consumer.getDefaultMQPushConsumerImpl();
+ if (consumer.getMessageModel() == MessageModel.CLUSTERING) {
+ consumer.changeInstanceNameToPID();
+ }
+ MQClientInstance mQClientFactory = MQClientManager.getInstance().getAndCreateMQClientInstance(defaultMQPushConsumer.getDefaultMQPushConsumer());
+ RemoteBrokerOffsetStore offsetStore = new RemoteBrokerOffsetStore(mQClientFactory, NamespaceUtil.wrapNamespace(consumer.getNamespace(), consumer.getConsumerGroup()));
+ consumer.setOffsetStore(new RocketMQOffset(offsetStore, this));//每个一分钟运行一次
+ }
+
+ @Override
+ protected boolean isNotDataSplit(String queueId) {
+ return queueId.toUpperCase().startsWith("%RETRY%");
+ }
+
+ /**
+ * 分片发生变化时，回调系统函数，发送系统消息，告知各个组件
+ *
+ * @param consumer
+ */
+ protected void addRebalanceCallback(DefaultMQPushConsumer consumer) {
+ DefaultMQPushConsumerImpl defaultMQPushConsumerImpl = consumer.getDefaultMQPushConsumerImpl();
+ // DefaultMQPushConsumerImpl defaultMQPushConsumerImpl=metaPushConsumer.getDefaultMQPushConsumerImpl();
+ ReflectUtil.setBeanFieldValue(defaultMQPushConsumerImpl, "rebalanceImpl", new RebalancePushImpl(defaultMQPushConsumerImpl) {
+ @Override
+ public void messageQueueChanged(String topic, Set<MessageQueue> mqAll, Set<MessageQueue> mqDivided) {
+ Set<String> queueIds = new HashSet<>();
+ for (MessageQueue messageQueue : mqAll) {
+ if (!mqDivided.contains(messageQueue)) {
+ ProcessQueue pq = this.processQueueTable.remove(messageQueue);
+ if (pq != null) {
+ pq.setDropped(true);
+ log.info("doRebalance, {}, truncateMessageQueueNotMyTopic remove unnecessary mq, {}", consumerGroup, messageQueue);
+ }
+ queueIds.add(RocketMQMessageQueue.getQueueId(messageQueue));
+ }
+ }
+ Set<String> newQueueIds = new HashSet<>();
+ for (MessageQueue messageQueue : mqDivided) {
+ if (!mqAll.contains(messageQueue)) {
+ newQueueIds.add(RocketMQMessageQueue.getQueueId(messageQueue));
+ }
+ }
+ removeSplit(queueIds);
+ addNewSplit(newQueueIds);
+
+ }
+ });
+ }
+
+ @Override
+ public boolean supportNewSplitFind() {
+ return true;
+ }
+
+ @Override
+ public boolean supportRemoveSplitFind() {
+ return true;
+ }
+
+ @Override
+ public boolean supportOffsetRest() {
+ return false;
+ }
+
+ public void destroyConsumer() {
+ List<DefaultMQPushConsumer> oldConsumers = new ArrayList<>(consumers);
+ try {
+ for (DefaultMQPushConsumer consumer : oldConsumers) {
+ consumer.shutdown();
+ }
+
+ } catch (Throwable t) {
+ if (LOG.isWarnEnabled()) {
+ LOG.warn(t.getMessage(), t);
+ }
+ }
+
+ }
+
+ @Override
+ public void destroy() {
+ super.destroy();
+ destroyConsumer();
+ }
+
+ public String getTags() {
+ return tags;
+ }
+
+ public void setTags(String tags) {
+ this.tags = tags;
+ }
+
+ public List<DefaultMQPushConsumer> getConsumers() {
+ return consumers;
+ }
+
+ public void setConsumers(List<DefaultMQPushConsumer> consumers) {
+ this.consumers = consumers;
+ }
+
+ public Long getPullIntervalMs() {
+ return pullIntervalMs;
+ }
+
+ public void setPullIntervalMs(Long pullIntervalMs) {
+ this.pullIntervalMs = pullIntervalMs;
+ }
+
+ public String getNamesrvAddr() {
+ return namesrvAddr;
+ }
+
+ public void setNamesrvAddr(String namesrvAddr) {
+ this.namesrvAddr = namesrvAddr;
+ }
+
+ public ConsumeFromWhere getConsumeFromWhere() {
+ return consumeFromWhere;
+ }
+
+ public void setConsumeFromWhere(ConsumeFromWhere consumeFromWhere) {
+ this.consumeFromWhere = consumeFromWhere;
+ }
+
+ public String getConsumerOffset() {
+ return consumerOffset;
+ }
+
+ public void setConsumerOffset(String consumerOffset) {
+ this.consumerOffset = consumerOffset;
+ }
+}
\ No newline at end of file
diff --git a/rocketmq-streams-channel-rocketmq/src/test/java/org/apache/rocketmq/streams/AbstractChannelTest.java b/rocketmq-streams-channel-rocketmq/src/test/java/org/apache/rocketmq/streams/AbstractChannelTest.java
new file mode 100644
index 0000000..34fbf10
--- /dev/null
+++ b/rocketmq-streams-channel-rocketmq/src/test/java/org/apache/rocketmq/streams/AbstractChannelTest.java
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.streams;
+
+import com.alibaba.fastjson.JSONObject;
+import org.apache.rocketmq.streams.common.channel.sink.ISink;
+import org.apache.rocketmq.streams.common.channel.source.ISource;
+import org.apache.rocketmq.streams.common.context.Message;
+import org.junit.Test;
+
+public abstract class AbstractChannelTest {
+
+ @Test
+ public void testChannel() throws InterruptedException {
+ ISource channel = createSource();
+ channel.setGroupName("CID_YUNDUN_SOC_DIPPER_TEST");
+ channel.setMaxThread(1);
+ channel.start((message, context) -> {
+ //System.out.println(message.getMessageBody().getString(IChannel.OFFSET)+"-"+message.getMessageBody()
+ // .getString(IChannel.QUEUE_ID)+"-"+message.getMessageBody().getString(IChannel.IS_BATCH)+"-"+Thread
+ // .currentThread().getId());
+ System.out.println(message.getMessageBody());
+ return message;
+ });
+ while (true) {
+ Thread.sleep(1000L);
+ }
+ }
+
+ protected abstract ISource createSource();
+
+ protected abstract ISink createSink();
+
+ @Test
+ public void testOutput() {
+ ISink channel = createSink();
+ JSONObject message = new JSONObject();
+ message.put("name", "chris");
+ message.put("age", 18);
+ channel.batchAdd(new Message(message));
+ channel.flush();
+ }
+}
diff --git a/rocketmq-streams-channel-rocketmq/src/test/java/org/apache/rocketmq/streams/RocketMQChannelTest.java b/rocketmq-streams-channel-rocketmq/src/test/java/org/apache/rocketmq/streams/RocketMQChannelTest.java
new file mode 100644
index 0000000..e0a0503
--- /dev/null
+++ b/rocketmq-streams-channel-rocketmq/src/test/java/org/apache/rocketmq/streams/RocketMQChannelTest.java
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.streams;
+
+import com.alibaba.fastjson.JSONObject;
+
+import java.util.*;
+
+import org.apache.rocketmq.streams.common.channel.sink.AbstractSupportShuffleSink;
+import org.apache.rocketmq.streams.common.channel.sink.ISink;
+import org.apache.rocketmq.streams.common.channel.source.ISource;
+import org.apache.rocketmq.streams.common.channel.split.ISplit;
+import org.apache.rocketmq.streams.common.component.ComponentCreator;
+import org.apache.rocketmq.streams.common.context.AbstractContext;
+import org.apache.rocketmq.streams.common.context.IMessage;
+import org.apache.rocketmq.streams.common.context.Message;
+import org.apache.rocketmq.streams.common.interfaces.IStreamOperator;
+import org.apache.rocketmq.streams.common.utils.DateUtil;
+import org.apache.rocketmq.streams.common.utils.MapKeyUtil;
+import org.junit.Test;
+
+public class RocketMQChannelTest extends AbstractChannelTest {
+ private static final String WINDOW_TASK_PROPERTY_KEY_PREFIX = "dipper.window.shuffle.rocketmq.dispatch.channel.";
+
+ @Test
+ public void testSource() throws InterruptedException {
+ ISource source = createSource();
+ source.start(new IStreamOperator() {
+ @Override
+ public Object doMessage(IMessage iMessage, AbstractContext context) {
+ System.out.println(iMessage.getMessageBody().toJSONString());
+ return null;
+ }
+ });
+
+ Thread.sleep(100000000l);
+ }
+
+ @Test
+ public void testSink() {
+ ISink sink = createSink();
+ sink.setBatchSize(1);
+ for (int i = 0; i < 1000; i++) {
+ sink.batchSave(createMsg());
+ }
+ }
+
+ @Test
+ public void testGetSplit() {
+ AbstractSupportShuffleSink sink = (AbstractSupportShuffleSink)createSink();
+ List<ISplit> splits = sink.getSplitList();
+ System.out.println(splits.size());
+ }
+
+ // @Test
+ // public void testCreateChannel() throws InterruptedException {
+ // RocketMQChannelBuilder builder = new RocketMQChannelBuilder();
+ // IChannel consumer = builder.createChannel(getWindowNameSpace(),getWindowName(), createChannelProperties(), null);
+ // IChannel producer = builder.createChannel(getWindowNameSpace(),getWindowName(), createChannelProperties(), null);
+ // consumer.start(new IMessageProcssor() {
+ // @Override
+ // public Object doMessage(IMessage message, AbstractContext context) {
+ // System.out.println(message.getMessageBody().toJSONString());
+ // return null;
+ // }
+ // });((AbstractBatchMessageChannel)producer).getQueueList();
+ //
+ // List<ChannelQueue> queueList = ((AbstractBatchMessageChannel)producer).getQueueList();
+ // for (int i=0; i<100000; i++) {
+ //
+ // producer.batchSave(createMsg(queueList.get(0)));
+ // }
+ // producer.flush();
+ //
+ // Thread.sleep(10000000l);
+ // }
+
+ public List<IMessage> createMsg() {
+ JSONObject obj = new JSONObject();
+ obj.put("test", "11111");
+ obj.put("time", DateUtil.format(new Date()));
+ IMessage message = new Message(obj);
+ List<IMessage> msgs = new ArrayList<>();
+ msgs.add(message);
+ return msgs;
+ }
+
+ protected Properties createChannelProperties() {
+ Properties properties = new Properties();
+ // properties = PropertiesUtils.getResourceProperties(this.getClass(), "com/aliyun/yundun/dipper/channel/adapter/dipper.properties");
+ Iterator<Map.Entry<Object, Object>> it = ComponentCreator.getProperties().entrySet().iterator();
+ while (it.hasNext()) {
+ Map.Entry<Object, Object> entry = it.next();
+ String key = (String)entry.getKey();
+ String value = (String)entry.getValue();
+ if (key.startsWith(WINDOW_TASK_PROPERTY_KEY_PREFIX)) {
+ String channelKey = key.replace(WINDOW_TASK_PROPERTY_KEY_PREFIX, "");
+ properties.put(channelKey, value);
+ }
+
+ }
+ String dynamicProperty = properties.getProperty("dynamic.property");
+ if (dynamicProperty != null) {
+ String namespace = this.getWindowNameSpace();
+ String name = this.getWindowName();
+ String startTime = this.getStartTime();
+ String endTime = this.getEndTime();
+ String startTimeStr = startTime.replace("-", "").replace(" ", "").replace(":", "");
+ String endTimeStr = endTime.replace("-", "").replace(" ", "").replace(":", "");
+ String dynamicPropertyValue = MapKeyUtil.createKeyBySign("_", namespace, name, startTimeStr + "",
+ endTimeStr + "");
+ dynamicPropertyValue = dynamicPropertyValue.replaceAll("\\.", "_");
+ String[] mutilPropertys = dynamicProperty.split(",");
+ String groupName = MapKeyUtil.createKeyBySign("_", namespace, name).replaceAll("\\.", "_");
+ for (String properyKey : mutilPropertys) {
+ if (properyKey.equals("group")) {
+ properties.put(properyKey, groupName);
+ } else {
+ properties.put(properyKey, dynamicPropertyValue);
+ }
+
+ }
+ }
+ return properties;
+ }
+
+ public String getWindowNameSpace() {
+ return "windowNameSpace";
+ }
+
+ public String getWindowName() {
+ return "windowName";
+ }
+
+ public String getStartTime() {
+ return "startTime";
+ }
+
+ public String getEndTime() {
+ return "endTime";
+ }
+
+ @Override
+ protected ISource createSource() {
+ RocketMQChannelBuilder builder = new RocketMQChannelBuilder();
+ ISource rocketMQSource = builder.createSource(getWindowNameSpace(), getWindowName(), createChannelProperties(), null);
+ // RocketMQSource rocketMQSource = new RocketMQSource();
+ // rocketMQSource.setTopic("TOPIC_DIPPER_WINDOW_STATISTICS");
+ // rocketMQSource.setTags("test");
+ // rocketMQSource.setAccessKey();
+ return rocketMQSource;
+ }
+
+ @Override
+ protected ISink createSink() {
+ RocketMQChannelBuilder builder = new RocketMQChannelBuilder();
+ ISink sink = builder.createSink(getWindowNameSpace(), getWindowName(), createChannelProperties(), null);
+ return sink;
+ }
+
+}
diff --git a/rocketmq-streams-channel-rocketmq/target/classes/META-INF/services/org.apache.rocketmq.streams.common.channel.builder.IChannelBuilder b/rocketmq-streams-channel-rocketmq/target/classes/META-INF/services/org.apache.rocketmq.streams.common.channel.builder.IChannelBuilder
new file mode 100644
index 0000000..fb9e053
--- /dev/null
+++ b/rocketmq-streams-channel-rocketmq/target/classes/META-INF/services/org.apache.rocketmq.streams.common.channel.builder.IChannelBuilder
@@ -0,0 +1 @@
+org.apache.rocketmq.streams.RocketMQChannelBuilder
diff --git a/rocketmq-streams-channel-rocketmq/target/classes/org/apache/rocketmq/streams/RocketMQChannelBuilder.class b/rocketmq-streams-channel-rocketmq/target/classes/org/apache/rocketmq/streams/RocketMQChannelBuilder.class
new file mode 100644
index 0000000..bcd8f08
Binary files /dev/null and b/rocketmq-streams-channel-rocketmq/target/classes/org/apache/rocketmq/streams/RocketMQChannelBuilder.class differ
diff --git a/rocketmq-streams-channel-rocketmq/target/classes/org/apache/rocketmq/streams/RocketMQOffset.class b/rocketmq-streams-channel-rocketmq/target/classes/org/apache/rocketmq/streams/RocketMQOffset.class
new file mode 100644
index 0000000..b1f45cf
Binary files /dev/null and b/rocketmq-streams-channel-rocketmq/target/classes/org/apache/rocketmq/streams/RocketMQOffset.class differ
diff --git a/rocketmq-streams-channel-rocketmq/target/classes/org/apache/rocketmq/streams/queue/RocketMQMessageQueue.class b/rocketmq-streams-channel-rocketmq/target/classes/org/apache/rocketmq/streams/queue/RocketMQMessageQueue.class
new file mode 100644
index 0000000..a9c8a39
Binary files /dev/null and b/rocketmq-streams-channel-rocketmq/target/classes/org/apache/rocketmq/streams/queue/RocketMQMessageQueue.class differ
diff --git a/rocketmq-streams-channel-rocketmq/target/classes/org/apache/rocketmq/streams/sink/RocketMQSink$1.class b/rocketmq-streams-channel-rocketmq/target/classes/org/apache/rocketmq/streams/sink/RocketMQSink$1.class
new file mode 100644
index 0000000..b2f09df
Binary files /dev/null and b/rocketmq-streams-channel-rocketmq/target/classes/org/apache/rocketmq/streams/sink/RocketMQSink$1.class differ
diff --git a/rocketmq-streams-channel-rocketmq/target/classes/org/apache/rocketmq/streams/sink/RocketMQSink.class b/rocketmq-streams-channel-rocketmq/target/classes/org/apache/rocketmq/streams/sink/RocketMQSink.class
new file mode 100644
index 0000000..12fbfb1
Binary files /dev/null and b/rocketmq-streams-channel-rocketmq/target/classes/org/apache/rocketmq/streams/sink/RocketMQSink.class differ
diff --git a/rocketmq-streams-channel-rocketmq/target/classes/org/apache/rocketmq/streams/source/RocketMQSource$1.class b/rocketmq-streams-channel-rocketmq/target/classes/org/apache/rocketmq/streams/source/RocketMQSource$1.class
new file mode 100644
index 0000000..e9fe35c
Binary files /dev/null and b/rocketmq-streams-channel-rocketmq/target/classes/org/apache/rocketmq/streams/source/RocketMQSource$1.class differ
diff --git a/rocketmq-streams-channel-rocketmq/target/classes/org/apache/rocketmq/streams/source/RocketMQSource.class b/rocketmq-streams-channel-rocketmq/target/classes/org/apache/rocketmq/streams/source/RocketMQSource.class
new file mode 100644
index 0000000..e186384
Binary files /dev/null and b/rocketmq-streams-channel-rocketmq/target/classes/org/apache/rocketmq/streams/source/RocketMQSource.class differ
diff --git a/rocketmq-streams-channel-rocketmq/target/maven-archiver/pom.properties b/rocketmq-streams-channel-rocketmq/target/maven-archiver/pom.properties
new file mode 100644
index 0000000..a605b53
--- /dev/null
+++ b/rocketmq-streams-channel-rocketmq/target/maven-archiver/pom.properties
@@ -0,0 +1,5 @@
+#Generated by Maven
+#Fri Jul 30 11:10:37 CST 2021
+version=2.0.0-SNAPSHOT
+groupId=org.apache.rocketmq
+artifactId=rocketmq-streams-channel-rocketmq
diff --git a/rocketmq-streams-channel-rocketmq/target/rocketmq-streams-channel-rocketmq-2.0.0-SNAPSHOT-sources.jar b/rocketmq-streams-channel-rocketmq/target/rocketmq-streams-channel-rocketmq-2.0.0-SNAPSHOT-sources.jar
new file mode 100644
index 0000000..6a2ed7f
Binary files /dev/null and b/rocketmq-streams-channel-rocketmq/target/rocketmq-streams-channel-rocketmq-2.0.0-SNAPSHOT-sources.jar differ
diff --git a/rocketmq-streams-channel-rocketmq/target/rocketmq-streams-channel-rocketmq-2.0.0-SNAPSHOT.jar b/rocketmq-streams-channel-rocketmq/target/rocketmq-streams-channel-rocketmq-2.0.0-SNAPSHOT.jar
new file mode 100644
index 0000000..333a58d
Binary files /dev/null and b/rocketmq-streams-channel-rocketmq/target/rocketmq-streams-channel-rocketmq-2.0.0-SNAPSHOT.jar differ
diff --git a/rocketmq-streams-serviceloader/pom.xml b/rocketmq-streams-serviceloader/pom.xml
new file mode 100755
index 0000000..f48f581
--- /dev/null
+++ b/rocketmq-streams-serviceloader/pom.xml
@@ -0,0 +1,24 @@
+<?xml version="1.0" encoding="utf-8"?>
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams</artifactId>
+ <version>2.0.0-SNAPSHOT</version>
+ </parent>
+ <artifactId>rocketmq-streams-serviceloader</artifactId>
+ <packaging>jar</packaging>
+ <name>ROCKETMQ STREAMS :: serviceloader</name>
+ <dependencies>
+
+ <!-- 测试依赖 -->
+ <dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams-commons</artifactId>
+ </dependency>
+
+ </dependencies>
+</project>
diff --git a/rocketmq-streams-serviceloader/rocketmq-streams-serviceloader.iml b/rocketmq-streams-serviceloader/rocketmq-streams-serviceloader.iml
new file mode 100644
index 0000000..5a29414
--- /dev/null
+++ b/rocketmq-streams-serviceloader/rocketmq-streams-serviceloader.iml
@@ -0,0 +1,61 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<module org.jetbrains.idea.maven.project.MavenProjectsManager.isMavenModule="true" type="JAVA_MODULE" version="4">
+ <component name="NewModuleRootManager" LANGUAGE_LEVEL="JDK_1_8">
+ <output url="file://$MODULE_DIR$/target/classes" />
+ <output-test url="file://$MODULE_DIR$/target/test-classes" />
+ <content url="file://$MODULE_DIR$">
+ <sourceFolder url="file://$MODULE_DIR$/src/main/java" isTestSource="false" />
+ <sourceFolder url="file://$MODULE_DIR$/src/main/resources" type="java-resource" />
+ <sourceFolder url="file://$MODULE_DIR$/src/test/java" isTestSource="true" />
+ <sourceFolder url="file://$MODULE_DIR$/src/test/resources" type="java-test-resource" />
+ <excludeFolder url="file://$MODULE_DIR$/target" />
+ </content>
+ <orderEntry type="inheritedJdk" />
+ <orderEntry type="sourceFolder" forTests="false" />
+ <orderEntry type="module" module-name="rocketmq-streams-commons" />
+ <orderEntry type="library" name="Maven: org.apache.commons:commons-lang3:3.11" level="project" />
+ <orderEntry type="library" name="Maven: junit:junit:4.12" level="project" />
+ <orderEntry type="library" name="Maven: org.hamcrest:hamcrest-core:1.3" level="project" />
+ <orderEntry type="library" name="Maven: com.alibaba:fastjson:1.2.27" level="project" />
+ <orderEntry type="library" name="Maven: commons-logging:commons-logging:1.1" level="project" />
+ <orderEntry type="library" name="Maven: logkit:logkit:1.0.1" level="project" />
+ <orderEntry type="library" name="Maven: avalon-framework:avalon-framework:4.1.3" level="project" />
+ <orderEntry type="library" name="Maven: javax.servlet:servlet-api:2.3" level="project" />
+ <orderEntry type="library" name="Maven: commons-io:commons-io:2.5" level="project" />
+ <orderEntry type="library" name="Maven: log4j:log4j:1.2.17" level="project" />
+ <orderEntry type="library" name="Maven: com.google.code.gson:gson:2.8.5" level="project" />
+ <orderEntry type="library" name="Maven: com.google.auto.service:auto-service:1.0-rc5" level="project" />
+ <orderEntry type="library" name="Maven: com.google.auto.service:auto-service-annotations:1.0-rc5" level="project" />
+ <orderEntry type="library" name="Maven: com.google.auto:auto-common:0.10" level="project" />
+ <orderEntry type="library" name="Maven: com.google.guava:guava:25.1-jre" level="project" />
+ <orderEntry type="library" name="Maven: com.google.code.findbugs:jsr305:3.0.2" level="project" />
+ <orderEntry type="library" name="Maven: org.checkerframework:checker-qual:2.0.0" level="project" />
+ <orderEntry type="library" name="Maven: com.google.errorprone:error_prone_annotations:2.1.3" level="project" />
+ <orderEntry type="library" name="Maven: com.google.j2objc:j2objc-annotations:1.1" level="project" />
+ <orderEntry type="library" name="Maven: org.codehaus.mojo:animal-sniffer-annotations:1.14" level="project" />
+ <orderEntry type="library" name="Maven: com.lmax:disruptor:3.2.0" level="project" />
+ <orderEntry type="library" name="Maven: com.gliwka.hyperscan:hyperscan:5.4.0-2.0.0" level="project" />
+ <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:5.4.0-1.0.0" level="project" />
+ <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:linux-x86_64:5.4.0-1.0.0" level="project" />
+ <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:windows-x86_64:5.4.0-1.0.0" level="project" />
+ <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:macosx-x86_64:5.4.0-1.0.0" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp-platform:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-arm:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-arm64:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-x86:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-x86_64:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:ios-arm64:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:ios-x86_64:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-armhf:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-arm64:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-ppc64le:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-x86:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-x86_64:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:macosx-x86_64:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:windows-x86:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:windows-x86_64:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: net.java.dev.jna:platform:3.5.2" level="project" />
+ <orderEntry type="library" name="Maven: net.java.dev.jna:jna:3.5.2" level="project" />
+ </component>
+</module>
\ No newline at end of file
diff --git a/rocketmq-streams-serviceloader/src/main/java/org/apache/rocketmq/streams/serviceloader/IServiceLoaderService.java b/rocketmq-streams-serviceloader/src/main/java/org/apache/rocketmq/streams/serviceloader/IServiceLoaderService.java
new file mode 100644
index 0000000..1e63c38
--- /dev/null
+++ b/rocketmq-streams-serviceloader/src/main/java/org/apache/rocketmq/streams/serviceloader/IServiceLoaderService.java
@@ -0,0 +1,31 @@
+package org.apache.rocketmq.streams.serviceloader;
+
+import java.util.List;
+
+public interface IServiceLoaderService<T> {
+
+ /**
+ * 获取某个指定名称的服务对象
+ *
+ * @param name
+ * @param
+ * @return
+ */
+ T loadService(String name);
+
+ /**
+ * 获取多个实现类
+ *
+ * @param
+ * @return
+ */
+ List<T> loadService();
+
+ /**
+ * 如果forceRefresh＝＝false, refresh多次调用只会执行一次； 如果forceRefresh==true,每次调用都会做一次重新扫描
+ *
+ * @param forceRefresh
+ */
+ void refresh(boolean forceRefresh);
+
+}
diff --git a/rocketmq-streams-serviceloader/src/main/java/org/apache/rocketmq/streams/serviceloader/ServiceLoaderComponent.java b/rocketmq-streams-serviceloader/src/main/java/org/apache/rocketmq/streams/serviceloader/ServiceLoaderComponent.java
new file mode 100644
index 0000000..847c033
--- /dev/null
+++ b/rocketmq-streams-serviceloader/src/main/java/org/apache/rocketmq/streams/serviceloader/ServiceLoaderComponent.java
@@ -0,0 +1,132 @@
+package org.apache.rocketmq.streams.serviceloader;
+
+import org.apache.rocketmq.streams.common.component.ComponentCreator;
+import org.apache.rocketmq.streams.common.model.ServiceName;
+import org.apache.rocketmq.streams.serviceloader.namefinder.IServiceNameGetter;
+import org.apache.rocketmq.streams.common.component.AbstractComponent;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+
+import java.util.*;
+
+public class ServiceLoaderComponent<T> extends AbstractComponent<IServiceLoaderService<T>>
+ implements IServiceLoaderService<T> {
+ private Properties properties;
+ private Class<T> interfaceClass;
+ private Map<String, T> name2Service = new HashMap<>();
+ private List<T> serviceList = new ArrayList<T>();
+ private boolean hasRefresh = false;
+ private boolean needServieName = true;
+
+ @Override
+ public boolean stop() {
+ return true;
+ }
+
+ @Override
+ public IServiceLoaderService<T> getService() {
+ return this;
+ }
+
+ public static ServiceLoaderComponent getInstance(Class interfaceClass) {
+ ServiceLoaderComponent serviceLoaderComponent =
+ ComponentCreator.getComponent(interfaceClass.getName(), ServiceLoaderComponent.class);
+ return serviceLoaderComponent;
+ }
+
+ @Override
+ protected boolean startComponent(String interfaceClassName) {
+ try {
+ Class clazz = Class.forName(interfaceClassName);
+ this.interfaceClass = clazz;
+ refresh(false);
+ } catch (ClassNotFoundException e) {
+ throw new RuntimeException("class not found " + interfaceClassName, e);
+ }
+ return true;
+ }
+
+ @Override
+ protected boolean initProperties(Properties properties) {
+ this.properties = properties;
+ return true;
+ }
+
+ @Override
+ public T loadService(String serviceName) {
+ if (!this.hasRefresh) {
+ refresh(false);
+ }
+ return (T)this.name2Service.get(serviceName);
+ }
+
+ @Override
+ public List<T> loadService() {
+ if (!this.hasRefresh) {
+ refresh(false);
+ }
+ return serviceList;
+ }
+
+ @Override
+ public void refresh(boolean forceRefresh) {
+ if (!forceRefresh && hasRefresh) {
+ return;
+ }
+ synchronized (this) {
+ if (!forceRefresh && hasRefresh) {
+ return;
+ }
+ Map<String, T> name2Service = new HashMap<>();
+ Iterable<T> iterable = ServiceLoader.load(interfaceClass);
+ List<T> allService = new ArrayList<>();
+ for (T t : iterable) {
+ if (needServieName) {
+ List<String> serviceNames = loadServiceName(t);
+ if (serviceNames == null) {
+ name2Service.put(t.getClass().getSimpleName(), t);
+ } else {
+ for (String serviceName : serviceNames) {
+ name2Service.put(serviceName, t);
+ }
+ }
+
+ }
+ allService.add(t);
+ }
+ this.name2Service = name2Service;
+ this.hasRefresh = true;
+ this.serviceList = allService;
+ }
+ }
+
+ static ServiceLoaderComponent nameLoaderComponent = new ServiceLoaderComponent();
+
+ static {
+ nameLoaderComponent.init();
+ nameLoaderComponent.needServieName = true;
+ nameLoaderComponent.startComponent(IServiceNameGetter.class.getName());
+ }
+
+ protected List<String> loadServiceName(T t) {
+ List<String> serviceNames = new ArrayList();
+ Class tClass = t.getClass();
+ String serviceName = properties.getProperty(tClass.getName());
+ if (properties != null && StringUtil.isNotEmpty(serviceName)) {
+
+ serviceNames.add(serviceName);
+ return serviceNames;
+ }
+ ServiceName annotation = (ServiceName)tClass.getAnnotation(ServiceName.class);
+ if (annotation == null) {
+ return null;
+ }
+ if (StringUtil.isNotEmpty(annotation.value())) {
+ serviceNames.add(annotation.value());
+ }
+ if (StringUtil.isNotEmpty(annotation.aliasName())) {
+ serviceNames.add(annotation.aliasName());
+ }
+
+ return serviceNames;
+ }
+}
diff --git a/rocketmq-streams-serviceloader/src/main/java/org/apache/rocketmq/streams/serviceloader/namefinder/IServiceNameGetter.java b/rocketmq-streams-serviceloader/src/main/java/org/apache/rocketmq/streams/serviceloader/namefinder/IServiceNameGetter.java
new file mode 100644
index 0000000..2753b5b
--- /dev/null
+++ b/rocketmq-streams-serviceloader/src/main/java/org/apache/rocketmq/streams/serviceloader/namefinder/IServiceNameGetter.java
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.serviceloader.namefinder;
+
+/**
+ * 获取service
+ */
+public interface IServiceNameGetter {
+ /**
+ * 根据class获取class的name
+ *
+ * @param clazz
+ * @return
+ */
+ String getServiceName(Class clazz);
+}
diff --git a/rocketmq-streams-serviceloader/src/main/java/org/apache/rocketmq/streams/serviceloader/namefinder/impl/AnnotationServiceNameGetter.java b/rocketmq-streams-serviceloader/src/main/java/org/apache/rocketmq/streams/serviceloader/namefinder/impl/AnnotationServiceNameGetter.java
new file mode 100644
index 0000000..803e800
--- /dev/null
+++ b/rocketmq-streams-serviceloader/src/main/java/org/apache/rocketmq/streams/serviceloader/namefinder/impl/AnnotationServiceNameGetter.java
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.serviceloader.namefinder.impl;
+
+import org.apache.rocketmq.streams.common.model.ServiceName;
+import org.apache.rocketmq.streams.serviceloader.namefinder.IServiceNameGetter;
+import com.google.auto.service.AutoService;
+
+/**
+ * 根据class的标注获取servicename
+ */
+@AutoService(IServiceNameGetter.class)
+@ServiceName(AnnotationServiceNameGetter.SERVICE_NAME)
+public class AnnotationServiceNameGetter implements IServiceNameGetter {
+ public static final String SERVICE_NAME = "SERVICE_ANNOTATION";
+
+ @Override
+ public String getServiceName(Class clazz) {
+ ServiceName annotation = (ServiceName)clazz.getAnnotation(ServiceName.class);
+ if (annotation == null) {
+ return null;
+ }
+ return annotation.value();
+ }
+}
diff --git a/rocketmq-streams-serviceloader/src/main/java/org/apache/rocketmq/streams/serviceloader/utils/FileUtil.java b/rocketmq-streams-serviceloader/src/main/java/org/apache/rocketmq/streams/serviceloader/utils/FileUtil.java
new file mode 100644
index 0000000..cc2efaf
--- /dev/null
+++ b/rocketmq-streams-serviceloader/src/main/java/org/apache/rocketmq/streams/serviceloader/utils/FileUtil.java
@@ -0,0 +1,73 @@
+package org.apache.rocketmq.streams.serviceloader.utils;
+
+import com.google.common.io.Closer;
+
+import java.io.*;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Set;
+
+import static com.google.common.base.Charsets.UTF_8;
+
+public class FileUtil extends org.apache.rocketmq.streams.common.utils.FileUtil {
+ public static final String SERVICES_PATH = "META-INF/services";
+
+ /**
+ * Returns an absolute path to a service file given the class name of the service.
+ *
+ * @param serviceName not {@code null}
+ * @return SERVICES_PATH + serviceName
+ */
+ static String getPath(String serviceName) {
+ return SERVICES_PATH + "/" + serviceName;
+ }
+
+ /**
+ * Reads the set of service classes from a service file.
+ *
+ * @param input not {@code null}. Closed after use.
+ * @return a not {@code null Set} of service class names.
+ * @throws IOException
+ */
+ public static Set<String> readServiceFile(InputStream input) throws IOException {
+ HashSet<String> serviceClasses = new HashSet<String>();
+ Closer closer = Closer.create();
+ try {
+ // TODO(gak): use CharStreams
+ BufferedReader r = closer.register(new BufferedReader(new InputStreamReader(input, UTF_8)));
+ String line;
+ while ((line = r.readLine()) != null) {
+ int commentStart = line.indexOf('#');
+ if (commentStart >= 0) {
+ line = line.substring(0, commentStart);
+ }
+ line = line.trim();
+ if (!line.isEmpty()) {
+ serviceClasses.add(line);
+ }
+ }
+ return serviceClasses;
+ } catch (Throwable t) {
+ throw closer.rethrow(t);
+ } finally {
+ closer.close();
+ }
+ }
+
+ /**
+ * Writes the set of service class names to a service file.
+ *
+ * @param output not {@code null}. Not closed after use.
+ * @param services a not {@code null Collection} of service class names.
+ * @throws IOException
+ */
+ public static void writeServiceFile(Collection<String> services, OutputStream output) throws IOException {
+ BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(output, UTF_8));
+ for (String service : services) {
+ writer.write(service);
+ writer.newLine();
+ }
+ writer.flush();
+ }
+}
+
diff --git a/rocketmq-streams-serviceloader/src/main/resources/component/ServiceLoaderComponent.properties b/rocketmq-streams-serviceloader/src/main/resources/component/ServiceLoaderComponent.properties
new file mode 100644
index 0000000..0f1b2bb
--- /dev/null
+++ b/rocketmq-streams-serviceloader/src/main/resources/component/ServiceLoaderComponent.properties
@@ -0,0 +1 @@
+#serviceName=class.getName()
\ No newline at end of file
diff --git a/rocketmq-streams-serviceloader/src/test/java/org/apache/rocketmq/streams/serviceloader/ServiceLoaderComponentTest.java b/rocketmq-streams-serviceloader/src/test/java/org/apache/rocketmq/streams/serviceloader/ServiceLoaderComponentTest.java
new file mode 100644
index 0000000..b68ea36
--- /dev/null
+++ b/rocketmq-streams-serviceloader/src/test/java/org/apache/rocketmq/streams/serviceloader/ServiceLoaderComponentTest.java
@@ -0,0 +1,17 @@
+package org.apache.rocketmq.streams.serviceloader;
+
+import org.apache.rocketmq.streams.serviceloader.namefinder.IServiceNameGetter;
+import org.apache.rocketmq.streams.serviceloader.namefinder.impl.AnnotationServiceNameGetter;
+import org.junit.Test;
+
+import static junit.framework.TestCase.assertTrue;
+
+public class ServiceLoaderComponentTest {
+
+ @Test
+ public void testLoadService() {
+ ServiceLoaderComponent serviceLoaderComponent = ServiceLoaderComponent.getInstance(IServiceNameGetter.class);
+ AnnotationServiceNameGetter getter = (AnnotationServiceNameGetter)serviceLoaderComponent.getService().loadService(AnnotationServiceNameGetter.SERVICE_NAME);
+ assertTrue(getter != null && AnnotationServiceNameGetter.class.isInstance(getter));
+ }
+}
diff --git a/rocketmq-streams-serviceloader/src/test/resources/log4j.xml b/rocketmq-streams-serviceloader/src/test/resources/log4j.xml
new file mode 100755
index 0000000..7812fe7
--- /dev/null
+++ b/rocketmq-streams-serviceloader/src/test/resources/log4j.xml
@@ -0,0 +1,20 @@
+<?xml version="1.0" encoding="UTF-8" ?>
+<!DOCTYPE log4j:configuration SYSTEM "http://toolkit.alibaba-inc.com/dtd/log4j/log4j.dtd">
+<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
+
+ <appender name="Console" class="org.apache.log4j.ConsoleAppender">
+ <layout class="org.apache.log4j.PatternLayout">
+ <param name="ConversionPattern" value="%d{ISO8601} %l [%t] %-5p - %m%n%n"/>
+ </layout>
+ <filter class="org.apache.log4j.varia.LevelRangeFilter">
+ <param name="LevelMin" value="INFO"/>
+ <param name="LevelMax" value="ERROR"/>
+ </filter>
+ </appender>
+
+ <root>
+ <priority value="INFO"/>
+ <appender-ref ref="Console"/>
+ </root>
+
+</log4j:configuration>
\ No newline at end of file
diff --git a/rocketmq-streams-serviceloader/target/classes/META-INF/services/org.apache.rocketmq.streams.serviceloader.namefinder.IServiceNameGetter b/rocketmq-streams-serviceloader/target/classes/META-INF/services/org.apache.rocketmq.streams.serviceloader.namefinder.IServiceNameGetter
new file mode 100644
index 0000000..30fdc46
--- /dev/null
+++ b/rocketmq-streams-serviceloader/target/classes/META-INF/services/org.apache.rocketmq.streams.serviceloader.namefinder.IServiceNameGetter
@@ -0,0 +1 @@
+org.apache.rocketmq.streams.serviceloader.namefinder.impl.AnnotationServiceNameGetter
diff --git a/rocketmq-streams-serviceloader/target/classes/component/ServiceLoaderComponent.properties b/rocketmq-streams-serviceloader/target/classes/component/ServiceLoaderComponent.properties
new file mode 100644
index 0000000..0f1b2bb
--- /dev/null
+++ b/rocketmq-streams-serviceloader/target/classes/component/ServiceLoaderComponent.properties
@@ -0,0 +1 @@
+#serviceName=class.getName()
\ No newline at end of file
diff --git a/rocketmq-streams-serviceloader/target/classes/org/apache/rocketmq/streams/serviceloader/IServiceLoaderService.class b/rocketmq-streams-serviceloader/target/classes/org/apache/rocketmq/streams/serviceloader/IServiceLoaderService.class
new file mode 100644
index 0000000..bb8c29e
Binary files /dev/null and b/rocketmq-streams-serviceloader/target/classes/org/apache/rocketmq/streams/serviceloader/IServiceLoaderService.class differ
diff --git a/rocketmq-streams-serviceloader/target/classes/org/apache/rocketmq/streams/serviceloader/ServiceLoaderComponent.class b/rocketmq-streams-serviceloader/target/classes/org/apache/rocketmq/streams/serviceloader/ServiceLoaderComponent.class
new file mode 100644
index 0000000..3831e75
Binary files /dev/null and b/rocketmq-streams-serviceloader/target/classes/org/apache/rocketmq/streams/serviceloader/ServiceLoaderComponent.class differ
diff --git a/rocketmq-streams-serviceloader/target/classes/org/apache/rocketmq/streams/serviceloader/namefinder/IServiceNameGetter.class b/rocketmq-streams-serviceloader/target/classes/org/apache/rocketmq/streams/serviceloader/namefinder/IServiceNameGetter.class
new file mode 100644
index 0000000..84de58a
Binary files /dev/null and b/rocketmq-streams-serviceloader/target/classes/org/apache/rocketmq/streams/serviceloader/namefinder/IServiceNameGetter.class differ
diff --git a/rocketmq-streams-serviceloader/target/classes/org/apache/rocketmq/streams/serviceloader/namefinder/impl/AnnotationServiceNameGetter.class b/rocketmq-streams-serviceloader/target/classes/org/apache/rocketmq/streams/serviceloader/namefinder/impl/AnnotationServiceNameGetter.class
new file mode 100644
index 0000000..193c479
Binary files /dev/null and b/rocketmq-streams-serviceloader/target/classes/org/apache/rocketmq/streams/serviceloader/namefinder/impl/AnnotationServiceNameGetter.class differ
diff --git a/rocketmq-streams-serviceloader/target/classes/org/apache/rocketmq/streams/serviceloader/utils/FileUtil.class b/rocketmq-streams-serviceloader/target/classes/org/apache/rocketmq/streams/serviceloader/utils/FileUtil.class
new file mode 100644
index 0000000..e3b051c
Binary files /dev/null and b/rocketmq-streams-serviceloader/target/classes/org/apache/rocketmq/streams/serviceloader/utils/FileUtil.class differ
diff --git a/rocketmq-streams-serviceloader/target/maven-archiver/pom.properties b/rocketmq-streams-serviceloader/target/maven-archiver/pom.properties
new file mode 100644
index 0000000..037c58d
--- /dev/null
+++ b/rocketmq-streams-serviceloader/target/maven-archiver/pom.properties
@@ -0,0 +1,5 @@
+#Generated by Maven
+#Fri Jul 30 11:10:32 CST 2021
+version=2.0.0-SNAPSHOT
+groupId=org.apache.rocketmq
+artifactId=rocketmq-streams-serviceloader
diff --git a/rocketmq-streams-serviceloader/target/rocketmq-streams-serviceloader-2.0.0-SNAPSHOT-sources.jar b/rocketmq-streams-serviceloader/target/rocketmq-streams-serviceloader-2.0.0-SNAPSHOT-sources.jar
new file mode 100644
index 0000000..1695e65
Binary files /dev/null and b/rocketmq-streams-serviceloader/target/rocketmq-streams-serviceloader-2.0.0-SNAPSHOT-sources.jar differ
diff --git a/rocketmq-streams-serviceloader/target/rocketmq-streams-serviceloader-2.0.0-SNAPSHOT.jar b/rocketmq-streams-serviceloader/target/rocketmq-streams-serviceloader-2.0.0-SNAPSHOT.jar
new file mode 100644
index 0000000..9dc634b
Binary files /dev/null and b/rocketmq-streams-serviceloader/target/rocketmq-streams-serviceloader-2.0.0-SNAPSHOT.jar differ
diff --git a/rocketmq-streams-serviceloader/target/test-classes/log4j.xml b/rocketmq-streams-serviceloader/target/test-classes/log4j.xml
new file mode 100755
index 0000000..7812fe7
--- /dev/null
+++ b/rocketmq-streams-serviceloader/target/test-classes/log4j.xml
@@ -0,0 +1,20 @@
+<?xml version="1.0" encoding="UTF-8" ?>
+<!DOCTYPE log4j:configuration SYSTEM "http://toolkit.alibaba-inc.com/dtd/log4j/log4j.dtd">
+<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
+
+ <appender name="Console" class="org.apache.log4j.ConsoleAppender">
+ <layout class="org.apache.log4j.PatternLayout">
+ <param name="ConversionPattern" value="%d{ISO8601} %l [%t] %-5p - %m%n%n"/>
+ </layout>
+ <filter class="org.apache.log4j.varia.LevelRangeFilter">
+ <param name="LevelMin" value="INFO"/>
+ <param name="LevelMax" value="ERROR"/>
+ </filter>
+ </appender>
+
+ <root>
+ <priority value="INFO"/>
+ <appender-ref ref="Console"/>
+ </root>
+
+</log4j:configuration>
\ No newline at end of file
diff --git a/rocketmq-streams-serviceloader/target/test-classes/org/apache/rocketmq/streams/serviceloader/ServiceLoaderComponentTest.class b/rocketmq-streams-serviceloader/target/test-classes/org/apache/rocketmq/streams/serviceloader/ServiceLoaderComponentTest.class
new file mode 100644
index 0000000..100837f
Binary files /dev/null and b/rocketmq-streams-serviceloader/target/test-classes/org/apache/rocketmq/streams/serviceloader/ServiceLoaderComponentTest.class differ
diff --git a/rocketmq-streams-window/pom.xml b/rocketmq-streams-window/pom.xml
new file mode 100755
index 0000000..8c5a709
--- /dev/null
+++ b/rocketmq-streams-window/pom.xml
@@ -0,0 +1,36 @@
+<?xml version="1.0" encoding="utf-8"?>
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams</artifactId>
+ <version>2.0.0-SNAPSHOT</version>
+ </parent>
+ <artifactId>rocketmq-streams-window</artifactId>
+ <packaging>jar</packaging>
+ <name>ROCKETMQ STREAMS :: window</name>
+ <dependencies>
+
+ <dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams-db-operator</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams-lease</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams-dim</artifactId>
+ </dependency>
+
+
+ <dependency>
+ <groupId>org.rocksdb</groupId>
+ <artifactId>rocksdbjni</artifactId>
+ </dependency>
+ </dependencies>
+</project>
diff --git a/rocketmq-streams-window/rocketmq-streams-window.iml b/rocketmq-streams-window/rocketmq-streams-window.iml
new file mode 100644
index 0000000..793fdfa
--- /dev/null
+++ b/rocketmq-streams-window/rocketmq-streams-window.iml
@@ -0,0 +1,82 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<module org.jetbrains.idea.maven.project.MavenProjectsManager.isMavenModule="true" type="JAVA_MODULE" version="4">
+ <component name="NewModuleRootManager" LANGUAGE_LEVEL="JDK_1_8">
+ <output url="file://$MODULE_DIR$/target/classes" />
+ <output-test url="file://$MODULE_DIR$/target/test-classes" />
+ <content url="file://$MODULE_DIR$">
+ <sourceFolder url="file://$MODULE_DIR$/src/main/java" isTestSource="false" />
+ <sourceFolder url="file://$MODULE_DIR$/src/main/resources" type="java-resource" />
+ <sourceFolder url="file://$MODULE_DIR$/src/test/java" isTestSource="true" />
+ <sourceFolder url="file://$MODULE_DIR$/src/test/resources" type="java-test-resource" />
+ <excludeFolder url="file://$MODULE_DIR$/target" />
+ </content>
+ <orderEntry type="inheritedJdk" />
+ <orderEntry type="sourceFolder" forTests="false" />
+ <orderEntry type="module" module-name="rocketmq-streams-db-operator" />
+ <orderEntry type="module" module-name="rocketmq-streams-configurable" />
+ <orderEntry type="module" module-name="rocketmq-streams-serviceloader" />
+ <orderEntry type="library" name="Maven: org.springframework:spring-jdbc:3.2.13.RELEASE" level="project" />
+ <orderEntry type="library" name="Maven: org.springframework:spring-beans:3.2.13.RELEASE" level="project" />
+ <orderEntry type="library" name="Maven: org.springframework:spring-core:3.2.13.RELEASE" level="project" />
+ <orderEntry type="library" name="Maven: commons-logging:commons-logging:1.1" level="project" />
+ <orderEntry type="library" name="Maven: logkit:logkit:1.0.1" level="project" />
+ <orderEntry type="library" name="Maven: avalon-framework:avalon-framework:4.1.3" level="project" />
+ <orderEntry type="library" name="Maven: javax.servlet:servlet-api:2.3" level="project" />
+ <orderEntry type="library" name="Maven: org.springframework:spring-tx:3.2.13.RELEASE" level="project" />
+ <orderEntry type="library" name="Maven: mysql:mysql-connector-java:5.1.40" level="project" />
+ <orderEntry type="module" module-name="rocketmq-streams-lease" />
+ <orderEntry type="module" module-name="rocketmq-streams-channel-db" />
+ <orderEntry type="module" module-name="rocketmq-streams-dim" />
+ <orderEntry type="module" module-name="rocketmq-streams-filter" />
+ <orderEntry type="module" module-name="rocketmq-streams-script" />
+ <orderEntry type="library" name="Maven: io.krakens:java-grok:0.1.9" level="project" />
+ <orderEntry type="library" name="Maven: org.codehaus.groovy:groovy-all:2.1.8" level="project" />
+ <orderEntry type="library" name="Maven: org.python:jython-standalone:2.7.0" level="project" />
+ <orderEntry type="module" module-name="rocketmq-streams-channel-http" />
+ <orderEntry type="module" module-name="rocketmq-streams-commons" />
+ <orderEntry type="library" name="Maven: org.apache.commons:commons-lang3:3.11" level="project" />
+ <orderEntry type="library" name="Maven: junit:junit:4.12" level="project" />
+ <orderEntry type="library" name="Maven: org.hamcrest:hamcrest-core:1.3" level="project" />
+ <orderEntry type="library" name="Maven: com.alibaba:fastjson:1.2.27" level="project" />
+ <orderEntry type="library" name="Maven: log4j:log4j:1.2.17" level="project" />
+ <orderEntry type="library" name="Maven: com.google.code.gson:gson:2.8.5" level="project" />
+ <orderEntry type="library" name="Maven: com.google.auto.service:auto-service:1.0-rc5" level="project" />
+ <orderEntry type="library" name="Maven: com.google.auto.service:auto-service-annotations:1.0-rc5" level="project" />
+ <orderEntry type="library" name="Maven: com.google.auto:auto-common:0.10" level="project" />
+ <orderEntry type="library" name="Maven: com.google.guava:guava:25.1-jre" level="project" />
+ <orderEntry type="library" name="Maven: com.google.code.findbugs:jsr305:3.0.2" level="project" />
+ <orderEntry type="library" name="Maven: org.checkerframework:checker-qual:2.0.0" level="project" />
+ <orderEntry type="library" name="Maven: com.google.errorprone:error_prone_annotations:2.1.3" level="project" />
+ <orderEntry type="library" name="Maven: com.google.j2objc:j2objc-annotations:1.1" level="project" />
+ <orderEntry type="library" name="Maven: org.codehaus.mojo:animal-sniffer-annotations:1.14" level="project" />
+ <orderEntry type="library" name="Maven: com.lmax:disruptor:3.2.0" level="project" />
+ <orderEntry type="library" name="Maven: com.gliwka.hyperscan:hyperscan:5.4.0-2.0.0" level="project" />
+ <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:5.4.0-1.0.0" level="project" />
+ <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:linux-x86_64:5.4.0-1.0.0" level="project" />
+ <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:windows-x86_64:5.4.0-1.0.0" level="project" />
+ <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:macosx-x86_64:5.4.0-1.0.0" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp-platform:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-arm:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-arm64:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-x86:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-x86_64:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:ios-arm64:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:ios-x86_64:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-armhf:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-arm64:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-ppc64le:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-x86:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-x86_64:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:macosx-x86_64:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:windows-x86:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:windows-x86_64:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: net.java.dev.jna:platform:3.5.2" level="project" />
+ <orderEntry type="library" name="Maven: net.java.dev.jna:jna:3.5.2" level="project" />
+ <orderEntry type="library" name="Maven: commons-io:commons-io:2.5" level="project" />
+ <orderEntry type="library" name="Maven: org.apache.httpcomponents:httpclient:4.5.2" level="project" />
+ <orderEntry type="library" name="Maven: org.apache.httpcomponents:httpcore:4.4.4" level="project" />
+ <orderEntry type="library" name="Maven: commons-codec:commons-codec:1.9" level="project" />
+ <orderEntry type="library" name="Maven: org.rocksdb:rocksdbjni:6.6.4" level="project" />
+ </component>
+</module>
\ No newline at end of file
diff --git a/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/builder/WindowBuilder.java b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/builder/WindowBuilder.java
new file mode 100644
index 0000000..62025b8
--- /dev/null
+++ b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/builder/WindowBuilder.java
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.window.builder;
+
+import org.apache.rocketmq.streams.window.operator.AbstractWindow;
+import org.apache.rocketmq.streams.window.operator.impl.WindowOperator;
+import org.apache.rocketmq.streams.common.component.ComponentCreator;
+import org.apache.rocketmq.streams.common.configure.ConfigureFileKey;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+import org.apache.rocketmq.streams.window.operator.join.JoinWindow;
+import org.apache.rocketmq.streams.window.operator.impl.OverWindow;
+
+public class WindowBuilder {
+ /**
+ * 默认窗口大小
+ */
+ public static final int DEFAULT_WINDOW_INTERVAL_SIZE_MINUTE = 5;
+ private static boolean TEST_MODE = false;
+
+ public static WindowOperator createWindow() {
+ if (!TEST_MODE) {
+ return new WindowOperator();
+ } else {
+ return new WindowOperator();
+ }
+ }
+
+ public static void openTestModel() {
+ TEST_MODE = true;
+ }
+
+ public static void closeTestModel() {
+ TEST_MODE = false;
+ }
+
+ /**
+ * 创建join的窗口对象，join的窗口大小可以配置文件配置，如果未配置用写死的默认值
+ *
+ * @return
+ */
+ public static JoinWindow createDefaultJoinWindow() {
+ JoinWindow joinWindow = new JoinWindow();
+ if (TEST_MODE) {
+ joinWindow = new JoinWindow();
+ }
+ joinWindow.setSizeInterval(getIntValue(ConfigureFileKey.DIPPER_WINDOW_JOIN_DEFAULT_ITERVA_SIZE, 5));//默认5分钟一个窗口
+ joinWindow.setRetainWindowCount(getIntValue(ConfigureFileKey.DIPPER_WINDOW_JOIN_RETAIN_WINDOW_COUNT, 6));//join的时间窗口是20分钟
+ joinWindow.setWindowType(AbstractWindow.TUMBLE_WINDOW);
+ // joinWindow.setFireDelaySecond(getIntValue(ConfigureFileKey.DIPPER_WINDOW_DEFAULT_FIRE_DELAY_SECOND,5));//延迟5分钟触发
+ joinWindow.setTimeFieldName("");
+ joinWindow.setSlideInterval(getIntValue(ConfigureFileKey.DIPPER_WINDOW_JOIN_DEFAULT_ITERVA_SIZE, 5));
+ joinWindow.setWaterMarkMinute(0);
+ joinWindow.setWindowType(AbstractWindow.TUMBLE_WINDOW);
+ return joinWindow;
+ }
+
+ public static OverWindow createOvertWindow(String groupBy, String rowNumName) {
+ OverWindow overWindow = new OverWindow();
+ overWindow.setGroupByFieldName(groupBy);
+ overWindow.setRowNumerName(rowNumName);
+ overWindow.setTimeFieldName("");
+ overWindow.setSizeInterval(getIntValue(ConfigureFileKey.DIPPER_WINDOW_OVER_DEFAULT_ITERVA_SIZE, 60));
+ overWindow.setSlideInterval(overWindow.getSizeInterval());
+ return overWindow;
+ }
+
+ /**
+ * 获取配置配置文件的值，如果配置文件为配置，则用默认值
+ *
+ * @param propertyKey
+ * @param defalutValue
+ * @return
+ */
+ public static int getIntValue(String propertyKey, int defalutValue) {
+ String value = ComponentCreator.getProperties().getProperty(propertyKey);
+ if (StringUtil.isNotEmpty(value)) {
+ return Integer.valueOf(value);
+ }
+ return defalutValue;
+ }
+}
diff --git a/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/model/FunctionExecutor.java b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/model/FunctionExecutor.java
new file mode 100644
index 0000000..a6da407
--- /dev/null
+++ b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/model/FunctionExecutor.java
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.window.model;
+
+import java.util.List;
+
+import org.apache.rocketmq.streams.common.context.IMessage;
+import org.apache.rocketmq.streams.common.interfaces.IStreamOperator;
+
+public class FunctionExecutor {
+
+ private static final String KEY = "key";
+
+ /**
+ * the computed column defined by user or system
+ */
+ private String column;
+
+ /**
+ * the executor of operator(column=function(xxx))
+ */
+ private IStreamOperator<IMessage, List<IMessage>> executor;
+
+ public FunctionExecutor(String column, IStreamOperator<IMessage, List<IMessage>> executor) {
+ this.column = column;
+ this.executor = executor;
+ }
+
+ public String getColumn() {
+ return column;
+ }
+
+ public IStreamOperator<IMessage, List<IMessage>> getExecutor() {
+ return executor;
+ }
+
+}
diff --git a/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/model/WindowCache.java b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/model/WindowCache.java
new file mode 100644
index 0000000..3ac082d
--- /dev/null
+++ b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/model/WindowCache.java
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.window.model;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import com.alibaba.fastjson.JSONArray;
+import com.alibaba.fastjson.JSONObject;
+
+import org.apache.rocketmq.streams.common.channel.sink.AbstractSink;
+import org.apache.rocketmq.streams.common.channel.source.ISource;
+import org.apache.rocketmq.streams.common.channel.split.ISplit;
+import org.apache.rocketmq.streams.common.context.Message;
+import org.apache.rocketmq.streams.common.context.IMessage;
+import org.apache.rocketmq.streams.common.channel.IChannel;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+import org.apache.rocketmq.streams.window.shuffle.ShuffleChannel;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+/**
+ * 缓存数据，flush时，刷新完成数据落盘
+ */
+public abstract class WindowCache extends
+ AbstractSink {
+
+ private static final Log LOG = LogFactory.getLog(WindowCache.class);
+
+ public static final String SPLIT_SIGN = "##";
+
+ public static final String ORIGIN_OFFSET = "origin_offset";
+
+ public static final String ORIGIN_QUEUE_ID = "origin_queue_id";
+
+ public static final String ORIGIN_MESSAGE_HEADER = "origin_message_header";
+
+ public static final String SHUFFLE_KEY = "SHUFFLE_KEY";
+
+ public static final String ORIGIN_MESSAGE_TRACE_ID = "origin_request_id";
+
+ /**
+ * 分片转发channel
+ */
+ protected transient ShuffleChannel shuffleChannel;
+
+ @Override
+ protected boolean batchInsert(List<IMessage> messageList) {
+ Map<Integer, JSONArray> shuffleMap = translateToShuffleMap(messageList);
+ if (shuffleMap != null && shuffleMap.size() > 0) {
+ Set<String> splitIds=new HashSet<>();
+ for (Map.Entry<Integer, JSONArray> entry : shuffleMap.entrySet()) {
+ ISplit split=shuffleChannel.getSplit(entry.getKey());
+ JSONObject msg=shuffleChannel.createMsg(entry.getValue(),split);
+ shuffleChannel.getProducer().batchAdd(new Message(msg),split);
+ splitIds.add(split.getQueueId());
+ }
+ shuffleChannel.getProducer().flush();
+ }
+ return true;
+ }
+
+ /**
+ * 对接收的消息按照不同shuffle key进行分组
+ *
+ * @param messages
+ * @return
+ */
+ protected Map<Integer, JSONArray> translateToShuffleMap(List<IMessage> messages) {
+ Map<Integer, JSONArray> shuffleMap = new HashMap<>();
+ for (IMessage msg : messages) {
+ if (msg.getHeader().isSystemMessage()) {
+ continue;
+ }
+ String shuffleKey = generateShuffleKey(msg);
+ if (StringUtil.isEmpty(shuffleKey)) {
+ shuffleKey="<null>";
+ LOG.debug("there is no group by value in message! " + msg.getMessageBody().toString());
+ //continue;
+ }
+ Integer index = shuffleChannel.hash(shuffleKey);
+ JSONObject body = msg.getMessageBody();
+ String offset = msg.getHeader().getOffset();
+ String queueId = msg.getHeader().getQueueId();
+
+ body.put(ORIGIN_OFFSET, offset);
+ body.put(ORIGIN_QUEUE_ID, queueId);
+ body.put(ORIGIN_MESSAGE_HEADER, JSONObject.toJSONString(msg.getHeader()));
+ body.put(ORIGIN_MESSAGE_TRACE_ID, msg.getHeader().getTraceId());
+ body.put(SHUFFLE_KEY, shuffleKey);
+
+ addPropertyToMessage(msg, body);
+
+ JSONArray jsonArray = shuffleMap.get(index);
+ if (jsonArray == null) {
+ jsonArray = new JSONArray();
+ shuffleMap.put(index, jsonArray);
+ }
+ jsonArray.add(body);
+
+ }
+ return shuffleMap;
+ }
+
+
+
+ /**
+ * 根据message生成shuffle key
+ *
+ * @param message
+ * @return
+ */
+ protected abstract String generateShuffleKey(IMessage message);
+
+ /**
+ * 如果需要额外的字段附加到shuffle前的message，通过实现这个子类增加
+ *
+ * @param oriJson
+ */
+ protected void addPropertyToMessage(IMessage oriMessage, JSONObject oriJson){
+
+ }
+
+ public ShuffleChannel getShuffleChannel() {
+ return shuffleChannel;
+ }
+
+ public void setShuffleChannel(ShuffleChannel shuffleChannel) {
+ this.shuffleChannel = shuffleChannel;
+ }
+}
diff --git a/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/model/WindowInstance.java b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/model/WindowInstance.java
new file mode 100644
index 0000000..3acefad
--- /dev/null
+++ b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/model/WindowInstance.java
@@ -0,0 +1,506 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.window.model;
+
+import java.io.Serializable;
+import java.text.ParseException;
+import java.text.SimpleDateFormat;
+import java.util.ArrayList;
+import java.util.Calendar;
+import java.util.Collection;
+import java.util.Date;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+
+import com.alibaba.fastjson.JSONObject;
+
+import org.apache.rocketmq.streams.common.utils.SQLUtil;
+import org.apache.rocketmq.streams.window.operator.AbstractWindow;
+import org.apache.rocketmq.streams.common.model.Entity;
+
+import org.apache.rocketmq.streams.db.driver.orm.ORMUtil;
+
+import org.apache.rocketmq.streams.common.context.IMessage;
+import org.apache.rocketmq.streams.common.utils.CollectionUtil;
+import org.apache.rocketmq.streams.common.utils.DateUtil;
+import org.apache.rocketmq.streams.common.utils.MapKeyUtil;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+/**
+ * 具体的窗口实例
+ */
+public class WindowInstance extends Entity implements Serializable {
+
+ protected static final Log LOG = LogFactory.getLog(WindowInstance.class);
+
+ private static final long serialVersionUID = 6893491128670330569L;
+
+ /**
+ * 窗口实例的开始时间
+ */
+ protected String startTime;
+
+ /**
+ * 窗口实例的结束时间
+ */
+ protected String endTime;
+
+ /**
+ * fire!
+ */
+ protected String fireTime;
+
+ /**
+ * 使用configName
+ */
+ protected String windowName;
+
+ protected String splitId;
+ /**
+ * namespace
+ */
+ protected String windowNameSpace;
+ protected String windowInstanceName;//默认等于窗口名，需要区分不同窗口时使用
+
+ /**
+ * splitId,windowNameSpace,windowName,windowInstanceName,windowInstanceName 数据库中存储的是MD5值
+ */
+ protected String windowInstanceSplitName;
+ /**
+ * windowInstanceId, splitId,windowNameSpace,windowName,windowInstanceName,windowInstanceName,startTime,endTime" 数据库中存储的是MD5值
+ */
+ protected String windowInstanceKey;
+
+ protected transient Boolean isNewWindowInstance = false;//当第一次创建时设置为true，否则设置为false
+
+ /**
+ * 0：待计算；1：已经计算结束；-1：已经取消;
+ */
+ protected int status = 0;
+
+ //todo 建议之后改个名字，为了测试方便，暂时用这个字段
+ protected Integer version = 1;//用于标识channel的状态，如果值是1，表示第一次消费，否则是第二次消费
+
+ private static SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
+
+ private static final String SEPARATOR = "_";
+
+ protected transient Long lastMaxUpdateTime;//last max update time for session window
+
+ /**
+ * 创建window instance的唯一ID
+ *
+ * @return
+ */
+ public String createWindowInstanceId() {
+ return MapKeyUtil.createKey(splitId, windowNameSpace, windowName, windowInstanceName, startTime, endTime);
+ }
+
+ /**
+ * 创建window instance对象列表
+ *
+ * @param window
+ * @param startAndEndTimeList
+ * @param fireTimeList
+ * @return
+ */
+ public static List<WindowInstance> createWindowInstances(AbstractWindow window,
+ List<Pair<String, String>> startAndEndTimeList, List<String> fireTimeList, String queueId) {
+ List<WindowInstance> lostInstanceList = new ArrayList<>();
+ for (int index = 0; index < startAndEndTimeList.size(); index++) {
+ Pair<String, String> pair = startAndEndTimeList.get(index);
+ WindowInstance windowInstance = window.createWindowInstance(pair.getLeft(), pair.getRight(), fireTimeList.get(index), queueId);
+
+ lostInstanceList.add(windowInstance);
+ }
+ return lostInstanceList;
+ }
+
+ public String createWindowInstancePartitionId() {
+ return StringUtil.createMD5Str(MapKeyUtil.createKey(windowNameSpace, windowName, windowInstanceName, startTime, endTime, splitId));
+ }
+ //
+ ///**
+ // * 批量查询窗口实例，对于不存在的窗口实例创建并插入数据库
+ // */
+ //public static List<WindowInstance> queryAndCreateWindowInstances(AbstractWindow window,
+ // List<Pair<String, String>> startAndEndTimeList, List<String> fireTimeList) {
+ // List<WindowInstance> windowInstanceList = queryWindowInstances(window, startAndEndTimeList, fireTimeList);
+ // if (windowInstanceList == null) {
+ // //TODO db exception
+ // return new ArrayList<>();
+ // }
+ // List<Pair<String, String>> lostPairList = new ArrayList<>();
+ // List<String> lostFireList = new ArrayList<>();
+ // for (int index = 0; index < startAndEndTimeList.size(); index++) {
+ // Pair<String, String> pair = startAndEndTimeList.get(index);
+ // String fireTime = fireTimeList.get(index);
+ // List<WindowInstance> resultList = windowInstanceList.stream().filter(
+ // instance -> instance.startTime.equals(pair.getLeft()) && instance.endTime.equals(pair.getRight()) && instance.getFireTime().equals(fireTime))
+ // .collect(
+ // Collectors.toList());
+ // if (CollectionUtil.isEmpty(resultList)) {
+ // lostPairList.add(pair);
+ // lostFireList.add(fireTimeList.get(index));
+ // }
+ // }
+ // List<WindowInstance> lostInstanceList = createWindowInstances(window,lostPairList,lostFireList);
+ // try {
+ // //插入数据库前重写windowInstanceId
+ //
+ // ORMUtil.batchIgnoreInto(lostInstanceList);
+ // } catch (Exception e) {
+ // LOG.error("failed in creating new window instances", e);
+ // }
+ // windowInstanceList.addAll(lostInstanceList);
+ // return windowInstanceList;
+ //}
+
+ ///**
+ // * 在内存中没有找到instance，查询数据库
+ // */
+ //public static WindowInstance queryWindowInstance(String dbWindowInstanceId) {
+ // String sql = "select * from " + ORMUtil.getTableName(WindowInstance.class)
+ // + " where status = 0 and window_instance_key = '" + dbWindowInstanceId + "';";
+ // WindowInstance windowInstance = ORMUtil.queryForObject(sql, null, WindowInstance.class);
+ // return windowInstance;
+ //}
+
+ //public static List<WindowInstance> queryWindowInstances(AbstractWindow window, List<Pair<String, String>> windowTimeList,
+ // List<String> fireTimeList) {
+ // if (CollectionUtil.isEmpty(windowTimeList)) {
+ // return new ArrayList<>();
+ // }
+ // String sql = "select * from " + ORMUtil.getTableName(WindowInstance.class);
+ // StringBuilder builder = new StringBuilder();
+ // builder.append(" where status = 0 and window_instance_key in (");
+ // for (int index = 0; index < windowTimeList.size(); index++) {
+ // Pair<String, String> windowTime = windowTimeList.get(index);
+ // String instanceKey = window.createWindowInstance(windowTime.getLeft(), windowTime.getRight(), fireTimeList.get(index)).getWindowInstanceKey();
+ // builder.append("'").append(instanceKey).append("'");
+ // if (index != windowTimeList.size() - 1) {
+ // builder.append(",");
+ // }
+ // }
+ // builder.append(");");
+ // sql += builder.toString();
+ // try {
+ // List<WindowInstance> dbWindowInstanceList = ORMUtil.queryForList(sql, null, WindowInstance.class);
+ // return dbWindowInstanceList;
+ // } catch (Exception e) {
+ // LOG.error("failed in getting window instances batch", e);
+ // }
+ // return null;
+ //}
+
+ /**
+ * 触发时间比lastTime小的所有的有效的instance
+ *
+ * @param
+ * @return
+ */
+ public static List<WindowInstance> queryAllWindowInstance(String lastTime, AbstractWindow window,
+ Collection<String> splitIds) {
+ if (window.isLocalStorageOnly() || splitIds == null) {
+ return null;
+ }
+ List<String> splitIdList = new ArrayList<>();
+ splitIdList.addAll(splitIds);
+ String[] splitNames = new String[splitIds.size()];
+ for (int i = 0; i < splitNames.length; i++) {
+ splitNames[i] = MapKeyUtil.createKey(window.getNameSpace(), window.getConfigureName(), splitIdList.get(i));
+ splitNames[i] = StringUtil.createMD5Str(splitNames[i]);
+ }
+ String sql = "select * from window_instance where "
+ + " status =0 and window_instance_split_name in(" + SQLUtil.createInSql(splitNames) + ")";
+
+ List<WindowInstance> dbWindowInstanceList = null;
+ try {
+ dbWindowInstanceList = ORMUtil.queryForList(sql, null, WindowInstance.class);
+ } catch (Exception e) {
+ LOG.error("failed in getting unfired window instances", e);
+ }
+ return dbWindowInstanceList;
+ }
+
+ /**
+ * 清理window
+ *
+ * @param windowInstance
+ */
+ public static void cleanWindow(WindowInstance windowInstance) {
+ List<WindowInstance> windowInstances = new ArrayList<>();
+ windowInstances.add(windowInstance);
+ clearInstances(windowInstances);
+ }
+
+ public static void clearInstances(List<WindowInstance> windowInstances) {
+ if (CollectionUtil.isEmpty(windowInstances)) {
+ return;
+ }
+ StringBuilder deleteInstanceSql = new StringBuilder();
+ boolean isFirst = true;
+ for (WindowInstance windowInstance : windowInstances) {
+ LOG.debug("clear window instance in db, instance key: " + windowInstance.getWindowInstanceKey());
+ if (isFirst) {
+ isFirst = false;
+ } else {
+ deleteInstanceSql.append(",");
+ }
+ deleteInstanceSql.append("('" + windowInstance.getWindowInstanceKey() + "')");
+ }
+ String deleteInstanceById = "delete from " + ORMUtil.getTableName(WindowInstance.class)
+ + " where (window_instance_key) in (" + deleteInstanceSql.toString() + ")";
+ ORMUtil.executeSQL(deleteInstanceById, null);
+ }
+
+ public static Long getOccurTime(AbstractWindow window, IMessage message) {
+ Long occurTime = null;
+ if (StringUtil.isEmpty(window.getTimeFieldName())) {
+ occurTime = message.getMessageBody().getLong("time");
+ if (occurTime == null) {
+ occurTime = message.getHeader().getSendTime();
+ }
+ } else {
+ try {
+ occurTime = message.getMessageBody().getLong(window.getTimeFieldName());
+ } catch (Exception e) {
+ String occurTimeString = message.getMessageBody().getString(window.getTimeFieldName());
+ try {
+ occurTime = dateFormat.parse(occurTimeString).getTime();
+ } catch (ParseException parseException) {
+ throw new RuntimeException("can not parse the time field (" + window.getTimeFieldName() + ")");
+ }
+ }
+ }
+ if (occurTime == null) {
+ throw new RuntimeException("can not parse the time field (" + window.getTimeFieldName() + ")");
+ }
+ return occurTime;
+ }
+
+ /**
+ * 查询或者创建Window的实例，滑动窗口有可能返回多个，滚动窗口返回一个
+ *
+ * @param window
+ * @param occurTime
+ * @return
+ * @Param isWindowInstance2DB 如果是秒级窗口，可能windowinstacne不必存表，只在内存保存，可以通过这个标志设置
+ */
+ public static List<WindowInstance> getOrCreateWindowInstance(AbstractWindow window, Long occurTime, int timeUnitAdjust, String queueId) {
+ int windowSlideInterval = window.getSlideInterval();
+ int windowSizeInterval = window.getSizeInterval();
+ if (windowSlideInterval == 0) {
+ windowSlideInterval = windowSizeInterval;
+ }
+ int waterMarkMinute = window.getWaterMarkMinute();
+ List<Date> windowBeginTimeList = DateUtil.getWindowBeginTime(occurTime, windowSlideInterval * timeUnitAdjust * 1000,
+ windowSizeInterval * timeUnitAdjust * 1000);
+ List<WindowInstance> instanceList = new ArrayList<>();
+ List<Pair<String, String>> lostWindowTimeList = new ArrayList<>();
+ List<String> lostFireList = new ArrayList<>();
+ long now = System.currentTimeMillis();
+ for (Date begin : windowBeginTimeList) {
+ Date end = DateUtil.addDate(TimeUnit.SECONDS, begin, windowSizeInterval * timeUnitAdjust);
+ Date fire = null;
+ if (window.getFireMode() != 0) {
+ //非正常触发模式
+ if (now - end.getTime() < 0) {
+ fire = end;
+ } else {
+ Long nowEventTime = window.getWindowMaxValueManager().updateWindowEventTime(queueId, begin.getTime());
+ List<Date> currentWindowList = DateUtil.getWindowBeginTime(
+ nowEventTime, windowSlideInterval * timeUnitAdjust * 1000,
+ windowSizeInterval * timeUnitAdjust * 1000);
+ if (!CollectionUtil.isEmpty(currentWindowList)) {
+ Date soonBegin = currentWindowList.get(currentWindowList.size() - 1);
+ Date soonEnd = DateUtil.addDate(TimeUnit.SECONDS, soonBegin,
+ windowSizeInterval * timeUnitAdjust);
+ Date soonFire = soonEnd;
+ fire = soonFire;
+ }
+ // System.out.println(DateUtil.format(fire));
+ if (fire.getTime() - end.getTime() - waterMarkMinute * timeUnitAdjust * 1000 > 0) {
+ //超过最大watermark，消息需要丢弃
+ break;
+ }
+ }
+ } else {
+ fire = DateUtil.addDate(TimeUnit.SECONDS, end, waterMarkMinute * timeUnitAdjust);
+ }
+ String startTime = DateUtil.format(begin);
+ String endTime = DateUtil.format(end);
+ String fireTime = DateUtil.format(fire);
+ String windowInstanceId = window.createWindowInstance(startTime, endTime, fireTime, queueId).createWindowInstanceId();
+ WindowInstance windowInstance = window.getWindowInstanceMap().get(windowInstanceId);
+ if (windowInstance == null) {
+ lostWindowTimeList.add(Pair.of(startTime, endTime));
+ lostFireList.add(fireTime);
+ } else {
+ windowInstance.setFireTime(fireTime);
+ instanceList.add(windowInstance);
+ }
+ }
+ List<WindowInstance> lostInstanceList = null;
+ lostInstanceList = WindowInstance.createWindowInstances(window, lostWindowTimeList, lostFireList, queueId);
+
+ instanceList.addAll(lostInstanceList);
+ for (WindowInstance windowInstance : instanceList) {
+ window.getWindowInstanceMap().putIfAbsent(windowInstance.createWindowInstanceId(), windowInstance);
+ }
+
+ return instanceList;
+ }
+
+ //public WindowInstance copy() {
+ // WindowInstance windowInstance=new WindowInstance();
+ // windowInstance.setNewWindowInstance(this.getNewWindowInstance());
+ // windowInstance.setVersion(this.version);
+ // windowInstance.setStartTime(this.startTime);
+ // windowInstance.setEndTime(this.endTime);
+ // windowInstance.setStatus(this.status);
+ // windowInstance.setWindowNameSpace(this.windowNameSpace);
+ // windowInstance.setWindowName(this.windowName);
+ // windowInstance.setFireTime(this.fireTime);
+ // windowInstance.setWindowInstanceKey(this.windowInstanceKey);
+ // windowInstance.setGmtCreate(this.gmtCreate);
+ // windowInstance.setGmtModified(this.gmtModified);
+ // return windowInstance;
+ //}
+
+ //public WindowInstance toMd5Instance() {
+ // WindowInstance instance = copy();
+ // instance.setWindowInstanceKey(StringUtil.createMD5Str(instance.getWindowInstanceKey()));
+ // return instance;
+ //}
+
+ //public WindowInstance toOriginInstance(boolean supportOutDate) {
+ // WindowInstance instance = copy();
+ // instance.setWindowInstanceKey(null);
+ // instance.createWindowInstanceId(supportOutDate);
+ // return instance;
+ //}
+
+ public String getStartTime() {
+ return startTime;
+ }
+
+ public void setStartTime(String startTime) {
+ this.startTime = startTime;
+ }
+
+ public String getEndTime() {
+ return endTime;
+ }
+
+ public void setEndTime(String endTime) {
+ this.endTime = endTime;
+ }
+
+ public String getFireTime() {
+ return fireTime;
+ }
+
+ public void setFireTime(String fireTime) {
+ this.fireTime = fireTime;
+ }
+
+ public String getWindowName() {
+ return windowName;
+ }
+
+ public void setWindowName(String windowName) {
+ this.windowName = windowName;
+ }
+
+ public String getWindowNameSpace() {
+ return windowNameSpace;
+ }
+
+ public void setWindowNameSpace(String windowNameSpace) {
+ this.windowNameSpace = windowNameSpace;
+ }
+
+ public int getStatus() {
+ return status;
+ }
+
+ public void setStatus(int status) {
+ this.status = status;
+ }
+
+ public Integer getVersion() {
+ return version;
+ }
+
+ public void setVersion(Integer version) {
+ this.version = version;
+ }
+
+ public String getWindowInstanceKey() {
+ return windowInstanceKey;
+ }
+
+ public String getWindowInstanceName() {
+ return windowInstanceName;
+ }
+
+ public void setWindowInstanceName(String windowInstanceName) {
+ this.windowInstanceName = windowInstanceName;
+ }
+
+ public void setWindowInstanceKey(String windowInstanceKey) {
+ this.windowInstanceKey = windowInstanceKey;
+ }
+
+ public Boolean isNewWindowInstance() {
+ return isNewWindowInstance;
+ }
+
+ public void setNewWindowInstance(Boolean newWindowInstance) {
+ isNewWindowInstance = newWindowInstance;
+ }
+
+ public String getSplitId() {
+ return splitId;
+ }
+
+ public void setSplitId(String splitId) {
+ this.splitId = splitId;
+ }
+
+ public String getWindowInstanceSplitName() {
+ return windowInstanceSplitName;
+ }
+
+ public void setWindowInstanceSplitName(String windowInstanceSplitName) {
+ this.windowInstanceSplitName = windowInstanceSplitName;
+ }
+
+ public Long getLastMaxUpdateTime() {
+ return lastMaxUpdateTime;
+ }
+
+ public void setLastMaxUpdateTime(Long lastMaxUpdateTime) {
+ this.lastMaxUpdateTime = lastMaxUpdateTime;
+ }
+}
diff --git a/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/offset/IWindowMaxValueManager.java b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/offset/IWindowMaxValueManager.java
new file mode 100644
index 0000000..ae73327
--- /dev/null
+++ b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/offset/IWindowMaxValueManager.java
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.window.offset;
+
+import org.apache.rocketmq.streams.window.model.WindowInstance;
+
+import java.util.Set;
+
+/**
+ * create split sequence number for window instance output result order by split sequence number when windown instance fire
+ */
+public interface IWindowMaxValueManager {
+
+ /**
+ * create split sequence number if the generator is not in memory， need load from db or other storage if instance is new ，set the split sequence number = init value
+ *
+ * @param instance
+ * @param splitId
+ * @return plus one on the current max split sequence number
+ */
+ Long incrementAndGetSplitNumber(WindowInstance instance, String splitId);
+
+ /**
+ * create split sequence number if the generator is not in memory， need load from db or other storage if instance is new ，set the split sequence number = init value
+ *
+ * @param key
+ * @return plus one on the current max split sequence number
+ */
+ Long incrementAndGetSplitNumber(String key);
+
+ /**
+ * return the max time from window receive event if (eventTime > maxEventTime) { maxEventTime=eventTime; } return maxEventTime
+ *
+ * @param eventTime
+ * @return
+ */
+ Long updateWindowEventTime(String splitId, Long eventTime);
+
+ //return the max time from window receive event
+ Long updateWindowEventTime(String splitId, String formatEventTime);
+
+ /**
+ * load mutil window instance split's max split num
+ *
+ * @param windowInstances
+ * @param split
+ */
+ void loadMaxSplitNum(Set<WindowInstance> windowInstances, String split);
+
+ String createSplitNumberKey(WindowInstance instance, String splitId);
+
+ /**
+ * load mutil window instance split's max split num
+ *
+ * @param keys
+ * @return
+ */
+ void loadMaxSplitNum(Set<String> keys);
+
+ void removeKeyPrefixFromLocalCache(Set<String> keyPrefixs);
+
+ //load window max event time
+ void loadWindowMaxEventTime(Set<String> splitId);
+
+ /**
+ * save addition WindowMaxValue
+ */
+ void flush();
+
+ void resetSplitNum(WindowInstance instance, String splitId);
+
+ void resetSplitNum(String key);
+}
diff --git a/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/offset/WindowMaxValue.java b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/offset/WindowMaxValue.java
new file mode 100644
index 0000000..27b8b05
--- /dev/null
+++ b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/offset/WindowMaxValue.java
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.window.offset;
+
+import java.util.Date;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.apache.rocketmq.streams.common.model.Entity;
+
+/**
+ * save windowintance max offset
+ */
+public class WindowMaxValue extends Entity {
+ public static long MAX_VALUE_BASE_VALUE=100000000;
+ protected String msgKey;
+ protected AtomicLong maxValue=new AtomicLong(MAX_VALUE_BASE_VALUE);
+
+ protected AtomicLong maxEventTime=new AtomicLong();//只有window需要
+
+ public WindowMaxValue(){
+ this.gmtModified=new Date();
+ this.gmtCreate=new Date();
+ }
+
+ public Long getMaxEventTime() {
+ return maxEventTime.get();
+ }
+
+ public void setMaxEventTime(Long maxEventTime) {
+ if(maxEventTime==null){
+ return;
+ }
+ this.maxEventTime.set( maxEventTime);
+ }
+
+ public String getMsgKey() {
+ return msgKey;
+ }
+
+ public void setMsgKey(String msgKey) {
+ this.msgKey = msgKey;
+ }
+
+ public Long getMaxValue() {
+ return maxValue.get();
+ }
+
+ public void setMaxValue(Long maxValue) {
+ this.maxValue.set(maxValue);
+ }
+
+ public long comareAndSet(Long eventTime){
+ if(eventTime==null){
+ return maxEventTime.get();
+ }
+ long old=maxEventTime.get();
+ if(old>=eventTime){
+ return old;
+ }
+ boolean updateSuccess=false;
+ while (!updateSuccess){
+ old=maxEventTime.get();
+ if(eventTime>old){
+ updateSuccess= maxEventTime.compareAndSet(old,eventTime);
+ } else {
+ break;
+ }
+ }
+ return maxEventTime.get();
+ }
+
+ public long incrementAndGetMaxOffset(){
+ return maxValue.incrementAndGet();
+ }
+}
diff --git a/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/offset/WindowMaxValueManager.java b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/offset/WindowMaxValueManager.java
new file mode 100644
index 0000000..b6d537c
--- /dev/null
+++ b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/offset/WindowMaxValueManager.java
@@ -0,0 +1,247 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.window.offset;
+
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.rocketmq.streams.common.utils.DateUtil;
+import org.apache.rocketmq.streams.common.utils.MapKeyUtil;
+import org.apache.rocketmq.streams.common.utils.SQLUtil;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+import org.apache.rocketmq.streams.db.driver.orm.ORMUtil;
+import org.apache.rocketmq.streams.window.model.WindowInstance;
+import org.apache.rocketmq.streams.window.operator.AbstractWindow;
+
+import static org.apache.rocketmq.streams.window.offset.WindowMaxValue.MAX_VALUE_BASE_VALUE;
+
+public class WindowMaxValueManager implements IWindowMaxValueManager {
+ protected AbstractWindow window;
+
+ protected Map<String, WindowMaxValue> windowOffsetMap=new HashMap<>();//all window offsets
+ protected List<WindowMaxValue> needUpdateWindowValues =new ArrayList<>();//new windowoffset list, need save to storage when flush
+
+ public WindowMaxValueManager(AbstractWindow window){
+ this.window=window;
+ }
+
+ @Override
+ public String createSplitNumberKey(WindowInstance instance, String splitId){
+ String key= MapKeyUtil.createKey(splitId,instance.getWindowInstanceKey());
+ return key;
+ }
+
+ @Override
+ public Long incrementAndGetSplitNumber(WindowInstance instance, String splitId) {
+ String key=createSplitNumberKey(instance,splitId);
+ WindowMaxValue windowMaxValue=queryOrCreateWindowOffset(key,true);
+ return windowMaxValue.incrementAndGetMaxOffset();
+ }
+
+ @Override
+ public Long updateWindowEventTime(String splitId, Long eventTime) {
+ String windowId=StringUtil.createMD5Str(MapKeyUtil.createKey(window.getNameSpace(),window.getConfigureName()));
+ String key=MapKeyUtil.createKey(splitId, windowId);
+ WindowMaxValue windowOffset=queryOrCreateWindowOffset(key,true);
+ return windowOffset.comareAndSet(eventTime);
+ }
+
+ @Override
+ public Long updateWindowEventTime(String splitId, String formatEventTime) {
+ if(StringUtil.isEmpty(formatEventTime)){
+ return updateWindowEventTime(splitId,(Long)null);
+ }
+ Long time= DateUtil.parseTime(formatEventTime).getTime();
+ return updateWindowEventTime(splitId,time);
+ }
+
+ @Override
+ public Long incrementAndGetSplitNumber(String key) {
+ WindowMaxValue windowOffset=queryOrCreateWindowOffset(key,true);
+ return windowOffset.incrementAndGetMaxOffset();
+ }
+
+ @Override
+ public void loadMaxSplitNum(Set<WindowInstance> windowInstances, String splitId) {
+ if(windowInstances==null||StringUtil.isEmpty(splitId)){
+ return;
+ }
+ Set<String> keys=new HashSet<>();
+ for(WindowInstance instance:windowInstances){
+ String key= createSplitNumberKey(instance,splitId);
+ keys.add(key);
+ }
+ queryOrCreateWindowOffset(keys,window.isLocalStorageOnly());
+ }
+
+ @Override
+ public void loadMaxSplitNum(Set<String> keys) {
+ queryOrCreateWindowOffset(keys,window.isLocalStorageOnly());
+ }
+
+ @Override
+ public void loadWindowMaxEventTime(Set<String> splitIds) {
+ if(splitIds==null){
+ return;
+ }
+ Set<String> keys=new HashSet<>();
+ for(String splitId:splitIds){
+ String windowId=StringUtil.createMD5Str(MapKeyUtil.createKey(window.getNameSpace(),window.getConfigureName()));
+ String key=MapKeyUtil.createKey(splitId, windowId);
+ keys.add(key);
+ }
+
+ queryOrCreateWindowOffset(keys,window.isLocalStorageOnly());
+ }
+
+ @Override
+ public void flush(){
+ if(window.isLocalStorageOnly()){
+ needUpdateWindowValues =new ArrayList<>();
+ return;
+ }
+ List<WindowMaxValue> windowOffsetList=new ArrayList<>();
+ synchronized (this){
+ windowOffsetList.addAll(needUpdateWindowValues);
+ needUpdateWindowValues =new ArrayList<>();
+ }
+ ORMUtil.batchReplaceInto(windowOffsetList);
+ }
+
+ @Override
+ public void resetSplitNum(WindowInstance instance, String splitId) {
+ String key=createSplitNumberKey(instance,splitId);
+ resetSplitNum(key);
+ }
+
+ @Override
+ public synchronized void resetSplitNum(String key) {
+ WindowMaxValue windowMaxValue=queryOrCreateWindowOffset(key,window.isLocalStorageOnly());
+ windowMaxValue.maxValue.set(MAX_VALUE_BASE_VALUE);
+ needUpdateWindowValues.add(windowMaxValue);
+ }
+
+ @Override
+ public synchronized void removeKeyPrefixFromLocalCache(Set<String> keyPrefixs) {
+ Map<String, WindowMaxValue> copy=new HashMap<>();
+ copy.putAll(this.windowOffsetMap);
+ for(String key:copy.keySet()){
+ for(String keyPrefix:keyPrefixs){
+ if(key.startsWith(keyPrefix)){
+ this.windowOffsetMap.remove(key);
+ }
+ }
+ }
+
+ }
+
+ /**
+ * 查询window的总计数器
+ *
+ * @return
+ */
+ protected WindowMaxValue queryOrCreateWindowOffset(String key,boolean onlyLocal){
+ Set<String> keys=new HashSet<>();
+ keys.add(key);
+ Map<String,WindowMaxValue> windowMaxValueMap=queryOrCreateWindowOffset(keys,onlyLocal);
+ if(windowMaxValueMap==null){
+ return null;
+ }
+ return windowMaxValueMap.values().iterator().next();
+ }
+
+ /**
+ * 查询window的总计数器
+ *
+ * @return
+ */
+ protected Map<String,WindowMaxValue> queryOrCreateWindowOffset(Set<String> keys,boolean onlyLocal){
+ Map<String,WindowMaxValue> result=new HashMap<>();
+ if(keys==null){
+ return result;
+ }
+ List<String> keyNotInLocal=new ArrayList<>();
+ for(String key:keys){
+ WindowMaxValue windowMaxValue=windowOffsetMap.get(key);
+ if(windowMaxValue !=null){
+ result.put(key,windowMaxValue);
+ }else if(onlyLocal){
+ windowMaxValue=create(key);
+ result.put(key,windowMaxValue);
+ }else {
+ keyNotInLocal.add(key);
+ }
+ }
+
+ if(onlyLocal){
+ return result;
+ }
+ if(keyNotInLocal.size()==0){
+ return result;
+ }
+ synchronized (this){
+ List<String> synchKeyNotInLocal=new ArrayList<>();
+ for(String key:keyNotInLocal) {
+ WindowMaxValue windowMaxValue = windowOffsetMap.get(key);
+ if (windowMaxValue != null) {
+ result.put(key, windowMaxValue);
+ }else {
+ synchKeyNotInLocal.add(key);
+ }
+ }
+ List<WindowMaxValue> windowMaxValues=null;
+ if(synchKeyNotInLocal.size()>0){
+ String sql="select * from "+ ORMUtil.getTableName(WindowMaxValue.class)+ " where msg_key in ("+ SQLUtil.createInSql(synchKeyNotInLocal) +")";
+ windowMaxValues=ORMUtil.queryForList(sql,null, WindowMaxValue.class);
+
+ }
+ // String key= MapKeyUtil.createKey(window.getNameSpace(),window.getConfigureName(),split);
+ List<String> keysNotInDB=new ArrayList<>();
+ keysNotInDB.addAll(synchKeyNotInLocal);
+ if(windowMaxValues!=null){
+ for(WindowMaxValue windowMaxValue:windowMaxValues){
+ result.put(windowMaxValue.getMsgKey(), windowMaxValue);
+ keysNotInDB.remove(windowMaxValue);
+ windowOffsetMap.put(windowMaxValue.getMsgKey(),windowMaxValue);
+ }
+ }
+ if(keysNotInDB!=null&&keysNotInDB.size()>0){
+ for(String key:keysNotInDB){
+ result.put(key, create(key));
+ }
+ }
+ }
+ return result;
+ }
+
+ protected WindowMaxValue create(String key){
+ WindowMaxValue windowMaxValue=new WindowMaxValue();
+ windowMaxValue.setGmtCreate(new Date());
+ windowMaxValue.setGmtModified(new Date());
+ windowMaxValue.setMsgKey(key);
+ windowMaxValue.setMaxValue(MAX_VALUE_BASE_VALUE);
+ windowMaxValue.setMaxEventTime(null);
+ needUpdateWindowValues.add(windowMaxValue);
+ windowOffsetMap.put(key,windowMaxValue);
+ return windowMaxValue;
+ }
+}
diff --git a/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/operator/AbstractShuffleWindow.java b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/operator/AbstractShuffleWindow.java
new file mode 100644
index 0000000..cb20f75
--- /dev/null
+++ b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/operator/AbstractShuffleWindow.java
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.window.operator;
+
+import org.apache.rocketmq.streams.common.configurable.IAfterConfiguableRefreshListerner;
+import org.apache.rocketmq.streams.common.configurable.IConfigurableService;
+import org.apache.rocketmq.streams.common.context.IMessage;
+import org.apache.rocketmq.streams.window.model.WindowInstance;
+import org.apache.rocketmq.streams.window.shuffle.ShuffleChannel;
+import org.apache.rocketmq.streams.window.source.WindowRireSource;
+import org.apache.rocketmq.streams.window.storage.WindowStorage;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+public abstract class AbstractShuffleWindow extends AbstractWindow implements IAfterConfiguableRefreshListerner {
+
+ protected transient ShuffleChannel shuffleChannel;
+ protected transient AtomicBoolean hasCreated = new AtomicBoolean(false);
+
+ @Override
+ protected boolean initConfigurable() {
+ storage = new WindowStorage();
+ storage.setLocalStorageOnly(isLocalStorageOnly);
+ return super.initConfigurable();
+ }
+
+ @Override
+ public void doProcessAfterRefreshConfigurable(IConfigurableService configurableService) {
+ if (hasCreated.compareAndSet(false, true)) {
+ this.windowFireSource = new WindowRireSource(this);
+ this.windowFireSource.init();
+ this.windowFireSource.start(getFireReceiver());
+ this.shuffleChannel = new ShuffleChannel(this);
+ windowCache.setBatchSize(5000);
+ windowCache.setShuffleChannel(shuffleChannel);
+ shuffleChannel.startChannel();
+ }
+ }
+
+ @Override
+ public int fireWindowInstance(WindowInstance windowInstance, Map<String, String> queueId2Offset) {
+ Set<String> splitIds = new HashSet<>();
+ splitIds.add(windowInstance.getSplitId());
+ shuffleChannel.flush(splitIds);
+ int fireCount = fireWindowInstance(windowInstance, windowInstance.getSplitId(), queueId2Offset);
+ return fireCount;
+ }
+
+ /**
+ * 接收shuffle后的消息进行计算，子类实现具体计算逻辑
+ *
+ * @param messages
+ * @param instance
+ */
+ public abstract void shuffleCalculate(List<IMessage> messages, WindowInstance instance, String queueId);
+
+ /**
+ * 触发window
+ *
+ * @param instance
+ */
+ protected abstract int fireWindowInstance(WindowInstance instance, String queueId, Map<String, String> queueId2Offset);
+
+ public abstract void clearCache(String queueId);
+}
diff --git a/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/operator/AbstractWindow.java b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/operator/AbstractWindow.java
new file mode 100644
index 0000000..6d23157
--- /dev/null
+++ b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/operator/AbstractWindow.java
@@ -0,0 +1,781 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.window.operator;
+
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+
+import com.alibaba.fastjson.JSONObject;
+
+import org.apache.rocketmq.streams.common.configurable.BasedConfigurable;
+import org.apache.rocketmq.streams.common.context.Message;
+import org.apache.rocketmq.streams.common.topology.ChainStage.PiplineRecieverAfterCurrentNode;
+import org.apache.rocketmq.streams.common.topology.stages.udf.IReducer;
+import org.apache.rocketmq.streams.common.topology.stages.udf.IRedurce;
+import org.apache.rocketmq.streams.common.utils.Base64Utils;
+import org.apache.rocketmq.streams.common.utils.InstantiationUtil;
+import org.apache.rocketmq.streams.db.driver.orm.ORMUtil;
+import org.apache.rocketmq.streams.script.utils.FunctionUtils;
+import org.apache.rocketmq.streams.window.model.FunctionExecutor;
+import org.apache.rocketmq.streams.window.model.WindowInstance;
+import org.apache.rocketmq.streams.window.model.WindowCache;
+import org.apache.rocketmq.streams.window.offset.IWindowMaxValueManager;
+import org.apache.rocketmq.streams.window.offset.WindowMaxValueManager;
+import org.apache.rocketmq.streams.window.source.WindowRireSource;
+import org.apache.rocketmq.streams.window.state.impl.WindowValue;
+import org.apache.rocketmq.streams.common.context.AbstractContext;
+import org.apache.rocketmq.streams.common.context.IMessage;
+import org.apache.rocketmq.streams.common.context.MessageHeader;
+import org.apache.rocketmq.streams.common.topology.ChainStage;
+import org.apache.rocketmq.streams.common.topology.builder.IStageBuilder;
+import org.apache.rocketmq.streams.common.topology.builder.PipelineBuilder;
+import org.apache.rocketmq.streams.common.topology.stages.WindowChainStage;
+import org.apache.rocketmq.streams.common.topology.model.IWindow;
+import org.apache.rocketmq.streams.common.utils.CollectionUtil;
+import org.apache.rocketmq.streams.common.utils.DateUtil;
+import org.apache.rocketmq.streams.common.utils.MapKeyUtil;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+import org.apache.rocketmq.streams.script.parser.imp.FunctionParser;
+import org.apache.rocketmq.streams.script.operator.impl.AggregationScript;
+import org.apache.rocketmq.streams.script.operator.impl.FunctionScript;
+import org.apache.rocketmq.streams.script.service.IScriptExpression;
+import org.apache.rocketmq.streams.script.service.IScriptParamter;
+import org.apache.rocketmq.streams.script.operator.expression.ScriptExpression;
+import org.apache.rocketmq.streams.script.service.IAccumulator;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.rocketmq.streams.window.storage.WindowStorage;
+
+import static java.util.concurrent.CompletableFuture.supplyAsync;
+
+/**
+ * window definition in the pipeline, created by user's configure in WindowChainStage
+ */
+public abstract class AbstractWindow extends BasedConfigurable implements IWindow, IStageBuilder<ChainStage> {
+
+ protected static final Log LOG = LogFactory.getLog(AbstractWindow.class);
+
+ /**
+ * tumble or hop window 目前不再使用了
+ */
+ protected String windowType;
+
+ /**
+ * 用消息中的哪个字段做时间字段
+ */
+ protected String timeFieldName ;
+
+ /**
+ * having column in having clause eg: key:'having_sum_0001' value:'having_sum_0001=SUM(OrderPrice)<2000' note: here ignore the logical relation value may be multi expression which split by ${SCRIPT_SPLIT_CHAR} update: change sql(move the function into select clause) to escape function in having clause
+ */
+ private Map<String, String> havingMap = new HashMap<>(16);
+
+ /**
+ * computed column in select clause eg: key:'max_valid_user_fail_host_cnt' value:'max_valid_user_fail_host_cnt=max(base_line_invalid_user_fail_host_cnt)' note: 1) value may be multi expression which split by ${SCRIPT_SPLIT_CHAR} 2) computed column can't be same
+ */
+ private Map<String, String> selectMap = new HashMap<>(16);
+
+ /**
+ * SQL中group by的字段，使用;拼接，如"name;age"
+ */
+ protected String groupByFieldName ;
+
+ /**
+ * 意义同blink中，允许最晚的消息到达时间，单位是分钟
+ */
+ protected int waterMarkMinute = 0;
+
+ /**
+ * size or step of window, unit: minute
+ */
+ protected int sizeInterval;
+ /**
+ * the period of hop window, unit: minute
+ */
+ protected int slideInterval;
+ /**
+ * 主要是做兼容，以前设计的窗口时间是分钟为单位，如果有秒作为窗口时间的，通过设置timeUntiAdjust=1来实现。 后续需要调整成直接秒级窗口
+ */
+ protected int timeUnitAdjust=60;
+ /**
+ * the variable name of window size which can be got from message
+ */
+ protected String sizeVariable;
+
+ /**
+ * the coefficient to adjust window size for use minute as unit
+ */
+ protected Integer sizeAdjust;
+
+ /**
+ * the coefficient to adjust window slide for use minute as unit
+ */
+ protected Integer slideAdjust;
+
+ /**
+ * the variable name of window slide which it's value can be got from message
+ */
+ protected String slideVariable;
+
+ /**
+ * 默认为空，窗口的触发类似flink，在测试模式下，因为消息有界，期望当消息发送完成后能触发，可以设置两条消息的最大间隔，超过这个间隔，将直接触发消息
+ */
+ protected Long msgMaxGapSecond;
+
+ /**
+ * 是否支持过期数据的计算 过期：当前时间大于数据所在窗口的触发时间
+ */
+ protected int fireMode=0;//0:普通触发,firetime后收到数据丢弃；1:多实例多次独立触发，在watermark时间内，同starttime，endtime创建多个实例，多次触发；2.单实例，多次独立触发，每次触发是最新值
+
+ protected boolean isLocalStorageOnly=false;//是否只用本地存储，可以提高性能，但不保证可靠性
+ protected String reduceSerializeValue;//用户自定义的operator的序列化字节数组，做了base64解码
+ protected transient IReducer reducer;
+ /**
+ * the computed column and it's process of computing
+ */
+ private transient Map<String, List<FunctionExecutor>> columnExecuteMap = new HashMap<>(16);
+
+ /**
+ * used in last part to filter and transfer field in case data lost during firing
+ */
+ protected transient Map<String, String> columnProjectMap = new HashMap<>();
+
+ /**
+ * 当前计算节点的PipeLine里的Window实例对象，方便基于时间快速定位 key：namespace;configName(这里理解成windowName);startTime;endTime value：WindowInstance
+ */
+ protected transient ConcurrentHashMap<String, WindowInstance> windowInstanceMap = new ConcurrentHashMap<>();
+
+ /**
+ * 触发窗口后需要执行的逻辑
+ */
+ protected transient PiplineRecieverAfterCurrentNode fireReceiver;
+
+ /**
+ * 全局名称
+ */
+ protected transient String WINDOW_NAME;
+
+ /**
+ * 内部使用,定期检查窗口有没有触发
+ */
+ //protected transient ScheduledExecutorService fireWindowInstanceChecker =new ScheduledThreadPoolExecutor(3);
+
+ // protected transient ExecutorService deleteService = Executors.newSingleThreadExecutor();
+
+ protected volatile transient WindowCache windowCache;
+ protected transient WindowStorage storage;
+ protected transient WindowRireSource windowFireSource;
+
+ //create and save window instacne max partitionNum and window max eventTime
+ protected transient IWindowMaxValueManager windowMaxValueManager;
+
+ public AbstractWindow() {
+ setType(IWindow.TYPE);
+ }
+
+ @Override
+ protected boolean initConfigurable() {
+ boolean success = super.initConfigurable();
+ /**
+ * 如果没有db配置，不开启远程存储服务
+ */
+ if(!ORMUtil.hasConfigueDB()){
+ isLocalStorageOnly=true;
+ }
+ AbstractWindow window=this;
+ windowCache=new WindowCache(){
+
+ @Override
+ protected String generateShuffleKey(IMessage message) {
+ return window.generateShuffleKey(message);
+ }
+ };
+ windowCache.init();
+ windowCache.openAutoFlush();
+
+ WINDOW_NAME = MapKeyUtil.createKey(getNameSpace(), getConfigureName());
+ //fireDelaySecond时间检查一次是否有窗口需要触发
+ //fireWindowInstanceChecker.scheduleWithFixedDelay(this, 0, 5, TimeUnit.SECONDS);
+ initFunctionExecutor();
+ //启动shuffle channel 实现消息shuffle以及接收shuffle消息并处理
+ // FireManager.getInstance().startFireCheck();
+ if(StringUtil.isNotEmpty(this.reduceSerializeValue)){
+ byte[] bytes= Base64Utils.decode( this.reduceSerializeValue);
+ reducer = InstantiationUtil.deserializeObject(bytes);
+ }
+ windowMaxValueManager = new WindowMaxValueManager(this);
+ return success;
+ }
+
+ /**
+ * 对于一条消息来说，window 首先需要检查是否有窗口实例，如果没有则创建。如果窗口实例已经超过最大的water mark，数据丢弃，否则进行消息积累 消息会先经历batchAdd 然后flush
+ *
+ * @param message
+ * @param context
+ * @return
+ */
+ @Override
+ public AbstractContext<IMessage> doMessage(IMessage message, AbstractContext context) {
+ if (StringUtils.isNotEmpty(sizeVariable)) {
+ if (message.getMessageBody().containsKey(sizeVariable)) {
+ try {
+ this.sizeInterval = sizeAdjust * message.getMessageBody().getInteger(sizeVariable);
+ } catch (Exception e) {
+ LOG.error("failed in getting the size value, message = " + message.toString(), e);
+ }
+ }
+ }
+ if (StringUtils.isNotEmpty(slideVariable)) {
+ if (message.getMessageBody().containsKey(slideVariable)) {
+ try {
+ this.slideInterval = slideAdjust * message.getMessageBody().getInteger(slideVariable);
+ } catch (Exception e) {
+ LOG.error("failed in getting the slide value, message = " + message.toString(), e);
+ }
+ }
+ }
+ // List<WindowInstance> windowInstanceList = queryOrCreateWindowInstance(message);
+ JSONObject msg = message.getMessageBody();
+ msg.put(MessageHeader.class.getSimpleName(), message.getHeader());
+ msg.put(AbstractWindow.class.getSimpleName(), this);
+ windowCache.batchAdd(message);
+ //主要为了在单元测试中，写入和触发一体化使用，无实际意义，不要在业务场景使用这个字段
+
+ // TraceUtil.debug(message.getHeader().getTraceId(), "origin message in", message.getMessageBody().toJSONString());
+ return context;
+
+ }
+
+ /*
+
+ public String createWindowInstance(String startTime,String endTime,String fireTime){
+
+ }
+ */
+
+ public WindowInstance createWindowInstance(String startTime, String endTime, String fireTime,String splitId) {
+ WindowInstance windowInstance =new WindowInstance();
+ windowInstance.setFireTime(fireTime);
+ windowInstance.setStartTime(startTime);
+ windowInstance.setEndTime(endTime);
+ windowInstance.setSplitId(splitId);
+ windowInstance.setGmtCreate(new Date());
+ windowInstance.setGmtModified(new Date());
+ windowInstance.setWindowInstanceName(createWindowInstanceName(startTime,endTime,fireTime));
+ windowInstance.setWindowName(getConfigureName());
+ windowInstance.setWindowNameSpace(getNameSpace());
+ String windowInstanceId =windowInstance.createWindowInstanceId();
+ String dbWindowInstanceId = StringUtil.createMD5Str(windowInstanceId);
+ windowInstance.setWindowInstanceKey(dbWindowInstanceId);
+
+ windowInstance.setWindowInstanceSplitName(StringUtil.createMD5Str(MapKeyUtil.createKey(getNameSpace(), getConfigureName(),splitId)));
+ windowInstance.setNewWindowInstance(true);
+ return windowInstance;
+ }
+
+ /**
+ * 创建window instance name
+ *
+ * @param startTime
+ * @param endTime
+ * @param fireTime
+ * @return
+ */
+ public String createWindowInstanceName(String startTime, String endTime, String fireTime){
+ return fireMode==1?fireTime:getConfigureName();
+ }
+
+ /**
+ * 获取这个窗口实例，这个分片最大的序列号，如果是新窗口，从1开始
+ *
+ * @param instance
+ * @param shuffleId
+ * @return
+ */
+
+ public long incrementAndGetSplitNumber(WindowInstance instance,String shuffleId){
+ long maxValue= windowMaxValueManager.incrementAndGetSplitNumber(instance,shuffleId);
+ return maxValue;
+ }
+
+ public abstract Class getWindowBaseValueClass();
+
+ public abstract int fireWindowInstance(WindowInstance windowInstance,Map<String,String>queueId2Offset) ;
+
+ /**
+ * 计算每条记录的group by值，对于groupby分组，里面任何字段不能为null值，如果为null值，这条记录会被忽略
+ *
+ * @param message
+ * @return
+ */
+ protected String generateShuffleKey(IMessage message){
+ if (StringUtil.isEmpty(groupByFieldName)) {
+ return null;
+ }
+ JSONObject msg=message.getMessageBody();
+ String[] fieldNames = groupByFieldName.split(";");
+ String[] values=new String[fieldNames.length];
+ boolean isFirst = true;
+ int i=0;
+ for (String filedName : fieldNames) {
+ if (isFirst) {
+ isFirst = false;
+ }
+ String value = msg.getString(filedName);
+ values[i]=value;
+ i++;
+ }
+ return MapKeyUtil.createKey(values);
+ }
+
+ public abstract void clearFire(List<WindowInstance> windowInstances);
+
+ public void clearFire(WindowInstance windowInstance){
+ if(windowInstance==null){
+ return;
+ }
+ List<WindowInstance>windowInstances=new ArrayList<>();
+ windowInstances.add(windowInstance);
+ clearFire(windowInstances);
+ }
+
+ /**
+ * init the function executor TODO: 1) function executor may be parsed in parser module;
+ */
+ protected void initFunctionExecutor() {
+ //
+ columnExecuteMap.clear();
+ columnProjectMap.clear();
+ //
+ for (Entry<String, String> entry : selectMap.entrySet()) {
+ String computedColumn = entry.getKey();
+ columnProjectMap.put(computedColumn, computedColumn);
+ String scriptString = entry.getValue();
+ if (StringUtil.isEmpty(computedColumn) || StringUtil.isEmpty(scriptString)) {
+ LOG.warn(
+ "computed column or it's expression can not be empty! column = " + computedColumn + " expression = "
+ + scriptString);
+ continue;
+ }
+ if (computedColumn.equals(scriptString)) {
+ continue;
+ }
+ //
+ LinkedList<FunctionExecutor> scriptExecutorList = new LinkedList<>();
+ List<IScriptExpression> functionList = new ArrayList<>();
+ try {
+ functionList = FunctionParser.getInstance().parse(scriptString);
+ } catch (Exception e) {
+ LOG.error("failed in parsing script expression = " + scriptString + " window = " + WINDOW_NAME);
+ throw new RuntimeException("failed in parsing operator expression = " + scriptString);
+ }
+ if (CollectionUtil.isNotEmpty(functionList)) {
+ StringBuilder scriptBuilder = new StringBuilder();
+ for (IScriptExpression expression : functionList) {
+ String functionName = expression.getFunctionName();
+ List<IScriptParamter> scriptParameterList = expression.getScriptParamters();
+ String theScript = expression.getExpressionDescription();
+ IAccumulator director = AggregationScript.getAggregationFunction(functionName);
+ if (director != null) {
+ if (scriptBuilder.length() != 0) {
+ FunctionScript scalarEngine = new FunctionScript(scriptBuilder.toString());
+ scalarEngine.init();
+ scriptExecutorList.add(
+ new FunctionExecutor(computedColumn + "_" + scriptExecutorList.size(), scalarEngine));
+ scriptBuilder = new StringBuilder();
+ }
+ String[] functionParameterNames = scriptParameterList.stream().map(
+ scriptParameter -> scriptParameter.getScriptParameterStr()).collect(Collectors.toList())
+ .toArray(new String[0]);
+ AggregationScript accEngine = new AggregationScript(
+ ((ScriptExpression)expression).getNewFieldName(), functionName,
+ functionParameterNames);
+ accEngine.setDirector(director);
+ scriptExecutorList.add(
+ new FunctionExecutor(computedColumn + "_" + scriptExecutorList.size(), accEngine));
+ } else {
+ scriptBuilder.append(theScript).append(SCRIPT_SPLIT_CHAR);
+ }
+ }
+ if (scriptBuilder.length() != 0) {
+ FunctionScript scalarEngine = new FunctionScript(scriptBuilder.toString());
+ scalarEngine.init();
+ scriptExecutorList.add(
+ new FunctionExecutor(computedColumn + "_" + scriptExecutorList.size(), scalarEngine));
+ }
+ columnExecuteMap.put(computedColumn, scriptExecutorList);
+ } else {
+ LOG.error("parser's result is empty, script expression = " + scriptString + " window = " + WINDOW_NAME);
+ throw new RuntimeException("parser's result is empty, operator expression = " + scriptString);
+ }
+ }
+ if (LOG.isDebugEnabled()) {
+ Iterator<Entry<String, List<FunctionExecutor>>> iterator = columnExecuteMap.entrySet().iterator();
+ LOG.debug("window function execute split as follows:\t");
+ while (iterator.hasNext()) {
+ Entry<String, List<FunctionExecutor>> entry = iterator.next();
+ StringBuilder builder = new StringBuilder();
+ for (FunctionExecutor executor : entry.getValue()) {
+ if (executor.getExecutor() instanceof AggregationScript) {
+ builder.append(((AggregationScript)executor.getExecutor()).getFunctionName()).append("\t");
+ } else {
+ builder.append(((FunctionScript)executor.getExecutor()).getScript()).append("\t");
+ }
+ }
+ LOG.debug(entry.getKey() + " -> " + builder.toString());
+ }
+ }
+ }
+
+ /**
+ * 根据消息获取对应的window instance 列表
+ *
+ * @param message
+ * @return
+ */
+ public List<WindowInstance> queryOrCreateWindowInstance(IMessage message,String queueId) {
+ List<WindowInstance> windowInstances=WindowInstance.getOrCreateWindowInstance(this, WindowInstance.getOccurTime(this, message), timeUnitAdjust,
+ queueId);
+ if(fireMode==2){
+ if(windowInstances==null){
+ return null;
+ }
+ for(WindowInstance windowInstance:windowInstances){
+ Date endTime=DateUtil.parseTime(windowInstance.getEndTime());
+ Date lastFireTimne=DateUtil.addDate(TimeUnit.SECONDS,endTime,getWaterMarkMinute()*timeUnitAdjust);
+ //if fireMode==2， need clear data in lastFireTime
+ WindowInstance lastClearWindowInstance=createWindowInstance(windowInstance.getStartTime(),windowInstance.getEndTime(),DateUtil.format(lastFireTimne),queueId);
+ getWindowFireSource().registFireWindowInstanceIfNotExist(lastClearWindowInstance,this);
+ }
+ }
+ return windowInstances;
+ }
+
+ /**
+ * 获取实际触发的时间，增加了固定延迟和water marker
+ *
+ * @param windowInstance
+ * @return
+ */
+ public Date getRealFireTime(WindowInstance windowInstance){
+ Date fireDate=DateUtil.parse(windowInstance.getFireTime());
+ return DateUtil.addSecond(fireDate,FIRE_DELAY_SECOND);
+ }
+
+ /**
+ * 获取window处理的消息中最大的时间
+ * @param msg
+ * @return
+ */
+ public Long updateMaxEventTime(IMessage msg){
+ String splitId=msg.getHeader().getQueueId();
+ if(StringUtil.isEmpty(this.timeFieldName)){
+ return updateWindowEventTime(splitId,System.currentTimeMillis());
+ }
+ String time=msg.getMessageBody().getString(timeFieldName);
+ if(StringUtil.isEmpty(time)){
+ LOG.error(timeFieldName+ " is null, may be delay window fire");
+ return null;
+ }
+ if(FunctionUtils.isLong(time)){
+ return updateWindowEventTime(splitId,Long.valueOf(time));
+ }
+ return updateWindowEventTime(splitId,time);
+ }
+
+ protected transient Map<String,Long> maxTime=new HashMap<>();
+ private Long updateWindowEventTime(String splitId, String currentTime) {
+ return updateWindowEventTime(splitId,DateUtil.parseTime(currentTime).getTime());
+ }
+ private Long updateWindowEventTime(String splitId, Long currentTime) {
+ Long old=maxTime.get(splitId);
+
+ if(old==null||currentTime>old){
+ maxTime.put(splitId,currentTime);
+ }
+ return maxTime.get(splitId);
+ }
+
+
+ /**
+ * 聚合后的数据，继续走规则引擎的规则
+ *
+ * @param windowValueList
+ */
+ public void sendFireMessage(List<WindowValue> windowValueList,String queueId) {
+ int count = 0;
+ for (WindowValue windowValue : windowValueList) {
+ JSONObject message = new JSONObject();
+
+ if(JSONObject.class.isInstance(windowValue.getcomputedResult())){
+ message=(JSONObject)windowValue.getcomputedResult();
+ }else {
+ Iterator<Entry<String, Object>> it = windowValue.iteratorComputedColumnResult();
+ while (it.hasNext()) {
+ Entry<String, Object> entry = it.next();
+ message.put(entry.getKey(), entry.getValue());
+ }
+ }
+
+ Long fireTime=DateUtil.parseTime(windowValue.getFireTime()).getTime();
+ long baseTime= 1577808000000L ;//set base time from 2021-01-01 00:00:00
+ int sameFireCount=0;
+ if(fireMode!=0){
+ Long endTime=DateUtil.parseTime(windowValue.getEndTime()).getTime();
+ sameFireCount=(int)((fireTime-endTime)/1000)/sizeInterval*timeUnitAdjust;
+ if(sameFireCount>=1){
+ sameFireCount=1;
+ }
+ }
+ //can keep offset in order
+ Long offset=((fireTime-baseTime)/1000*10+sameFireCount)*100000000+windowValue.getPartitionNum();
+ message.put("windowInstanceId",windowValue.getWindowInstancePartitionId());
+ message.put("offset",offset);
+ Message newMessage=windowFireSource.createMessage(message,queueId,offset+"",false);
+ newMessage.getHeader().setOffsetIsLong(true);
+ if (count == windowValueList.size() - 1) {
+ newMessage.getHeader().setNeedFlush(true);
+ }
+ windowFireSource.executeMessage(newMessage);
+ count++;
+ }
+ }
+
+ @Override
+ public ChainStage createStageChain(PipelineBuilder pipelineBuilder) {
+ pipelineBuilder.addConfigurables(this);
+ WindowChainStage windowChainStage = new WindowChainStage();
+ windowChainStage.setWindow(this);
+ windowChainStage.setNameSpace(getNameSpace());
+ return windowChainStage;
+ }
+
+ @Override
+ public void addConfigurables(PipelineBuilder pipelineBuilder) {
+
+ }
+
+ public void setSizeVariable(String variableName) {
+ sizeVariable = variableName;
+ }
+
+ public String getWindowType() {
+ return windowType;
+ }
+
+ public void setWindowType(String windowType) {
+ this.windowType = windowType;
+ }
+
+ public String getTimeFieldName() {
+ return timeFieldName;
+ }
+
+ public void setTimeFieldName(String timeFieldName) {
+ this.timeFieldName = timeFieldName;
+ }
+
+ public Map<String, String> getSelectMap() {
+ return selectMap;
+ }
+
+ public void setSelectMap(Map<String, String> selectMap) {
+ this.selectMap = selectMap;
+ }
+
+ public Map<String, List<FunctionExecutor>> getColumnExecuteMap() {
+ return columnExecuteMap;
+ }
+
+ public Map<String, String> getColumnProjectMap() {
+ return columnProjectMap;
+ }
+
+ public String getGroupByFieldName() {
+ return groupByFieldName;
+ }
+
+ public void setGroupByFieldName(String groupByFieldName) {
+ this.groupByFieldName = groupByFieldName;
+ }
+
+ public int getWaterMarkMinute() {
+ return waterMarkMinute;
+ }
+
+ public void setWaterMarkMinute(int waterMarkMinute) {
+ this.waterMarkMinute = waterMarkMinute;
+ }
+
+ public int getSizeInterval() {
+ return sizeInterval;
+ }
+
+ public void setSizeInterval(int sizeInterval) {
+ this.sizeInterval = sizeInterval;
+ }
+
+ public ConcurrentHashMap<String, WindowInstance> getWindowInstanceMap() {
+ return windowInstanceMap;
+ }
+
+ public void setWindowInstanceMap(
+ ConcurrentHashMap<String, WindowInstance> windowInstanceMap) {
+ this.windowInstanceMap = windowInstanceMap;
+ }
+
+ public PiplineRecieverAfterCurrentNode getFireReceiver() {
+ return fireReceiver;
+ }
+
+ @Override
+ public void setFireReceiver(
+ PiplineRecieverAfterCurrentNode fireReceiver) {
+ this.fireReceiver = fireReceiver;
+ }
+
+ @Override
+ public boolean isSynchronous() {
+ return false;
+ }
+
+ public Map<String, String> getHavingMap() {
+ return havingMap;
+ }
+
+ public void setHavingMap(Map<String, String> havingMap) {
+ this.havingMap = havingMap;
+ }
+
+ public int getSlideInterval() {
+ return slideInterval;
+ }
+
+ public void setSlideInterval(int slideInterval) {
+ this.slideInterval = slideInterval;
+ }
+
+ public String getSizeVariable() {
+ return sizeVariable;
+ }
+
+ public Integer getSizeAdjust() {
+ return sizeAdjust;
+ }
+
+ public void setSizeAdjust(Integer sizeAdjust) {
+ this.sizeAdjust = sizeAdjust;
+ }
+
+ public Integer getSlideAdjust() {
+ return slideAdjust;
+ }
+
+ public void setSlideAdjust(Integer slideAdjust) {
+ this.slideAdjust = slideAdjust;
+ }
+
+ public String getSlideVariable() {
+ return slideVariable;
+ }
+
+ public void setSlideVariable(String slideVariable) {
+ this.slideVariable = slideVariable;
+ }
+
+ public String getReduceSerializeValue() {
+ return reduceSerializeValue;
+ }
+
+ public void setReduceSerializeValue(String reduceSerializeValue) {
+ this.reduceSerializeValue = reduceSerializeValue;
+ }
+
+ public IReducer getReducer() {
+ return reducer;
+ }
+
+ public void setReducer(IReducer reducer) {
+ this.reducer = reducer;
+ byte[] bytes = InstantiationUtil.serializeObject(reducer);
+ this.reduceSerializeValue=Base64Utils.encode(bytes);
+ }
+
+ public int getTimeUnitAdjust() {
+ return timeUnitAdjust;
+ }
+
+ public void setTimeUnitAdjust(int timeUnitAdjust) {
+ this.timeUnitAdjust = timeUnitAdjust;
+ }
+
+ public boolean isLocalStorageOnly() {
+ return isLocalStorageOnly;
+ }
+
+ public void setLocalStorageOnly(boolean localStorageOnly) {
+ isLocalStorageOnly = localStorageOnly;
+ }
+
+ public int getFireMode() {
+ return fireMode;
+ }
+
+ public void setFireMode(int fireMode) {
+ this.fireMode = fireMode;
+ }
+
+ public void removeInstanceFromMap(WindowInstance windowInstance) {
+ this.windowInstanceMap.remove(windowInstance.createWindowInstanceId());
+
+ }
+
+ @Override
+ public WindowCache getWindowCache() {
+ return windowCache;
+ }
+
+ public WindowStorage getStorage() {
+ return storage;
+ }
+
+ public WindowRireSource getWindowFireSource() {
+ return windowFireSource;
+ }
+
+ public IWindowMaxValueManager getWindowMaxValueManager() {
+ return windowMaxValueManager;
+ }
+
+ public Long getMsgMaxGapSecond() {
+ return msgMaxGapSecond;
+ }
+
+ public void setMsgMaxGapSecond(Long msgMaxGapSecond) {
+ this.msgMaxGapSecond = msgMaxGapSecond;
+ }
+}
diff --git a/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/operator/impl/OverWindow.java b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/operator/impl/OverWindow.java
new file mode 100644
index 0000000..1cc2aea
--- /dev/null
+++ b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/operator/impl/OverWindow.java
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.window.operator.impl;
+
+import org.apache.rocketmq.streams.common.cache.compress.impl.IntValueKV;
+import org.apache.rocketmq.streams.common.context.AbstractContext;
+import org.apache.rocketmq.streams.common.context.IMessage;
+import org.apache.rocketmq.streams.common.utils.DateUtil;
+import org.apache.rocketmq.streams.window.model.WindowInstance;
+import org.apache.rocketmq.streams.window.operator.AbstractWindow;
+
+import java.util.Date;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * 只支持 时间去重的场景，日志是按系统时间顺序，所以不落盘。需要设置groupByFieldName和rowNumerName字段
+ */
+public class OverWindow extends AbstractWindow {
+
+ private static int MAX_SIZE = 1000000;
+ protected transient IntValueKV intValueKV;
+ protected transient WindowInstance windowInstance;
+ /**
+ * 需要把生成的序列号返回设置到message，这个是序列号对应的名字
+ */
+ protected String rowNumerName;
+
+ /**
+ * 针对这个窗口实例完成计算，实际上是写入了缓存，在flush时完成真正的计算。写入缓存时把上下文（header，windowinstance，window）保存在消息中
+ *
+ * @param message
+ * @param context
+ */
+ @Override
+ public AbstractContext<IMessage> doMessage(IMessage message, AbstractContext context) {
+ String key = generateShuffleKey(message);
+ createWindowInstanceByDate(new Date());
+ Integer value = intValueKV.get(key);
+ if (value == null) {
+ synchronized (this) {
+ value = intValueKV.get(key);
+ if (value == null) {
+ value = 1;
+ intValueKV.put(key, value);
+ }
+ }
+
+ }
+ message.getMessageBody().put(rowNumerName, value);
+ /**
+ * 如果超过最大值，直接归0
+ */
+ if (intValueKV.getSize() > MAX_SIZE) {
+ synchronized (this) {
+ if (intValueKV.getSize() > MAX_SIZE) {
+ intValueKV = new IntValueKV(MAX_SIZE);
+ }
+ }
+ }
+ return context;
+ }
+
+ /**
+ * 如果时间内无instance，创建，如果不在现有的instance中，现有的失效，重新创建
+ *
+ * @param date
+ * @return
+ */
+ protected void createWindowInstanceByDate(Date date) {
+ if (windowInstance == null) {
+ synchronized (this) {
+ if (windowInstance == null) {
+ windowInstance = createWindowInstance(date);
+ intValueKV = new IntValueKV(MAX_SIZE);
+ return;
+ }
+ }
+ }
+ String dateStr = DateUtil.format(date);
+ if (dateStr.compareTo(windowInstance.getStartTime()) >= 0 && dateStr.compareTo(windowInstance.getEndTime()) <= 0) {
+ return;
+ } else {
+ synchronized (this) {
+ windowInstance = createWindowInstance(date);
+ intValueKV = new IntValueKV(MAX_SIZE);
+ }
+ }
+ }
+
+ /**
+ * 根据日期创建window instance
+ *
+ * @param date
+ * @return
+ */
+ protected WindowInstance createWindowInstance(Date date) {
+ List<Date> instanceStartTimes = DateUtil.getWindowBeginTime(date.getTime(), slideInterval, sizeInterval);
+ Date instanceStartTime = instanceStartTimes.get(0);
+ WindowInstance windowInstance = new WindowInstance();
+ windowInstance.setStartTime(DateUtil.format(instanceStartTime));
+ Date endDate = DateUtil.addMinute(instanceStartTime, sizeInterval);
+ windowInstance.setEndTime(DateUtil.format(endDate));
+ return windowInstance;
+ }
+
+ @Override
+ public boolean isSynchronous() {
+ return true;
+ }
+
+ @Override
+ protected boolean initConfigurable() {
+ return true;
+ }
+
+ @Override
+ public Class getWindowBaseValueClass() {
+ return null;
+ }
+
+ @Override
+ public int fireWindowInstance(WindowInstance windowInstance, Map<String, String> queueId2Offsets) {
+ return 0;
+ }
+
+ @Override
+ public void clearFire(List<WindowInstance> windowInstances) {
+
+ }
+
+ public String getRowNumerName() {
+ return rowNumerName;
+ }
+
+ public void setRowNumerName(String rowNumerName) {
+ this.rowNumerName = rowNumerName;
+ }
+
+}
diff --git a/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/operator/impl/SessionWindow.java b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/operator/impl/SessionWindow.java
new file mode 100644
index 0000000..fcdae1a
--- /dev/null
+++ b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/operator/impl/SessionWindow.java
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.window.operator.impl;
+
+import org.apache.rocketmq.streams.common.context.IMessage;
+import org.apache.rocketmq.streams.common.utils.DateUtil;
+import org.apache.rocketmq.streams.common.utils.MapKeyUtil;
+import org.apache.rocketmq.streams.common.utils.SQLUtil;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+import org.apache.rocketmq.streams.db.driver.DriverBuilder;
+import org.apache.rocketmq.streams.window.model.WindowInstance;
+import org.apache.rocketmq.streams.window.state.WindowBaseValue;
+import org.apache.rocketmq.streams.window.state.impl.WindowValue;
+import org.apache.rocketmq.streams.window.storage.WindowStorage.WindowBaseValueIterator;
+
+import java.util.*;
+
+/**
+ * 实现思路： 1.每个分片一个windowinstance，starttime=2020-12-30:00:00:00 endtime=2999-01-01 01:01:01 2.firetime，第一次创建窗口，firetime=当前时间计算+window size 3.增加存储，按window instance所有groupby的触发时间排序(设计前缀)，每次有数据来时，更新触发时间，触发时间算法如2 4.窗口触发时，检查存储中最近的触发时间是否<=触发时间，如果符合触发条件触发，然后一直迭代到触发时间>当前时间，把最近的触发时间当作window instance的触发时间，修改window instance的firetime 5.清理触发的数据（触发时间<=窗口实例的触发时间）
+ */
+public class SessionWindow extends WindowOperator {
+ private static final String ORDER_BY_FIRE_TIME = "_order_by_fire_time_ ";//key=_order;queueid,windowinstanceid,partitionNum
+
+ @Override
+ protected boolean initConfigurable() {
+ this.fireMode = 2;
+ return super.initConfigurable();
+ }
+
+ @Override
+ public int fireWindowInstance(WindowInstance instance, String queueId, Map<String, String> queueId2Offset) {
+ List<WindowValue> fireWindowValues = new ArrayList<>();
+ int fireCount = 0;
+ //for(String queueId:queueIds){
+ WindowBaseValueIterator<WindowBaseValue> it = storage.loadWindowInstanceSplitData(getOrderBypPrefix(), queueId, instance.createWindowInstanceId(), null, getWindowBaseValueClass());
+ if (queueId2Offset != null) {
+ String offset = queueId2Offset.get(queueId);
+ if (StringUtil.isNotEmpty(offset)) {
+ it.setPartitionNum(Long.valueOf(offset));
+ }
+ }
+ boolean hasFinished = true;
+ while (it.hasNext()) {
+ WindowBaseValue windowBaseValue = it.next();
+ if (windowBaseValue == null) {
+ continue;
+ }
+ Date realFireTime = getRealFireTime(instance);
+ Long currentMaxTime = instance.getLastMaxUpdateTime();
+ Long realFireTimeLong = realFireTime.getTime();
+ // System.out.println(DateUtil.format(new Date(currentMaxTime)));
+ /**
+ * first not fire window value
+ */
+ if (currentMaxTime - realFireTimeLong < 0) {
+ instance.setFireTime(windowBaseValue.getFireTime());
+ shuffleChannel.addNeedFlushWindowInstance(instance);
+ windowFireSource.registFireWindowInstanceIfNotExist(instance, this);
+ hasFinished = false;
+ break;
+ }
+ fireWindowValues.add((WindowValue)windowBaseValue);
+ if (fireWindowValues.size() >= windowCache.getBatchSize()) {
+ sendFireMessage(fireWindowValues, queueId);
+ fireCount += fireWindowValues.size();
+ clearWindowValues(fireWindowValues, queueId, instance);
+ fireWindowValues = new ArrayList<>();
+ }
+
+ }
+ if (fireWindowValues.size() > 0) {
+ sendFireMessage(fireWindowValues, queueId);
+ fireCount += fireWindowValues.size();
+ clearWindowValues(fireWindowValues, queueId, instance);
+ }
+ if (hasFinished) {
+ this.windowInstanceMap.remove(instance.createWindowInstanceId());
+ }
+
+ //}
+
+ return fireCount;
+ }
+
+ @Override
+ protected void saveStorage(Map<String, WindowBaseValue> newWindowValues, Map<String, WindowBaseValue> exisitWindowValues, WindowInstance windowInstance, String queueId) {
+ List<String> oldKeys = new ArrayList<>();
+ Map<String, WindowBaseValue> partionNumOrders = new HashMap<>();//需要基于key前缀排序partitionnum
+ for (WindowBaseValue windowBaseValue : exisitWindowValues.values()) {
+ WindowValue windowValue = (WindowValue)windowBaseValue;
+ String oldPartitionNumKey = createStoreKey(getOrderBypPrefix() + queueId, MapKeyUtil.createKey(getOrderBypFieldName(windowValue), windowValue.getGroupBy()), windowInstance);
+
+ windowBaseValue.setPartitionNum(createPartitionNum((WindowValue)windowBaseValue, queueId, windowInstance));
+ windowBaseValue.setFireTime(createSessionFireTime(windowValue.getPartition(), windowValue.getLastUpdateTime()));
+ String partitionNumKey = createStoreKey(getOrderBypPrefix() + queueId, MapKeyUtil.createKey(getOrderBypFieldName(windowValue), windowValue.getGroupBy()), windowInstance);
+ if (!partitionNumKey.equals(oldPartitionNumKey)) {
+ oldKeys.add(oldPartitionNumKey);
+ partionNumOrders.put(partitionNumKey, windowValue);
+ }
+
+ }
+ //
+ for (WindowBaseValue windowBaseValue : newWindowValues.values()) {
+ WindowValue windowValue = (WindowValue)windowBaseValue;
+ String oldPartitionNumKey = createStoreKey(getOrderBypPrefix() + queueId, MapKeyUtil.createKey(getOrderBypFieldName(windowValue), windowValue.getGroupBy()), windowInstance);
+ windowBaseValue.setFireTime(createSessionFireTime(windowValue.getPartition(), windowValue.getLastUpdateTime()));
+ String partitionNumKey = createStoreKey(getOrderBypPrefix() + queueId, MapKeyUtil.createKey(getOrderBypFieldName(windowValue), windowValue.getGroupBy()), windowInstance);
+ if (!partitionNumKey.equals(oldPartitionNumKey)) {
+ oldKeys.add(oldPartitionNumKey);
+ partionNumOrders.put(partitionNumKey, windowValue);
+ }
+ }
+ this.storage.getLocalStorage().removeKeys(oldKeys);
+ storage.multiPut(exisitWindowValues);
+ storage.multiPut(newWindowValues);
+ storage.multiPut(partionNumOrders);
+ }
+
+ @Override
+ public List<WindowInstance> queryOrCreateWindowInstance(IMessage message, String queueId) {
+ Long occurTime = WindowInstance.getOccurTime(this, message);
+ Date fireTime = createSessionFireDate(queueId, occurTime);
+ WindowInstance windowInstance = this.createWindowInstance("2020-01-01 00:00:00", "2999-01-01 00:00:00", DateUtil.format(fireTime), queueId);
+ WindowInstance existWindowInstance = this.getWindowInstanceMap().get(windowInstance.createWindowInstanceId());
+ if (existWindowInstance != null) {
+ Date windowInstanceFireTime = DateUtil.parse(existWindowInstance.getFireTime());
+ boolean hasFired = false;
+ while (WindowInstance.getOccurTime(this, message) - windowInstanceFireTime.getTime() > 0) {
+ hasFired = true;
+ System.out.println(DateUtil.format(new Date(WindowInstance.getOccurTime(this, message))));
+ existWindowInstance.setLastMaxUpdateTime(WindowInstance.getOccurTime(this, message));
+ this.windowFireSource.executeFireTask(existWindowInstance, true);
+ existWindowInstance = this.getWindowInstanceMap().get(windowInstance.createWindowInstanceId());
+ if (existWindowInstance == null) {
+ break;
+ }
+ windowInstanceFireTime = DateUtil.parse(existWindowInstance.getFireTime());
+
+ }
+ if (existWindowInstance != null) {
+ windowInstance = existWindowInstance;
+ }
+ if (hasFired) {
+ windowInstance.setFireTime(DateUtil.format(fireTime));
+ }
+
+ } else {
+ windowInstance.setNewWindowInstance(true);
+ windowInstance.setFireTime(DateUtil.format(fireTime));
+ this.getWindowInstanceMap().put(windowInstance.createWindowInstanceId(), windowInstance);
+ }
+ List<WindowInstance> windowInstances = new ArrayList<>();
+ windowInstances.add(windowInstance);
+ return windowInstances;
+ }
+
+ /**
+ * clear has fired window value
+ *
+ * @param windowValues
+ * @param queueId
+ * @param instance
+ */
+ protected void clearWindowValues(List<WindowValue> windowValues, String queueId, WindowInstance instance) {
+ if (windowValues == null || windowValues.size() == 0) {
+ return;
+ }
+ Set<String> deleteKeys = new HashSet<>();
+ List<String> msgKeys = new ArrayList<>();
+ for (WindowValue windowValue : windowValues) {
+ String storeKey = createStoreKey(queueId, windowValue.getGroupBy(), instance);
+ String partitionNumKey = createStoreKey(getOrderBypPrefix() + queueId, MapKeyUtil.createKey(getOrderBypFieldName(windowValue), windowValue.getGroupBy()) + "", instance);
+ deleteKeys.add(storeKey);
+ deleteKeys.add(partitionNumKey);
+ msgKeys.add(windowValue.getMsgKey());
+ }
+ String sql = "delete from window_value where msg_key in(" + SQLUtil.createInSql(msgKeys) + ")";
+ DriverBuilder.createDriver().execute(sql);
+ storage.getLocalStorage().removeKeys(deleteKeys);
+ }
+
+ protected static String getOrderBypFieldName(WindowValue windowValue) {
+ return MapKeyUtil.createKey(windowValue.getFireTime(), windowValue.getPartitionNum() + "");
+ }
+
+ /**
+ * create min session fire time, the current time+window size
+ *
+ * @param splitId
+ * @param occurTime
+ * @return
+ */
+ protected String createSessionFireTime(String splitId, Long occurTime) {
+ Date newFireTime = createSessionFireDate(splitId, occurTime);
+ return DateUtil.format(newFireTime);
+ }
+
+ /**
+ * 获取实际触发的时间，增加了固定延迟和water marker
+ *
+ * @param windowInstance
+ * @return
+ */
+ @Override
+ public Date getRealFireTime(WindowInstance windowInstance) {
+ Date fireDate = DateUtil.parse(windowInstance.getFireTime());
+ return fireDate;
+ }
+
+ /**
+ * create min session fire time, the current time+window size
+ *
+ * @param splitId
+ * @param lastUpdateTime
+ * @return
+ */
+ protected Date createSessionFireDate(String splitId, Long lastUpdateTime) {
+ if (lastUpdateTime == null) {
+ lastUpdateTime = this.getWindowMaxValueManager().updateWindowEventTime(splitId, (Long)null);
+ }
+ Date currentDate = new Date(lastUpdateTime);
+ Date newFireTime = DateUtil.addSecond(currentDate, this.sizeInterval * this.timeUnitAdjust);
+ return newFireTime;
+ }
+
+ @Override
+ public long incrementAndGetSplitNumber(WindowInstance instance, String shuffleId) {
+ long numer = super.incrementAndGetSplitNumber(instance, shuffleId);
+ if (numer > 900000000) {
+ this.getWindowMaxValueManager().resetSplitNum(instance, shuffleId);
+ }
+ return numer;
+ }
+}
diff --git a/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/operator/impl/WindowOperator.java b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/operator/impl/WindowOperator.java
new file mode 100644
index 0000000..aaddfb0
--- /dev/null
+++ b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/operator/impl/WindowOperator.java
@@ -0,0 +1,451 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.window.operator.impl;
+
+import org.apache.rocketmq.streams.common.channel.split.ISplit;
+import org.apache.rocketmq.streams.common.context.IMessage;
+import org.apache.rocketmq.streams.common.utils.CollectionUtil;
+import org.apache.rocketmq.streams.common.utils.DateUtil;
+import org.apache.rocketmq.streams.common.utils.MapKeyUtil;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+import org.apache.rocketmq.streams.db.driver.batchloader.IRowOperator;
+import org.apache.rocketmq.streams.db.driver.orm.ORMUtil;
+import org.apache.rocketmq.streams.window.model.WindowInstance;
+import org.apache.rocketmq.streams.window.operator.AbstractShuffleWindow;
+import org.apache.rocketmq.streams.window.operator.AbstractWindow;
+import org.apache.rocketmq.streams.window.state.WindowBaseValue;
+import org.apache.rocketmq.streams.window.state.impl.WindowValue;
+import org.apache.rocketmq.streams.window.storage.IWindowStorage;
+import org.apache.rocketmq.streams.window.storage.ShufflePartitionManager;
+import org.apache.rocketmq.streams.window.storage.WindowStorage.WindowBaseValueIterator;
+
+import java.util.*;
+import java.util.Map.Entry;
+
+public class WindowOperator extends AbstractShuffleWindow {
+
+ private static final String ORDER_BY_SPLIT_NUM="_order_by_split_num_";//key=_order;queueid,windowinstanceid,partitionNum
+
+ public WindowOperator() {
+ super();
+ }
+
+ @Deprecated
+ public WindowOperator(String timeFieldName, int windowPeriodMinute) {
+ super();
+ super.timeFieldName = timeFieldName;
+ super.sizeInterval = windowPeriodMinute;
+ }
+
+ @Deprecated
+ public WindowOperator(String timeFieldName, int windowPeriodMinute, String calFieldName) {
+ super();
+ super.timeFieldName = timeFieldName;
+ super.sizeInterval = windowPeriodMinute;
+ }
+
+ public WindowOperator(int sizeInterval, String groupByFieldName, Map<String, String> select) {
+ this.sizeInterval = sizeInterval;
+ this.slideInterval = sizeInterval;
+ this.groupByFieldName = groupByFieldName;
+ this.setSelectMap(select);
+ }
+
+ // protected transient AtomicInteger shuffleCount=new AtomicInteger(0);
+ //protected transient AtomicInteger fireCountAccumulator=new AtomicInteger(0);
+ @Override
+ public int fireWindowInstance(WindowInstance instance, String queueId,Map<String,String> queueId2Offset) {
+ List<WindowValue> windowValues=new ArrayList<>();
+ int fireCount=0;
+ //Set<String> currentQueueIds=new HashSet<>(queueIds);
+ long startTime=System.currentTimeMillis();
+ int sendCost=0;
+ int currentCount=0;
+ //for(String queueId:currentQueueIds){
+ WindowBaseValueIterator<WindowBaseValue> it = storage.loadWindowInstanceSplitData(getOrderBypPrefix(),queueId,instance.createWindowInstanceId(),null,getWindowBaseValueClass());
+ if(queueId2Offset!=null){
+ String offset=queueId2Offset.get(queueId);
+ if(StringUtil.isNotEmpty(offset)){
+ it.setPartitionNum(Long.valueOf(offset));
+ }
+ }
+ while (it.hasNext()){
+ WindowBaseValue windowBaseValue=it.next();
+ if(windowBaseValue==null){
+ continue;
+ }
+ //WindowValue windowValue=(WindowValue)windowBaseValue;
+ //Integer currentValue=(Integer)windowValue.getComputedColumnResultByKey("total");
+ //if(currentValue==null){
+ // currentValue=0;
+ //}
+ // fireCountAccumulator.addAndGet(currentValue);
+ // WindowValue windowValue=(WindowValue)windowBaseValue;
+ windowValues.add((WindowValue)windowBaseValue);
+ if(windowValues.size()>=windowCache.getBatchSize()){
+ long sendFireCost=System.currentTimeMillis();
+ sendFireMessage(windowValues,queueId);
+ sendCost+=(System.currentTimeMillis()-sendFireCost);
+ fireCount+=windowValues.size();
+ windowValues=new ArrayList<>();
+ }
+
+ }
+ if(windowValues.size()>0){
+ long sendFireCost=System.currentTimeMillis();
+ sendFireMessage(windowValues,queueId);
+ sendCost+=(System.currentTimeMillis()-sendFireCost);
+ fireCount+=windowValues.size();
+ }
+ // }
+
+ //if(fireCount==0){
+ // System.out.println(shuffleWindowInstanceId2MsgCount.get(instance.createWindowInstanceId()));
+ //}
+ // System.out.println("fire count is "+fireCountAccumulator.get());
+ //long clearStart=System.currentTimeMillis();
+ clearFire(instance);
+ // System.out.println("=============== fire cost is "+(System.currentTimeMillis()-startTime)+"send cost is "+sendCost+" clear cost is "+(System.currentTimeMillis()-clearStart));
+ return fireCount;
+ }
+
+ protected transient Map<String,Integer> shuffleWindowInstanceId2MsgCount=new HashMap<>();
+
+ @Override
+ public void shuffleCalculate(List<IMessage> messages, WindowInstance instance, String queueId) {
+ Map<String,List<IMessage>> groupBy=groupByGroupName(messages);
+ Set<String> groupByKeys=groupBy.keySet();
+ List<String> storeKeys=new ArrayList<>();
+ for(String groupByKey:groupByKeys){
+ String storeKey=createStoreKey(queueId,groupByKey,instance);
+ storeKeys.add(storeKey);
+ }
+ Map<String,WindowBaseValue> exisitWindowValues=new HashMap<>();
+ Map<String,WindowBaseValue> newWindowValues=new HashMap<>();
+
+ //从存储中，查找window value对象，value是对象的json格式
+ Map<String, WindowBaseValue> key2WindowValues=storage.multiGet(getWindowBaseValueClass(),storeKeys,instance.createWindowInstanceId(),queueId);
+ Iterator<Entry<String, List<IMessage>>> it = groupBy.entrySet().iterator();
+ while (it.hasNext()){
+ Entry<String, List<IMessage>> entry=it.next();
+ String groupByKey=entry.getKey();
+ String storeKey=createStoreKey(queueId,groupByKey,instance);
+ WindowValue windowValue=(WindowValue)key2WindowValues.get(storeKey);
+ List<IMessage> msgs=entry.getValue();
+ if(windowValue==null){
+ windowValue=createWindowValue(queueId,groupByKey,instance);
+ newWindowValues.put(storeKey,windowValue);
+ }else {
+ exisitWindowValues.put(storeKey,windowValue);
+ }
+
+ windowValue.incrementUpdateVersion();
+ //Integer origValue=(Integer)windowValue.getComputedColumnResultByKey("total");
+ //if(origValue==null){
+ // origValue=0;
+ //}
+ if(msgs!=null){
+ for(IMessage message:msgs){
+ windowValue.calculate(this,message);
+ }
+ }
+ //
+ //Integer currentValue=(Integer)windowValue.getComputedColumnResultByKey("total");
+ //if(currentValue==null){
+ // currentValue=0;
+ //}
+ //shuffleCount.addAndGet(-origValue);
+ //shuffleCount.addAndGet(currentValue);
+ //System.out.println("==========shuffle count is "+shuffleCount.get());
+
+ }
+
+ saveStorage(newWindowValues,exisitWindowValues,instance,queueId);
+ //Integer count=shuffleWindowInstanceId2MsgCount.get(instance.createWindowInstanceId());
+ //if(count==null){
+ // count=0;
+ //}
+ //count+=messages.size();
+ //shuffleWindowInstanceId2MsgCount.put(instance.createWindowInstanceId(),count);
+ }
+
+ protected void saveStorage(Map<String, WindowBaseValue> newWindowValues, Map<String, WindowBaseValue> exisitWindowValues,WindowInstance windowInstance,String queueId) {
+ String windowInstanceId=windowInstance.createWindowInstanceId();
+ Map<String, WindowBaseValue> allWindowValues=new HashMap<>();
+ allWindowValues.putAll(newWindowValues);
+ allWindowValues.putAll(exisitWindowValues);
+ storage.multiPut(allWindowValues,windowInstanceId,queueId);
+ Map<String,WindowBaseValue> partionNumOrders=new HashMap<>();//需要基于key前缀排序partitionnum
+ for(WindowBaseValue windowBaseValue:allWindowValues.values()){
+ WindowValue windowValue=(WindowValue)windowBaseValue;
+ String partitionNumKey=createStoreKey(getOrderBypPrefix()+queueId,MapKeyUtil.createKey(getOrderBypFieldName(windowValue),windowValue.getGroupBy()),windowInstance);
+ partionNumOrders.put(partitionNumKey,windowValue);
+ }
+ storage.getLocalStorage().multiPut(partionNumOrders);
+ }
+
+ @Override
+ public Class getWindowBaseValueClass() {
+ return WindowValue.class;
+ }
+
+ /**
+ * 按group name 进行分组
+ *
+ * @param messages
+ * @return
+ */
+ protected Map<String, List<IMessage>> groupByGroupName(List<IMessage> messages) {
+ if(messages==null||messages.size()==0){
+ return new HashMap<>();
+ }
+ Map<String,List<IMessage>> groupBy=new HashMap<>();
+ for(IMessage message:messages){
+ String groupByValue=generateShuffleKey(message);
+ if(StringUtil.isEmpty(groupByValue)){
+ groupByValue="<null>";
+ }
+ List<IMessage> messageList=groupBy.get(groupByValue);
+ if(messageList==null){
+ messageList=new ArrayList<>();
+ groupBy.put(groupByValue,messageList);
+ }
+ messageList.add(message);
+ }
+ return groupBy;
+ }
+
+ /**
+ * 创建新的window value对象
+ *
+ * @param groupBy
+ * @param instance
+ * @return
+ */
+ protected WindowValue createWindowValue(String queueId,String groupBy, WindowInstance instance) {
+ WindowValue windowValue = new WindowValue();
+ windowValue.setNameSpace(getNameSpace());
+ windowValue.setConfigureName(getConfigureName());
+ windowValue.setStartTime(instance.getStartTime());
+ windowValue.setEndTime(instance.getEndTime());
+ windowValue.setFireTime(instance.getFireTime());
+ windowValue.setGroupBy(groupBy==null?"":groupBy);
+ windowValue.setMsgKey(StringUtil.createMD5Str(MapKeyUtil.createKey(queueId, instance.createWindowInstanceId(), groupBy)));
+ String shuffleId=shuffleChannel.getChannelQueue(groupBy).getQueueId();
+ windowValue.setPartitionNum(createPartitionNum(windowValue,queueId,instance));
+ windowValue.setPartition(shuffleId);
+ windowValue.setWindowInstancePartitionId(instance.getWindowInstanceKey());
+ windowValue.setWindowInstanceId(instance.getWindowInstanceKey());
+ return windowValue;
+
+ }
+
+ protected long createPartitionNum(WindowValue windowValue, String shuffleId, WindowInstance instance) {
+ return incrementAndGetSplitNumber(instance,shuffleId);
+ }
+
+ /**
+ * 创建存储key
+ *
+ * @param groupByKey
+ * @param windowInstance
+ * @return
+ */
+ protected static String createStoreKey(String shuffleId, String groupByKey,WindowInstance windowInstance){
+ String storeKey = MapKeyUtil.createKey(shuffleId, windowInstance.createWindowInstanceId(), groupByKey);
+ return storeKey;
+ }
+
+ /**
+ * 需要排序的前缀
+ *
+ * @return
+ */
+ protected static String getOrderBypPrefix(){
+ return ORDER_BY_SPLIT_NUM;
+ }
+
+ /**
+ * 需要排序的字段值
+ *
+ * @return
+ */
+ protected static String getOrderBypFieldName(WindowValue windowValue){
+ return windowValue.getPartitionNum()+"";
+ }
+
+ /**
+ * 删除掉触发过的数据
+ *
+ * @param windowInstances
+ */
+ @Override
+ public void clearFire(List<WindowInstance> windowInstances) {
+ //clear window instances in db
+
+ //clear window value in db
+ for (WindowInstance windowInstance : windowInstances) {
+ Set<String> patitionNums=new HashSet<>();
+ patitionNums.add(getOrderBypPrefix()+ windowInstance.getSplitId());
+ Set<String> queueIds=new HashSet<>();
+ queueIds.add(windowInstance.getSplitId());
+ boolean canClear=false;
+ if(fireMode!=2){
+ canClear=true;
+ }
+ if(fireMode==2){
+ Date endTime=DateUtil.parse(windowInstance.getEndTime());
+ Date lastDate = DateUtil.addSecond(endTime, waterMarkMinute * timeUnitAdjust);
+
+ if((windowInstance.getLastMaxUpdateTime()>lastDate.getTime())){
+ canClear=true;
+ }
+ }
+
+ if(canClear){
+ this.windowInstanceMap.remove(windowInstance.createWindowInstanceId());
+ if(!isLocalStorageOnly){
+ WindowInstance.clearInstances(windowInstances);
+ }
+ shuffleChannel.clearCache(windowInstance);
+ ShufflePartitionManager.getInstance().clearWindowInstance(windowInstance.createWindowInstanceId());
+ storage.delete(windowInstance.createWindowInstanceId(),queueIds,getWindowBaseValueClass());
+ storage.getLocalStorage().delete(windowInstance.createWindowInstanceId(),patitionNums,getWindowBaseValueClass());
+ }
+
+ }
+
+ }
+
+ @Override
+ public void clearCache(String queueId) {
+ getStorage().clearCache(shuffleChannel.getChannelQueue(queueId),getWindowBaseValueClass());
+ getStorage().clearCache(getOrderByQueue(queueId,getOrderBypPrefix()),getWindowBaseValueClass());
+ ShufflePartitionManager.getInstance().clearSplit(queueId);
+ }
+
+ public ISplit getOrderByQueue(String key,String prefix){
+ int index=shuffleChannel.hash(key);
+ ISplit targetQueue = shuffleChannel.getQueueList().get(index);
+ return new ISplit() {
+ @Override
+ public String getQueueId() {
+ return prefix+targetQueue.getQueueId();
+ }
+
+ @Override
+ public String getPlusQueueId() {
+ return prefix+targetQueue.getPlusQueueId();
+ }
+
+ @Override
+ public Object getQueue() {
+ return targetQueue.getQueue();
+ }
+
+ @Override
+ public int compareTo(Object o) {
+ return targetQueue.compareTo(o);
+ }
+
+ @Override
+ public String toJson() {
+ return targetQueue.toJson();
+ }
+
+ @Override
+ public void toObject(String jsonString) {
+ targetQueue.toObject(jsonString);
+ }
+ };
+ }
+
+ public static void compareAndSet(WindowInstance windowInstance, IWindowStorage storage,List<WindowValue> windowValues){
+ if(windowValues==null||storage==null){
+ return;
+ }
+ synchronized (storage){
+ List<String> storeKeys = new ArrayList<>();
+ Map<String,WindowValue> windowValueMap=new HashMap<>();
+ for(WindowValue windowValue:windowValues){
+ String storeKey=createStoreKey(windowValue.getPartition(),windowValue.getGroupBy(),windowInstance);
+ storeKeys.add(storeKey);
+ windowValueMap.put(storeKey,windowValue);
+ String storeOrderKey=createStoreKey(windowValue.getPartition(),windowValue.getPartitionNum()+"",windowInstance);
+ windowValueMap.put(storeOrderKey,windowValue);
+ }
+ Map<String, WindowBaseValue> valueMap = storage.multiGet(WindowValue.class,storeKeys);
+ if(valueMap==null||valueMap.size()==0){
+ storage.multiPut(windowValueMap);
+ return;
+ }
+ Iterator<Entry<String, WindowBaseValue>> it = valueMap.entrySet().iterator();
+
+ while (it.hasNext()){
+ Entry<String, WindowBaseValue> entry=it.next();
+ String storeKey=entry.getKey();
+ WindowBaseValue localValue=entry.getValue();
+ WindowValue windowValue=windowValueMap.get(storeKey);
+ if (windowValue.getUpdateVersion() <= localValue.getUpdateVersion()) {
+ windowValueMap.remove(storeKey);
+ }
+ }
+ if(CollectionUtil.isNotEmpty(windowValueMap)){
+ storage.multiPut(windowValueMap);
+ }
+ }
+ }
+
+ public static class WindowRowOperator implements IRowOperator {
+
+ protected WindowInstance windowInstance;
+ protected String spiltId;
+ protected AbstractWindow window;
+
+ public WindowRowOperator(WindowInstance windowInstance,String spiltId,AbstractWindow window){
+ this.windowInstance=windowInstance;
+ this.spiltId=spiltId;
+ this.window=window;
+ }
+
+ @Override
+ public synchronized void doProcess(Map<String, Object> row) {
+ WindowValue windowValue = ORMUtil.convert(row, WindowValue.class);
+ List<String> keys = new ArrayList<>();
+ String storeKey=createStoreKey(spiltId,windowValue.getGroupBy(),windowInstance);
+ keys.add(storeKey);
+ String storeOrderKey=createStoreKey(getOrderBypPrefix()+windowValue.getPartition(),MapKeyUtil.createKey(getOrderBypFieldName(windowValue),windowValue.getGroupBy()),windowInstance);
+ Map<String, WindowBaseValue> valueMap = window.getStorage().getLocalStorage().multiGet(WindowValue.class,keys);
+ if (CollectionUtil.isEmpty(valueMap)) {
+ Map<String, WindowBaseValue> map = new HashMap<>(4);
+
+ map.put(storeKey, windowValue);
+ map.put(storeOrderKey, windowValue);
+ window.getStorage().getLocalStorage().multiPut(map);
+ return;
+ }
+ WindowValue localValue = (WindowValue)valueMap.values().iterator().next();
+ if (windowValue.getUpdateVersion() > localValue.getUpdateVersion()) {
+ Map<String, WindowBaseValue> map = new HashMap<>();
+ map.put(storeKey, windowValue);
+ map.put(storeOrderKey, windowValue);
+ window.getStorage().getLocalStorage().multiPut(map);
+ }
+ }
+ }
+
+}
diff --git a/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/operator/join/DBOperator.java b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/operator/join/DBOperator.java
new file mode 100644
index 0000000..b8bab5d
--- /dev/null
+++ b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/operator/join/DBOperator.java
@@ -0,0 +1,427 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.window.operator.join;
+
+import com.alibaba.fastjson.JSONArray;
+import com.alibaba.fastjson.JSONObject;
+import org.apache.rocketmq.streams.common.context.IMessage;
+import org.apache.rocketmq.streams.common.context.Message;
+import org.apache.rocketmq.streams.common.context.MessageHeader;
+import org.apache.rocketmq.streams.common.utils.DateUtil;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.rocketmq.streams.db.driver.orm.ORMUtil;
+import org.apache.rocketmq.streams.window.state.impl.JoinLeftState;
+import org.apache.rocketmq.streams.window.state.impl.JoinRightState;
+import org.apache.rocketmq.streams.window.state.impl.JoinState;
+
+import java.security.MessageDigest;
+import java.util.*;
+import java.util.concurrent.TimeUnit;
+
+public class DBOperator implements Operator {
+
+ private static final Log LOG = LogFactory.getLog(DBOperator.class);
+
+ /**
+ * 根据join流对原始数据进行处理并入库
+ *
+ * @param messageList
+ * @param joinType
+ */
+ public void addMessage(List<IMessage> messageList, String joinType) {
+ List<JoinLeftState> joinLeftStates = new ArrayList<JoinLeftState>();
+ List<JoinRightState> joinRightStates = new ArrayList<JoinRightState>();
+ for (IMessage message : messageList) {
+ String routeLabel = message.getHeader().getMsgRouteFromLable();
+ JoinState state = dealMessge(message, routeLabel);
+ if ("left".equalsIgnoreCase(routeLabel)) {
+ joinLeftStates.add((JoinLeftState)state);
+ } else if ("right".equalsIgnoreCase(routeLabel)) {
+ joinRightStates.add((JoinRightState)state);
+ }
+ // joinStates.add(state);
+ }
+
+ if (joinLeftStates.size() > 0) {
+ ORMUtil.batchReplaceInto(joinLeftStates);
+ }
+ if (joinRightStates.size() > 0) {
+ ORMUtil.batchReplaceInto(joinRightStates);
+ }
+
+ }
+
+ /**
+ * 生成joinstate对象
+ *
+ * @param message
+ * @param routeLabel
+ * @return
+ */
+ public JoinState dealMessge(IMessage message, String routeLabel) {
+
+ JSONObject window = message.getMessageBody().getJSONObject("Window");
+ String windowName = window.getString("configureName");
+ String windowNameSpace = window.getString("nameSpace");
+ MessageHeader header = message.getHeader();
+ String queueId = "_Dipper";
+ String offset = System.nanoTime() + "";
+ if (header.getQueueId() != null) {
+ queueId = header.getQueueId();
+ }
+ if (header.isEmptyOffset()) {
+ header.setOffset(offset);
+ offset = header.getOffset();
+ }
+
+ String messageId = windowNameSpace + "_" + windowName + "_" + queueId + "_" + offset;
+
+ List<String> leftJoinFieldNames = window.getJSONArray("leftJoinFieldNames") != null ?
+ window.getJSONArray("leftJoinFieldNames").toJavaList(String.class) :
+ new ArrayList<>();
+ List<String> rightJoinFieldNames = window.getJSONArray("rightJoinFieldNames") != null ?
+ window.getJSONArray("rightJoinFieldNames").toJavaList(String.class) :
+ new ArrayList<>();
+
+ String messageKey = generateKey(message.getMessageBody(), routeLabel, leftJoinFieldNames, rightJoinFieldNames);
+ JSONObject messageBody = (JSONObject)message.getMessageBody().clone();
+ messageBody.remove("WindowInstance");
+ messageBody.remove("Window");
+
+ JoinState state = null;
+ if ("left".equalsIgnoreCase(routeLabel)) {
+ state = new JoinLeftState();
+ } else if ("right".equalsIgnoreCase(routeLabel)) {
+ state = new JoinRightState();
+ }
+
+ state.setGmtCreate(new Date());
+ state.setGmtModified(new Date());
+ state.setWindowName(windowName);
+ state.setWindowNameSpace(windowNameSpace);
+ state.setMessageId(messageId);
+ state.setMessageKey(messageKey);
+ state.setMessageTime(new Date());
+ state.setMessageBody(messageBody.toJSONString());
+
+ return state;
+ }
+
+ /**
+ * 根据join条件生成消息比对key值
+ *
+ * @param messageBody
+ * @param joinLabel
+ * @param leftJoinFieldNames
+ * @param rightJoinFieldNames
+ * @return
+ */
+ public static String generateKey(JSONObject messageBody, String joinLabel, List<String> leftJoinFieldNames, List<String> rightJoinFieldNames) {
+ StringBuffer buffer = new StringBuffer();
+ if ("left".equalsIgnoreCase(joinLabel)) {
+ for (String field : leftJoinFieldNames) {
+ String value = messageBody.getString(field);
+ buffer.append(value).append("_");
+ }
+ } else {
+ for (String field : rightJoinFieldNames) {
+ String[] rightFields = field.split("\\.");
+ if (rightFields.length > 1) {
+ field = rightFields[1];
+ }
+ String value = messageBody.getString(field);
+ buffer.append(value).append("_");
+ }
+ }
+
+ return MD5(buffer.toString());
+ }
+
+ public static String MD5(String s) {
+ char hexDigits[] = {'0', '1', '2', '3', '4', '5', '6', '7', '8', '9', 'A', 'B', 'C', 'D', 'E', 'F'};
+
+ try {
+ byte[] btInput = s.getBytes();
+ // 获得MD5摘要算法的 MessageDigest 对象
+ MessageDigest mdInst = MessageDigest.getInstance("MD5");
+ // 使用指定的字节更新摘要
+ mdInst.update(btInput);
+ // 获得密文
+ byte[] md = mdInst.digest();
+ // 把密文转换成十六进制的字符串形式
+ int j = md.length;
+ char str[] = new char[j * 2];
+ int k = 0;
+ for (int i = 0; i < j; i++) {
+ byte byte0 = md[i];
+ str[k++] = hexDigits[byte0 >>> 4 & 0xf];
+ str[k++] = hexDigits[byte0 & 0xf];
+ }
+ return new String(str);
+ } catch (Exception e) {
+ e.printStackTrace();
+ throw new RuntimeException(e.getMessage(), e);
+ }
+ }
+
+ public List<Map<String, Object>> dealJoin(IMessage message) {
+ List<Map<String, Object>> result = new ArrayList<>();
+ JSONObject messageBody = message.getMessageBody();
+ JSONObject msg = messageBody.getJSONObject("msg");
+ String routeLabel = messageBody.getString("routeLabel");
+ JSONArray windowInstances = msg.getJSONArray("WindowInstance");
+ JSONObject windowInstance = null;
+ if (windowInstances != null && windowInstances.size() > 0) {
+ windowInstance = windowInstances.getJSONObject(0);
+ } else {
+ return result;
+ }
+
+ Integer retainWindowCount = messageBody.getInteger("retainWindowCount");
+ Integer sizeInterval = messageBody.getInteger("sizeInterval");
+
+ List<JSONObject> tmp = new ArrayList<>();
+ if ("left".equalsIgnoreCase(routeLabel)) {
+ String endTime = windowInstance.getString("endTime");
+ String startTime = addTime(windowInstance.getString("startTime"), TimeUnit.MINUTES, -retainWindowCount * sizeInterval);
+ String tableName = "join_right_state";
+ String messageKey = messageBody.getString("messageKey");
+ String windowName = windowInstance.getString("windowName");
+ String windowNameSpace = windowInstance.getString("windowNameSpace");
+ tmp = getJoinData(tableName, messageKey, windowName, windowNameSpace, startTime, endTime);
+
+ } else if ("right".equalsIgnoreCase(routeLabel)) {
+ // String startTime = addTime(windowInstance.getString("startTime"), TimeUnit.MINUTES, -sizeInterval);
+ String startTime = addTime(windowInstance.getString("startTime"), TimeUnit.MINUTES, -retainWindowCount * sizeInterval);
+ String endTime = addTime(windowInstance.getString("endTime"), TimeUnit.MINUTES, -sizeInterval);
+ String tableName = "join_left_state";
+ String messageKey = messageBody.getString("messageKey");
+ String windowName = windowInstance.getString("windowName");
+ String windowNameSpace = windowInstance.getString("windowNameSpace");
+ tmp = getJoinData(tableName, messageKey, windowName, windowNameSpace, startTime, endTime);
+ }
+
+ result = converToMapFromJson(tmp);
+ return result;
+
+ }
+
+ public List<JSONObject> connectJoin(IMessage message, List<Map<String, Object>> rows, String joinType, String rightAsName) {
+ List<JSONObject> result = new ArrayList<>();
+ if (rows.size() <= 0) {
+ return result;
+ }
+ if ("inner".equalsIgnoreCase(joinType)) {
+ result = connectInnerJoin(message, rows, rightAsName);
+ } else if ("left".equalsIgnoreCase(joinType)) {
+ result = connectLeftJoin(message, rows, rightAsName);
+ }
+ return result;
+ }
+
+ /**
+ * inner join 将匹配messageKey的各行与message进行连接
+ *
+ * @param message
+ * @param rows
+ * @return
+ */
+ public List<JSONObject> connectInnerJoin(IMessage message, List<Map<String, Object>> rows, String rightAsName) {
+ List<JSONObject> result = new ArrayList<>();
+ String routeLabel = message.getMessageBody().getString("routeLabel");
+ if ("left".equalsIgnoreCase(routeLabel)) {
+ JSONObject messageBody = message.getMessageBody().getJSONObject("msg");
+ for (Map<String, Object> raw : rows) {
+ // addAsName(raw, rightAsName);
+ JSONObject object = (JSONObject)messageBody.clone();
+ object.fluentPutAll(addAsName(raw, rightAsName));
+ result.add(object);
+ }
+ } else {
+ JSONObject messageBody = message.getMessageBody().getJSONObject("msg");
+ messageBody = (JSONObject)addAsName(messageBody, rightAsName);
+ for (Map<String, Object> raw : rows) {
+ JSONObject object = (JSONObject)messageBody.clone();
+ object.fluentPutAll(raw);
+ result.add(object);
+ }
+ }
+
+ return result;
+ }
+
+ private Map<String, Object> addAsName(Map<String, Object> raw, String rightAsName) {
+ Map<String, Object> asName = new HashMap<>();
+ for (Map.Entry<String, Object> tmp : raw.entrySet()) {
+ asName.put(rightAsName + "." + tmp.getKey(), tmp.getValue());
+ // raw.remove(tmp.getKey());
+ }
+ return asName;
+ }
+
+ public List<JSONObject> connectLeftJoin(IMessage message, List<Map<String, Object>> rows, String rightAsName) {
+ List<JSONObject> result = new ArrayList<>();
+ String routeLabel = message.getMessageBody().getString("routeLabel");
+ JSONObject messageBody = message.getMessageBody().getJSONObject("msg");
+ if ("left".equalsIgnoreCase(routeLabel)) {
+ if (rows != null && rows.size() > 0) {
+ for (Map<String, Object> raw : rows) {
+ // raw = addAsName(raw, rightAsName);
+ JSONObject object = (JSONObject)messageBody.clone();
+ object.fluentPutAll(addAsName(raw, rightAsName));
+ result.add(object);
+ }
+ return result;
+ } else {
+ result.add(messageBody);
+ }
+
+ } else {
+ if (rows != null && rows.size() > 0) {
+ messageBody = (JSONObject)addAsName(messageBody, rightAsName);
+ for (Map<String, Object> raw : rows) {
+ JSONObject object = (JSONObject)messageBody.clone();
+ object.fluentPutAll(raw);
+ result.add(object);
+ }
+ return result;
+ }
+ }
+
+ return result;
+
+ }
+
+ private List<Map<String, Object>> converToMapFromJson(List<JSONObject> list) {
+ List<Map<String, Object>> mapList = new ArrayList<>();
+ if (list != null && list.size() > 0) {
+ for (JSONObject object : list) {
+ Map<String, Object> tmp = object;
+ mapList.add(tmp);
+ }
+ }
+ return mapList;
+ }
+
+ public List<JSONObject> getJoinData(String tableName, String messageKey, String windowName, String windowNameSpace, String startTime, String endTime) {
+ Map<String, Object> paras = new HashMap<>();
+ paras.put("messageKey", messageKey);
+ paras.put("startTime", startTime);
+ paras.put("endTime", endTime);
+ paras.put("windowName", windowName);
+ paras.put("windowNameSpace", windowNameSpace);
+ // paras.put("tableName", tableName);
+ List<JoinState> result = new ArrayList<>();
+ if ("join_right_state".equalsIgnoreCase(tableName)) {
+ result = ORMUtil.queryForList("select message_body from join_right_state where message_key = #{messageKey} and window_name = #{windowName}" +
+ "and window_name_space = #{windowNameSpace} and gmt_create >= #{startTime} and gmt_create < #{endTime}", paras, JoinState.class);
+ } else if ("join_left_state".equalsIgnoreCase(tableName)) {
+ result = ORMUtil.queryForList("select message_body from join_left_state where message_key = #{messageKey} and window_name = #{windowName} " +
+ "and window_name_space = #{windowNameSpace} and gmt_create >= #{startTime} and gmt_create < #{endTime}", paras, JoinState.class);
+ }
+
+ List<JSONObject> bodys = new ArrayList<>();
+ for (JoinState tmp : result) {
+ try {
+ bodys.add(Message.parseObject(tmp.getMessageBody()));
+ } catch (Exception e) {
+ LOG.error("json parase error:", e);
+ }
+
+ }
+ return bodys;
+ }
+
+ public String addTime(String time, TimeUnit unit, int value) {
+ Date date = DateUtil.parseTime(time);
+ date = DateUtil.addDate(unit, date, value);
+ return DateUtil.format(date);
+ }
+
+ /**
+ * 根据window去除过期消息数据,消息去除时间为starttime加上窗口
+ *
+ * @param windowNameSpace
+ * @param windowName
+ * @param retainWindowCount
+ * @param sizeInterval
+ * @param startTime
+ */
+ public void cleanMessage(String windowNameSpace, String windowName, int retainWindowCount, int sizeInterval, String startTime) {
+ Map<String, Object> params = new HashMap<>();
+ String start = addTime(startTime, TimeUnit.MINUTES, -retainWindowCount * sizeInterval);
+ params.put("startTime", start);
+ params.put("windowNameSpace", windowNameSpace);
+ params.put("windowName", windowName);
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("dboperata delete param is " + JSONObject.toJSONString(params));
+ }
+
+ List<JoinLeftState> joinLeftStates = ORMUtil.queryForList("select id from join_left_state where window_name_space = #{windowNameSpace} and " +
+ "window_name = #{windowName} and gmt_create < #{startTime}", params, JoinLeftState.class);
+ if (joinLeftStates != null && joinLeftStates.size() > 0) {
+ List<String> deleteIds = this.getDeleteIds(joinLeftStates);
+ for (String ids : deleteIds) {
+ // params.put("ids", ids);
+ ORMUtil.executeSQL("delete from join_left_state where id in (" + ids + ")", null);
+ }
+
+ }
+
+ List<JoinRightState> joinRightStates = ORMUtil.queryForList("select id from join_right_state where window_name_space = #{windowNameSpace} and " +
+ "window_name = #{windowName} and gmt_create < #{startTime}", params, JoinRightState.class);
+ if (joinRightStates != null && joinRightStates.size() > 0) {
+ List<String> deleteIds = this.getDeleteIds(joinRightStates);
+ for (String ids : deleteIds) {
+ // params.put("ids", ids);
+ ORMUtil.executeSQL("delete from join_right_state where id in (" + ids + ")", null);
+ }
+
+ }
+
+ }
+
+ private List<String> getDeleteIds(List<? extends JoinState> instances) {
+ List<String> deteleIds = new ArrayList<>();
+ if (instances == null || instances.size() == 0) {
+ return deteleIds;
+ }
+ int count = 1;
+
+ StringBuilder builder = new StringBuilder();
+ for (; count <= instances.size(); count++) {
+ builder.append(instances.get(count - 1).getId());
+ if (count % 1000 == 0) {
+ deteleIds.add(builder.toString());
+ builder = new StringBuilder();
+ } else {
+ if (count == instances.size()) {
+ deteleIds.add(builder.toString());
+ } else {
+ builder.append(",");
+ }
+
+ }
+ // count++;
+ }
+
+ return deteleIds;
+ }
+
+}
\ No newline at end of file
diff --git a/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/operator/join/JoinWindow.java b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/operator/join/JoinWindow.java
new file mode 100644
index 0000000..4124ec6
--- /dev/null
+++ b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/operator/join/JoinWindow.java
@@ -0,0 +1,553 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.window.operator.join;
+
+import java.util.*;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import com.alibaba.fastjson.JSONObject;
+
+import org.apache.rocketmq.streams.window.operator.AbstractShuffleWindow;
+import org.apache.rocketmq.streams.window.state.impl.JoinLeftState;
+import org.apache.rocketmq.streams.window.state.impl.JoinRightState;
+import org.apache.rocketmq.streams.window.state.impl.JoinState;
+import org.apache.rocketmq.streams.window.model.WindowInstance;
+import org.apache.rocketmq.streams.window.model.WindowCache;
+import org.apache.rocketmq.streams.window.state.WindowBaseValue;
+import org.apache.rocketmq.streams.dim.model.AbstractDim;
+import org.apache.rocketmq.streams.common.context.AbstractContext;
+import org.apache.rocketmq.streams.common.context.Context;
+import org.apache.rocketmq.streams.common.context.IMessage;
+import org.apache.rocketmq.streams.common.context.Message;
+import org.apache.rocketmq.streams.common.context.MessageHeader;
+import org.apache.rocketmq.streams.common.utils.DateUtil;
+import org.apache.rocketmq.streams.common.utils.MapKeyUtil;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+
+public class JoinWindow extends AbstractShuffleWindow {
+
+ public static final String JOIN_KEY = "JOIN_KEY";
+
+ public static final String LABEL_LEFT = "left";
+
+ public static final String LABEL_RIGHT = "right";
+
+ //保存多少个周期的数据。比如window的滚动周期是5分钟，join需要1个小时数据，则retainWindowCount=12
+ protected int retainWindowCount = 4;
+ protected List<String> leftJoinFieldNames;//join等值条件中，左流的字段列表
+ protected List<String> rightJoinFieldNames;//join等值条件中，右流的字段列表
+ protected String rightAsName;//主要用于sql场景，默认右流都需要有别名。开发模式不需要
+
+ protected String joinType;//join类型，值为INNER,LEFT
+ protected String expression;//条件表达式。在存在非等值比较时使用
+ protected transient DBOperator joinOperator = new DBOperator();
+
+ @Override
+ protected boolean initConfigurable() {
+ // return super.initConfigurable();
+ super.initConfigurable();
+ doProcessAfterRefreshConfigurable(null);
+ return true;
+ }
+
+ // @Override
+ // protected void addPropertyToMessage(IMessage oriMessage, JSONObject oriJson){
+ // oriJson.put("AbstractWindow", this);
+ //
+ // }
+
+ @Override
+ protected int fireWindowInstance(WindowInstance instance, String shuffleId, Map<String, String> queueId2Offsets) {
+ List<WindowInstance> instances = new ArrayList<>();
+ instances.add(instance);
+ clearFire(instances);
+ return 0;
+ }
+
+ @Override
+ public void clearCache(String queueId) {
+
+ }
+
+ @Override
+ public void shuffleCalculate(List<IMessage> messages, WindowInstance instance, String queueId) {
+ Map<String, WindowBaseValue> joinLeftStates = new HashMap<>();
+ Map<String, WindowBaseValue> joinRightStates = new HashMap<>();
+ for (IMessage msg : messages) {
+ MessageHeader header = JSONObject.parseObject(msg.getMessageBody().
+ getString(WindowCache.ORIGIN_MESSAGE_HEADER), MessageHeader.class);
+ msg.setHeader(header);
+ String routeLabel = header.getMsgRouteFromLable();
+ String storeKey = createStoreKey(msg, routeLabel, instance);
+ JoinState state = createJoinState(msg, instance, routeLabel);
+ if ("left".equalsIgnoreCase(routeLabel)) {
+ joinLeftStates.put(storeKey, state);
+ } else if ("right".equalsIgnoreCase(routeLabel)) {
+ joinRightStates.put(storeKey, state);
+ }
+ }
+
+ if (joinLeftStates.size() > 0) {
+ storage.multiPut(joinLeftStates);
+ }
+ if (joinRightStates.size() > 0) {
+ storage.multiPut(joinRightStates);
+ }
+
+ for (IMessage msg : messages) {
+ MessageHeader header = JSONObject.parseObject(msg.getMessageBody().
+ getString(WindowCache.ORIGIN_MESSAGE_HEADER), MessageHeader.class);
+ String routeLabel = header.getMsgRouteFromLable();
+ // Map<String,WindowBaseValue> joinMessages = new HashMap<>();
+ String storeKeyPrefix = "";
+ Iterator<WindowBaseValue> iterator = null;
+ if (LABEL_LEFT.equalsIgnoreCase(routeLabel)) {
+ storeKeyPrefix = createStoreKeyPrefix(msg, LABEL_RIGHT, instance);
+ iterator = getMessageIterator(queueId, instance, msg, storeKeyPrefix, JoinRightState.class);
+ } else if (LABEL_RIGHT.equalsIgnoreCase(routeLabel)) {
+ storeKeyPrefix = createStoreKeyPrefix(msg, LABEL_LEFT, instance);
+ iterator = getMessageIterator(queueId, instance, msg, storeKeyPrefix, JoinLeftState.class);
+ }
+
+ // Iterator<WindowBaseValue> iterator = getMessageIterator(queueId, instance, msg, storeKeyPrefix, JoinState.class);
+ List<WindowBaseValue> tmpMessages = new ArrayList<>();
+ int count = 0;
+ while (iterator.hasNext()) {
+ tmpMessages.add(iterator.next());
+ count++;
+ if (count == 100) {
+ sendMessage(msg, tmpMessages);
+ tmpMessages.clear();
+ count = 0;
+ }
+ }
+ sendMessage(msg, tmpMessages);
+
+ }
+ }
+
+ private Iterator<WindowBaseValue> getMessageIterator(String queueId, WindowInstance instance,
+ IMessage msg, String keyPrefix, Class<? extends WindowBaseValue> clazz) {
+
+ List<WindowInstance> instances = new ArrayList<>();
+ for (Map.Entry<String, WindowInstance> entry : this.windowInstanceMap.entrySet()) {
+ instances.add(entry.getValue());
+ }
+ Iterator<WindowInstance> windowInstanceIter = instances.iterator();
+ return new Iterator<WindowBaseValue>() {
+ protected volatile boolean hasNext = true;
+ protected AtomicBoolean hasInit = new AtomicBoolean(false);
+ protected Iterator<WindowBaseValue> iterator = null;
+
+ @Override
+ public boolean hasNext() {
+ if (iterator != null && iterator.hasNext()) {
+ return true;
+ }
+ while (windowInstanceIter.hasNext()) {
+ WindowInstance instance = windowInstanceIter.next();
+ iterator = storage.loadWindowInstanceSplitData(null, queueId,
+ instance.createWindowInstanceId(),
+ keyPrefix,
+ clazz);
+ if (iterator != null && iterator.hasNext()) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ @Override
+ public WindowBaseValue next() {
+ return iterator.next();
+ }
+ };
+
+ }
+
+ private Iterator<WindowBaseValue> getIterator(String queueId, String keyPrefix, WindowInstance instance, Class<? extends WindowBaseValue> clazz) {
+
+ List<WindowInstance> instances = new ArrayList<>();
+ for (Map.Entry<String, WindowInstance> entry : this.windowInstanceMap.entrySet()) {
+ instances.add(entry.getValue());
+ }
+ Iterator<WindowInstance> windowInstanceIter = instances.iterator();
+ return new Iterator<WindowBaseValue>() {
+ protected volatile boolean hasNext = true;
+ protected AtomicBoolean hasInit = new AtomicBoolean(false);
+ protected Iterator<WindowBaseValue> iterator = null;
+
+ @Override
+ public boolean hasNext() {
+ if (iterator != null && iterator.hasNext()) {
+ return true;
+ }
+ if (windowInstanceIter.hasNext()) {
+ WindowInstance instance = windowInstanceIter.next();
+ iterator = storage.loadWindowInstanceSplitData(null, queueId, instance.createWindowInstanceId(), keyPrefix, clazz);
+ if (iterator != null && iterator.hasNext()) {
+ return true;
+ } else {
+ return false;
+ }
+ }
+ return false;
+ }
+
+ @Override
+ public WindowBaseValue next() {
+ return iterator.next();
+ }
+ };
+
+ }
+
+ public List<JSONObject> connectJoin(IMessage message, List<Map<String, Object>> rows, String joinType, String rightAsName) {
+ List<JSONObject> result = new ArrayList<>();
+ if (rows.size() <= 0) {
+ return result;
+ }
+ if ("inner".equalsIgnoreCase(joinType)) {
+ result = connectInnerJoin(message, rows, rightAsName);
+ }
+ // else if ("left".equalsIgnoreCase(joinType)) {
+ // result = connectLeftJoin(message, rows, rightAsName);
+ // }
+ return result;
+ }
+
+ /**
+ * inner join 将匹配messageKey的各行与message进行连接
+ *
+ * @param message
+ * @param rows
+ * @return
+ */
+ public List<JSONObject> connectInnerJoin(IMessage message, List<Map<String, Object>> rows, String rightAsName) {
+ List<JSONObject> result = new ArrayList<>();
+ String routeLabel = message.getHeader().getMsgRouteFromLable();
+ if (LABEL_LEFT.equalsIgnoreCase(routeLabel)) {
+ JSONObject messageBody = message.getMessageBody();
+ for (Map<String, Object> raw : rows) {
+ // addAsName(raw, rightAsName);
+ JSONObject object = (JSONObject)messageBody.clone();
+ object.fluentPutAll(addAsName(raw, rightAsName));
+ result.add(object);
+ }
+ } else {
+ JSONObject messageBody = message.getMessageBody();
+ messageBody = addAsName(messageBody, rightAsName);
+ for (Map<String, Object> raw : rows) {
+ JSONObject object = (JSONObject)messageBody.clone();
+ object.fluentPutAll(raw);
+ result.add(object);
+ }
+ }
+
+ return result;
+ }
+
+ private JSONObject addAsName(Map<String, Object> raw, String rightAsName) {
+ JSONObject object = new JSONObject();
+ if (StringUtil.isEmpty(rightAsName)) {
+ return object.fluentPutAll(raw);
+ }
+ for (Map.Entry<String, Object> tmp : raw.entrySet()) {
+ object.put(rightAsName + "." + tmp.getKey(), tmp.getValue());
+ // raw.remove(tmp.getKey());
+ }
+ return object;
+ }
+
+ /**
+ * 生成join消息key值 全局唯一 key值构成结构为 shuffleId：shuffe split id, windowNamespace：窗口命名空间, windowName： 窗口名称, startTime：窗口开始时间, endTime：窗口结束时间, shuffleKey： join key值, routeLabel：消息左右流标记, orginQueueId：原始消息queueid, originOffset：原始消息offset
+ *
+ * @param message
+ * @param routeLabel
+ * @param windowInstance
+ * @return
+ */
+ protected String createStoreKey(IMessage message, String routeLabel, WindowInstance windowInstance) {
+ String shuffleKey = message.getMessageBody().getString(WindowCache.SHUFFLE_KEY);
+ String shuffleId = shuffleChannel.getChannelQueue(shuffleKey).getQueueId();
+ String orginQueueId = message.getMessageBody().getString(WindowCache.ORIGIN_QUEUE_ID);
+ String originOffset = message.getMessageBody().getString(WindowCache.ORIGIN_OFFSET);
+ String windowNamespace = getNameSpace();
+ String windowName = getConfigureName();
+ String startTime = windowInstance.getStartTime();
+ String endTime = windowInstance.getEndTime();
+ String storeKey = MapKeyUtil.createKey(shuffleId, windowNamespace, windowName, startTime, endTime, shuffleKey, routeLabel, orginQueueId, originOffset);
+ return storeKey;
+ }
+
+ protected String createStoreKeyPrefix(IMessage message, String routeLabel, WindowInstance windowInstance) {
+ String shuffleKey = message.getMessageBody().getString(WindowCache.SHUFFLE_KEY);
+ String storeKey = MapKeyUtil.createKey(shuffleKey, routeLabel);
+ return storeKey;
+ }
+
+ /**
+ * 根据左右流标志对原始消息进行封装
+ *
+ * @param message 原始消息
+ * @param instance
+ * @param routeLabel 左右流标志
+ * @return
+ */
+ private JoinState createJoinState(IMessage message, WindowInstance instance, String routeLabel) {
+ MessageHeader header = message.getHeader();
+ String queueId = "_Dipper";
+ String offset = System.nanoTime() + "";
+ if (header.getQueueId() != null) {
+ queueId = header.getQueueId();
+ }
+ if (header.isEmptyOffset()) {
+ header.setOffset(offset);
+ offset = header.getOffset();
+ }
+
+ String messageId = this.getNameSpace() + "_" + this.getConfigureName() + "_" + queueId + "_" + offset;
+
+ String messageKey = generateKey(message.getMessageBody(), routeLabel, this.leftJoinFieldNames, this.rightJoinFieldNames);
+ JSONObject messageBody = (JSONObject)message.getMessageBody().clone();
+ messageBody.remove("WindowInstance");
+ messageBody.remove("AbstractWindow");
+
+ JoinState state = null;
+ if ("left".equalsIgnoreCase(routeLabel)) {
+ state = new JoinLeftState();
+ } else if ("right".equalsIgnoreCase(routeLabel)) {
+ state = new JoinRightState();
+ }
+
+ state.setGmtCreate(new Date());
+ state.setGmtModified(new Date());
+ state.setWindowName(this.getConfigureName());
+ state.setWindowNameSpace(this.getNameSpace());
+ state.setMessageId(messageId);
+ state.setMessageKey(messageKey);
+ state.setMessageTime(new Date());
+ state.setMessageBody(messageBody.toJSONString());
+ state.setMsgKey(createStoreKey(message, routeLabel, instance));
+ String shuffleKey = message.getMessageBody().getString(WindowCache.SHUFFLE_KEY);
+ String shuffleId = shuffleChannel.getChannelQueue(shuffleKey).getQueueId();
+ state.setPartition(shuffleId);
+ state.setWindowInstanceId(instance.getWindowInstanceKey());
+ state.setPartitionNum(incrementAndGetSplitNumber(instance, shuffleId));
+ state.setWindowInstancePartitionId(instance.getWindowInstanceKey());
+
+ return state;
+ }
+
+ /**
+ * 根据join条件生成消息比对key值
+ *
+ * @param messageBody
+ * @param joinLabel
+ * @param leftJoinFieldNames
+ * @param rightJoinFieldNames
+ * @return
+ */
+ public static String generateKey(JSONObject messageBody, String joinLabel, List<String> leftJoinFieldNames, List<String> rightJoinFieldNames) {
+ StringBuffer buffer = new StringBuffer();
+ if ("left".equalsIgnoreCase(joinLabel)) {
+ for (String field : leftJoinFieldNames) {
+ String value = messageBody.getString(field);
+ buffer.append(value).append("_");
+ }
+ } else {
+ for (String field : rightJoinFieldNames) {
+ String[] rightFields = field.split("\\.");
+ if (rightFields.length > 1) {
+ field = rightFields[1];
+ }
+ String value = messageBody.getString(field);
+ buffer.append(value).append("_");
+ }
+ }
+
+ return StringUtil.createMD5Str(buffer.toString());
+ }
+
+ @Override
+ protected String generateShuffleKey(IMessage message) {
+ String routeLabel = message.getHeader().getMsgRouteFromLable();
+ String messageKey = generateKey(message.getMessageBody(), routeLabel, leftJoinFieldNames, rightJoinFieldNames);
+ return messageKey;
+ }
+
+ @Override
+ public Class getWindowBaseValueClass() {
+ return JoinState.class;
+ }
+
+ // @Override
+ // public void finishWindowProcessAndSend2Receiver(List<IMessage> messageList,WindowInstance windowInstance) {
+ // for (IMessage message : messageList) {
+ // List<Map<String, Object>> result = joinOperator.dealJoin(message);
+ // List<Map<String,Object>> rows = matchRows(message.getMessageBody(), result);
+ // String rightAsName = message.getMessageBody().getString("rightAsName");
+ // String joinType = message.getMessageBody().getString("joinType");
+ // List<JSONObject> connectMsgs = joinOperator.connectJoin(message, rows, joinType, rightAsName);
+ // for (int i=0; i < connectMsgs.size(); i++) {
+ // if (i == connectMsgs.size() -1) {
+ // sendMessage(connectMsgs.get(i), true);
+ // } else {
+ // sendMessage(connectMsgs.get(i), false);
+ // }
+ //
+ // }
+ //
+ // }
+ // //todo 完成处理
+ // //todo 发送消息到下一个节点 sendFireMessage();
+ // }
+
+ /**
+ * window触发后的清理工作
+ * @param windowInstances
+ */
+ /**
+ * 删除掉触发过的数据
+ *
+ * @param windowInstances
+ */
+ @Override
+ public void clearFire(List<WindowInstance> windowInstances) {
+
+ if (windowInstances != null && windowInstances.size() > 0) {
+ for (WindowInstance instance : windowInstances) {
+ // System.out.println("clear window is " + instance.getStartTime() + " " + instance.getEndTime() + " " + instance.windowName);
+
+ WindowInstance.cleanWindow(instance);
+ joinOperator.cleanMessage(instance.getWindowNameSpace(), instance.getWindowName(), this.getRetainWindowCount(),
+ this.getSizeInterval(), instance.getStartTime());
+ }
+ }
+ //todo windowinstace
+ //todo left+right
+ }
+
+ protected List<Map<String, Object>> matchRows(JSONObject msg, List<Map<String, Object>> rows) {
+
+ return AbstractDim.matchExpressionByLoop(rows.iterator(), expression, msg, true);
+ }
+
+ private List<Map<String, Object>> converToMapFromList(List<WindowBaseValue> rows) {
+ List<Map<String, Object>> joinMessages = new ArrayList<>();
+ for (WindowBaseValue value : rows) {
+ JSONObject obj = Message.parseObject(((JoinState)value).getMessageBody());
+ joinMessages.add((Map<String, Object>)obj);
+ }
+ return joinMessages;
+ }
+
+ /**
+ * 把触发的数据，发送到下一个节点
+ *
+ * @param message
+ * @param needFlush
+ */
+ protected void sendMessage(JSONObject message, boolean needFlush) {
+ Message nextMessage = new Message(message);
+ if (needFlush) {
+ nextMessage.getHeader().setNeedFlush(true);
+ }
+ AbstractContext context = new Context(nextMessage);
+ this.getFireReceiver().doMessage(nextMessage, context);
+ }
+
+ protected void sendMessage(IMessage msg, List<WindowBaseValue> messages) {
+ if (messages == null || messages.size() == 0) {
+ return;
+ }
+ List<JSONObject> connectMsgs;
+ if (this.expression == null) {
+ List<Map<String, Object>> rows = converToMapFromList(messages);
+ connectMsgs = connectJoin(msg, rows, joinType, rightAsName);
+ } else {
+ List<Map<String, Object>> rows = matchRows(msg.getMessageBody(), converToMapFromList(messages));
+ connectMsgs = connectJoin(msg, rows, joinType, rightAsName);
+ }
+ for (int i = 0; i < connectMsgs.size(); i++) {
+ if (i == connectMsgs.size() - 1) {
+ sendMessage(connectMsgs.get(i), true);
+ } else {
+ sendMessage(connectMsgs.get(i), false);
+ }
+ }
+ }
+
+ @Override
+ public void removeInstanceFromMap(WindowInstance windowInstance) {
+ String begin = DateUtil.getBeforeMinutesTime(windowInstance.getStartTime(), (this.retainWindowCount - 1) * this.sizeInterval);
+ String deletePrefix = MapKeyUtil.createKey(windowInstance.getWindowNameSpace(), windowInstance.getWindowName(), begin);
+ for (Map.Entry<String, WindowInstance> tmp : windowInstanceMap.entrySet()) {
+ if (tmp.getKey().compareToIgnoreCase(deletePrefix) <= 0) {
+ windowInstanceMap.remove(tmp);
+ }
+ }
+ }
+
+ public int getRetainWindowCount() {
+ return retainWindowCount;
+ }
+
+ public void setRetainWindowCount(int retainWindowCount) {
+ this.retainWindowCount = retainWindowCount;
+ }
+
+ public List<String> getLeftJoinFieldNames() {
+ return leftJoinFieldNames;
+ }
+
+ public void setLeftJoinFieldNames(List<String> leftJoinFieldNames) {
+ this.leftJoinFieldNames = leftJoinFieldNames;
+ }
+
+ public List<String> getRightJoinFieldNames() {
+ return rightJoinFieldNames;
+ }
+
+ public void setRightJoinFieldNames(List<String> rightJoinFieldNames) {
+ this.rightJoinFieldNames = rightJoinFieldNames;
+ }
+
+ public String getRightAsName() {
+ return rightAsName;
+ }
+
+ public void setRightAsName(String rightAsName) {
+ this.rightAsName = rightAsName;
+ }
+
+ public String getJoinType() {
+ return joinType;
+ }
+
+ public void setJoinType(String joinType) {
+ this.joinType = joinType;
+ }
+
+ public String getExpression() {
+ return expression;
+ }
+
+ public void setExpression(String expression) {
+ this.expression = expression;
+ }
+}
diff --git a/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/operator/join/Operator.java b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/operator/join/Operator.java
new file mode 100644
index 0000000..e38c029
--- /dev/null
+++ b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/operator/join/Operator.java
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.window.operator.join;
+
+import com.alibaba.fastjson.JSONObject;
+
+import java.security.MessageDigest;
+import java.util.List;
+
+public interface Operator {
+
+ public static String generateKey(JSONObject messageBody, String joinLabel, List<String> leftJoinFieldNames, List<String> rightJoinFieldNames) {
+ StringBuffer buffer = new StringBuffer();
+ if ("left".equalsIgnoreCase(joinLabel)) {
+ for (String field : leftJoinFieldNames) {
+ String value = messageBody.getString(field);
+ buffer.append(value).append("_");
+ }
+ } else {
+ for (String field : rightJoinFieldNames) {
+ String[] rightFields = field.split(".");
+ if (rightFields.length > 1) {
+ field = rightFields[1];
+ }
+ String value = messageBody.getString(field);
+ buffer.append(value).append("_");
+ }
+ }
+
+ buffer.charAt(buffer.length() - 1);
+
+ return MD5(buffer.toString());
+ }
+
+ public static String MD5(String s) {
+ char hexDigits[] = {'0', '1', '2', '3', '4', '5', '6', '7', '8', '9', 'A', 'B', 'C', 'D', 'E', 'F'};
+
+ try {
+ byte[] btInput = s.getBytes();
+ // 获得MD5摘要算法的 MessageDigest 对象
+ MessageDigest mdInst = MessageDigest.getInstance("MD5");
+ // 使用指定的字节更新摘要
+ mdInst.update(btInput);
+ // 获得密文
+ byte[] md = mdInst.digest();
+ // 把密文转换成十六进制的字符串形式
+ int j = md.length;
+ char str[] = new char[j * 2];
+ int k = 0;
+ for (int i = 0; i < j; i++) {
+ byte byte0 = md[i];
+ str[k++] = hexDigits[byte0 >>> 4 & 0xf];
+ str[k++] = hexDigits[byte0 & 0xf];
+ }
+ return new String(str);
+ } catch (Exception e) {
+ e.printStackTrace();
+ throw new RuntimeException(e.getMessage(), e);
+ }
+ }
+
+}
\ No newline at end of file
diff --git a/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/shuffle/AbstractSystemChannel.java b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/shuffle/AbstractSystemChannel.java
new file mode 100644
index 0000000..b1ce2a5
--- /dev/null
+++ b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/shuffle/AbstractSystemChannel.java
@@ -0,0 +1,321 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.window.shuffle;
+
+import org.apache.rocketmq.streams.common.channel.builder.IShuffleChannelBuilder;
+import org.apache.rocketmq.streams.common.channel.impl.memory.MemoryCache;
+import org.apache.rocketmq.streams.common.channel.impl.memory.MemoryChannel;
+import org.apache.rocketmq.streams.common.channel.impl.memory.MemorySink;
+import org.apache.rocketmq.streams.common.channel.impl.memory.MemorySource;
+import org.apache.rocketmq.streams.common.channel.sink.AbstractSupportShuffleSink;
+import org.apache.rocketmq.streams.common.channel.sink.ISink;
+import org.apache.rocketmq.streams.common.channel.source.ISource;
+import org.apache.rocketmq.streams.common.component.ComponentCreator;
+import org.apache.rocketmq.streams.common.channel.builder.IChannelBuilder;
+import org.apache.rocketmq.streams.common.configurable.IConfigurableIdentification;
+import org.apache.rocketmq.streams.common.interfaces.IStreamOperator;
+import org.apache.rocketmq.streams.common.interfaces.ISystemMessageProcessor;
+import org.apache.rocketmq.streams.common.topology.ChainPipeline;
+import org.apache.rocketmq.streams.common.utils.ReflectUtil;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+import org.apache.rocketmq.streams.serviceloader.ServiceLoaderComponent;
+import org.apache.rocketmq.streams.common.context.AbstractContext;
+import org.apache.rocketmq.streams.common.context.IMessage;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+import java.lang.reflect.Field;
+import java.util.*;
+import java.util.concurrent.LinkedBlockingQueue;
+
+public abstract class AbstractSystemChannel implements IConfigurableIdentification, ISystemMessageProcessor, IStreamOperator {
+
+ protected static final Log LOG = LogFactory.getLog(AbstractSystemChannel.class);
+
+ protected static final String CHANNEL_PROPERTY_KEY_PREFIX = "CHANNEL_PROPERTY_KEY_PREFIX";
+ protected static final String CHANNEL_TYPE = "CHANNEL_TYPE";
+
+ protected ISource consumer;
+ protected AbstractSupportShuffleSink producer;
+ protected Map<String, String> channelConfig = new HashMap<>();
+ ;
+ protected boolean hasCreateShuffleChannel = false;
+
+ public void startChannel() {
+ if (consumer == null) {
+ return;
+ }
+ final AbstractSystemChannel channel = this;
+ consumer.start(this);
+ }
+
+ /**
+ * 如果用户未配置shuffle channel，根据pipline数据源动态创建
+ *
+ * @param pipeline
+ */
+ public void autoCreateShuffleChannel(ChainPipeline pipeline) {
+ if (!hasCreateShuffleChannel) {
+ synchronized (this) {
+ if (!hasCreateShuffleChannel) {
+ ISource piplineSource = pipeline.getSource();
+ ServiceLoaderComponent serviceLoaderComponent = ComponentCreator.getComponent(
+ IChannelBuilder.class.getName(), ServiceLoaderComponent.class);
+ IChannelBuilder builder = (IChannelBuilder)serviceLoaderComponent.loadService(piplineSource.getClass().getSimpleName());
+ if (builder == null) {
+ throw new RuntimeException("can not create shuffle channel, not find channel builder " + piplineSource.toJson());
+ }
+ if (!IShuffleChannelBuilder.class.isInstance(builder)) {
+ throw new RuntimeException("can not create shuffle channel, builder not imp IShuffleChannelBuilder " + piplineSource.toJson());
+ }
+ IShuffleChannelBuilder shuffleChannelBuilder = (IShuffleChannelBuilder)builder;
+ ISink sink = shuffleChannelBuilder.createBySource(piplineSource);
+ if (!MemoryChannel.class.isInstance(sink) && !AbstractSupportShuffleSink.class.isInstance(sink)) {
+ throw new RuntimeException("can not create shuffle channel, sink not extends AbstractSupportShuffleSink " + piplineSource.toJson());
+ }
+ ISource source = null;
+ if (MemoryChannel.class.isInstance(sink)) {
+ MemoryCache memoryCache = new MemoryCache();
+ memoryCache.setNameSpace(createShuffleChannelNameSpace(pipeline));
+ memoryCache.setConfigureName(createShuffleChannelName(pipeline));
+
+ sink = new MemorySink();
+ source = new MemorySource();
+ ((MemorySink)sink).setMemoryCache(memoryCache);
+ ((MemorySource)source).setMemoryCache(memoryCache);
+ memoryCache.init();
+ }
+
+ Properties properties = new Properties();
+ putDynamicPropertyValue(new HashSet<>(), properties);
+
+ AbstractSupportShuffleSink shuffleSink = (AbstractSupportShuffleSink)sink;
+ shuffleSink.setSplitNum(getShuffleSplitCount(shuffleSink));
+ shuffleSink.setNameSpace(createShuffleChannelNameSpace(pipeline));
+ shuffleSink.setConfigureName(createShuffleChannelName(pipeline));
+ String topicFiledName = shuffleSink.getShuffleTopicFieldName();
+ String shuffleTopic = null;
+ //内存模式，是无topic的
+ if (StringUtil.isNotEmpty(topicFiledName)) {
+ String topic = ReflectUtil.getDeclaredField(shuffleSink, topicFiledName);
+ shuffleTopic = createShuffleTopic(topic, pipeline);
+ ReflectUtil.setBeanFieldValue(shuffleSink, topicFiledName, shuffleTopic);
+ }
+
+ //修改和window有关的属性，如groupname，tags
+ List<Field> fields = ReflectUtil.getDeclaredFieldsContainsParentClass(sink.getClass());
+ for (Field field : fields) {
+ String fieldName = field.getName();
+ String value = properties.getProperty(fieldName);
+ if (StringUtil.isNotEmpty(value)) {
+ ReflectUtil.setBeanFieldValue(sink, fieldName, value);
+ }
+ }
+
+ shuffleSink.init();//在这里完成shuffle channel的创建
+ if (source == null) {
+ source = shuffleChannelBuilder.copy(piplineSource);
+ }
+
+ //修改和window有关的属性，如groupname，tags
+
+ fields = ReflectUtil.getDeclaredFieldsContainsParentClass(source.getClass());
+ for (Field field : fields) {
+ String fieldName = field.getName();
+ String value = properties.getProperty(fieldName);
+ if (StringUtil.isNotEmpty(value)) {
+ ReflectUtil.setBeanFieldValue(source, fieldName, value);
+ }
+ }
+
+ source.setNameSpace(sink.getNameSpace());
+ source.setConfigureName(sink.getConfigureName());
+ //修改主题
+ if (shuffleTopic != null && topicFiledName != null) {
+ ReflectUtil.setBeanFieldValue(source, topicFiledName, shuffleTopic);
+ }
+
+ source.init();
+
+ this.producer = shuffleSink;
+ this.consumer = source;
+ }
+ }
+ }
+ }
+
+ /**
+ * 根据数据源的名字，给shufflechannel取主题名
+ *
+ * @param topic
+ * @param message
+ * @return
+ */
+ protected abstract String createShuffleTopic(String topic,
+ ChainPipeline message);
+
+ protected abstract int getShuffleSplitCount(AbstractSupportShuffleSink shuffleSink);
+
+ /**
+ * shuffle channel的名字
+ *
+ * @param message
+ * @return
+ */
+ protected abstract String createShuffleChannelName(ChainPipeline message);
+
+ /**
+ * shuffle channel的名字
+ *
+ * @param message
+ * @return
+ */
+ protected abstract String createShuffleChannelNameSpace(ChainPipeline message);
+
+ protected Map<String, String> getChannelConfig() {
+ return channelConfig;
+ }
+
+ protected abstract String getDynamicPropertyValue();
+
+ /**
+ * 创建channel，根据配置文件配置channel的连接信息
+ *
+ * @return
+ */
+ protected ISource createSource(String namespace, String name) {
+
+ IChannelBuilder builder = createBuilder();
+ if (builder == null) {
+ return null;
+ }
+ Properties properties = createChannelProperties(namespace);
+ ISource source = builder.createSource(namespace, name, properties, null);
+ source.init();
+ return source;
+ }
+
+ /**
+ * 创建channel，根据配置文件配置channel的连接信息
+ *
+ * @return
+ */
+ protected AbstractSupportShuffleSink createSink(String namespace, String name) {
+
+ IChannelBuilder builder = createBuilder();
+ if (builder == null) {
+ return null;
+ }
+ Properties properties = createChannelProperties(namespace);
+
+ ISink sink = builder.createSink(namespace, name, properties, null);
+ if (!AbstractSupportShuffleSink.class.isInstance(sink)) {
+ throw new RuntimeException("can not support shuffle " + sink.toJson());
+ }
+ AbstractSupportShuffleSink abstractSupportShuffleSink = (AbstractSupportShuffleSink)sink;
+ abstractSupportShuffleSink.init();
+ return abstractSupportShuffleSink;
+ }
+
+ /**
+ * create channel builder
+ *
+ * @return
+ */
+ protected IChannelBuilder createBuilder() {
+ String type = ComponentCreator.getProperties().getProperty(getChannelConfig().get(CHANNEL_TYPE));
+ if (StringUtil.isEmpty(type)) {
+ return null;
+ }
+ ServiceLoaderComponent serviceLoaderComponent = ComponentCreator.getComponent(
+ IChannelBuilder.class.getName(), ServiceLoaderComponent.class);
+ IChannelBuilder builder = (IChannelBuilder)serviceLoaderComponent.loadService(type);
+ return builder;
+ }
+
+ /**
+ * 根据属性文件配置
+ *
+ * @return
+ */
+ protected Properties createChannelProperties(String namespace) {
+ Properties properties = new Properties();
+ Iterator<Map.Entry<Object, Object>> it = ComponentCreator.getProperties().entrySet().iterator();
+ while (it.hasNext()) {
+ Map.Entry<Object, Object> entry = it.next();
+ String key = (String)entry.getKey();
+ String value = (String)entry.getValue();
+ if (key.startsWith(getChannelConfig().get(CHANNEL_PROPERTY_KEY_PREFIX))) {
+ String channelKey = key.replace(getChannelConfig().get(CHANNEL_PROPERTY_KEY_PREFIX), "");
+ if (channelKey.startsWith(namespace)) {//支持基于namespace 做shuffle window共享
+ channelKey = channelKey.replace(namespace, "");
+ properties.put(channelKey, value);
+ } else {
+ if (!properties.containsKey(channelKey)) {
+ properties.put(channelKey, value);
+ }
+ }
+
+ }
+
+ }
+ Set<String> mutilPropertySet = new HashSet<>();
+ String dynamicProperty = properties.getProperty("dynamic.property");
+ if (dynamicProperty != null) {
+
+ String dynamicPropertyValue = getDynamicPropertyValue();
+ String[] mutilPropertys = dynamicProperty.split(",");
+
+ for (String properyKey : mutilPropertys) {
+ properties.put(properyKey, dynamicPropertyValue);
+ mutilPropertySet.add(properyKey);
+ }
+
+ }
+ putDynamicPropertyValue(mutilPropertySet, properties);
+ return properties;
+ }
+
+ /**
+ * 如果需要额外的动态属性，可以在子类添加
+ *
+ * @param dynamiPropertySet
+ */
+ protected void putDynamicPropertyValue(Set<String> dynamiPropertySet, Properties properties) {
+
+ }
+
+ public ISource getConsumer() {
+ return consumer;
+ }
+
+ public ISink getProducer() {
+ return producer;
+ }
+
+ public void sendMessage(IMessage message) {
+ List<IMessage> msgs = new ArrayList<>();
+ msgs.add(message);
+ producer.batchSave(msgs);
+ producer.flush();
+ }
+
+ //public void flush() {
+ // producer.flush();
+ //}
+
+}
diff --git a/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/shuffle/ShuffleChannel.java b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/shuffle/ShuffleChannel.java
new file mode 100644
index 0000000..184b8b4
--- /dev/null
+++ b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/shuffle/ShuffleChannel.java
@@ -0,0 +1,542 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.window.shuffle;
+
+import com.alibaba.fastjson.JSONArray;
+import com.alibaba.fastjson.JSONObject;
+
+import org.apache.rocketmq.streams.common.channel.sink.AbstractSupportShuffleSink;
+import org.apache.rocketmq.streams.common.channel.source.AbstractSource;
+import org.apache.rocketmq.streams.common.channel.source.systemmsg.NewSplitMessage;
+import org.apache.rocketmq.streams.common.channel.source.systemmsg.RemoveSplitMessage;
+import org.apache.rocketmq.streams.common.channel.split.ISplit;
+import org.apache.rocketmq.streams.common.checkpoint.CheckPointMessage;
+import org.apache.rocketmq.streams.common.checkpoint.CheckPointState;
+import org.apache.rocketmq.streams.common.configure.ConfigureFileKey;
+import org.apache.rocketmq.streams.common.interfaces.ISystemMessage;
+import org.apache.rocketmq.streams.common.topology.ChainPipeline;
+import org.apache.rocketmq.streams.common.topology.model.Pipeline;
+import org.apache.rocketmq.streams.common.utils.CollectionUtil;
+import org.apache.rocketmq.streams.common.utils.DateUtil;
+import org.apache.rocketmq.streams.common.utils.TraceUtil;
+import org.apache.rocketmq.streams.db.driver.orm.ORMUtil;
+import org.apache.rocketmq.streams.window.operator.AbstractShuffleWindow;
+import org.apache.rocketmq.streams.window.operator.AbstractWindow;
+import org.apache.rocketmq.streams.common.context.AbstractContext;
+import org.apache.rocketmq.streams.common.context.IMessage;
+import org.apache.rocketmq.streams.common.context.Message;
+import org.apache.rocketmq.streams.common.utils.MapKeyUtil;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+import org.apache.rocketmq.streams.window.model.WindowInstance;
+import org.apache.rocketmq.streams.window.model.WindowCache;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.rocketmq.streams.window.operator.impl.WindowOperator.WindowRowOperator;
+import org.apache.rocketmq.streams.window.source.WindowRireSource;
+import org.apache.rocketmq.streams.window.storage.ShufflePartitionManager;
+
+import java.util.*;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * 负责处理分片
+ */
+public class ShuffleChannel extends AbstractSystemChannel {
+
+ protected static final Log LOG = LogFactory.getLog(ShuffleChannel.class);
+
+ protected static final String SHUFFLE_QUEUE_ID = "SHUFFLE_QUEUE_ID";
+ private static final String SHUFFLE_KEY = "SHUFFLE_KEY";
+ private static final String PROCESS_NAME = "PROCESS_NAME";
+
+ protected static final String SHUFFLE_MESSAGES = "SHUFFLE_MESSAGES";
+ protected String MSG_OWNER = "MSG_OWNER";//消息所属的window
+
+ protected String WINDOW_INSTANCE_START_TIME = "_start_time";
+ protected String WINDOW_INSTANCE_END_TIME = "_end_time";
+
+ private static final String SHUFFLE_TRACE_ID = "SHUFFLE_TRACE_ID";
+
+ protected ShuffleOutputDataSource shuffleSink;
+
+ protected Map<String, ISplit> queueMap = new ConcurrentHashMap<>();
+ protected List<ISplit> queueList;//所有的分片
+
+ // protected NotifyChannel notfiyChannel;//负责做分片的通知管理
+ protected AbstractShuffleWindow window;
+ private Set<String> currentQueueIds;//当前管理的分片
+
+ protected transient WindowRireSource windowRireSource;
+
+ public ShuffleChannel(AbstractShuffleWindow window) {
+ this.window = window;
+ channelConfig = new HashMap<>();
+ channelConfig.put(CHANNEL_PROPERTY_KEY_PREFIX, ConfigureFileKey.WINDOW_SHUFFLE_CHANNEL_PROPERTY_PREFIX);
+ channelConfig.put(CHANNEL_TYPE, ConfigureFileKey.WINDOW_SHUFFLE_CHANNEL_TYPE);
+ this.consumer = createSource(window.getNameSpace(),window.getConfigureName());
+
+ this.producer = createSink(window.getNameSpace(),window.getConfigureName());
+ if(this.consumer==null||this.producer==null){
+ autoCreateShuffleChannel(window.getFireReceiver().getPipeline());
+ }
+ if(this.consumer instanceof AbstractSource){
+ ((AbstractSource)this.consumer).setJsonData(true);
+ }
+ this.shuffleSink = createWindowTaskOutputDataSource();
+ this.shuffleSink.openAutoFlush();
+ //this.notfiyChannel = NotifyChannel.getInstance();
+ //if(this.notfiyChannel.producer==null||this.notfiyChannel.consumer==null){
+ // this.notfiyChannel.autoCreateShuffleChannel(window.getFireReceiver().getPipeline());
+ //}
+ //this.notfiyChannel.startChannel();//启动通知管理，里面做了重入，最终只启动一个
+ if (producer!=null&&(queueList == null || queueList.size() == 0) ){
+ queueList = producer.getSplitList();
+ Map<String, ISplit> tmp = new ConcurrentHashMap<>();
+ for (ISplit queue : queueList) {
+ tmp.put(queue.getQueueId(), queue);
+ }
+
+ this.queueMap = tmp;
+ }
+ }
+
+ @Override
+ protected String getDynamicPropertyValue() {
+ String dynamicPropertyValue= MapKeyUtil.createKey(window.getNameSpace(),window.getConfigureName());
+ dynamicPropertyValue = dynamicPropertyValue.replaceAll("\\.", "_").replaceAll(";","_");
+ return dynamicPropertyValue;
+ }
+
+ @Override
+ protected int getShuffleSplitCount(AbstractSupportShuffleSink shuffleSink) {
+ int splitNum=shuffleSink.getSplitNum();
+ return splitNum>0?splitNum:32;
+ }
+
+ /**
+ * 接收到分片信息，如果是系统消息，做缓存刷新，否则把消息放入缓存，同时计算存储的有效性
+ *
+ * @param oriMessage
+ * @param context
+ * @return
+ */
+
+
+ @Override
+ public Object doMessage(IMessage oriMessage, AbstractContext context) {
+ if (oriMessage.getHeader().isSystemMessage()) {
+ ISystemMessage systemMessage=oriMessage.getSystemMessage();
+ if(systemMessage instanceof CheckPointMessage){
+ this.checkpoint(oriMessage, context,(CheckPointMessage)systemMessage);
+ }else if(systemMessage instanceof NewSplitMessage){
+ this.addNewSplit(oriMessage,context,(NewSplitMessage)systemMessage);
+ }else if(systemMessage instanceof RemoveSplitMessage){
+ this.removeSplit(oriMessage,context,(RemoveSplitMessage)systemMessage);
+ }else {
+ throw new RuntimeException("can not support this system message "+systemMessage.getClass().getName());
+ }
+ afterFlushCallback(oriMessage,context);
+ return null;
+
+ }
+ /**
+ * 过滤不是这个window的消息，一个shuffle通道，可能多个window共享，这里过滤掉非本window的消息
+ */
+ boolean isFilter=filterNotOwnerMessage(oriMessage);
+ if(isFilter){
+ return null;
+ }
+ String queueId=oriMessage.getHeader().getQueueId();
+ //ISplit channelQueue=queueMap.get(queueId);
+ //boolean containQueueId=notfiyChannel.contains(queueId);
+ //notfiyChannel.dealMessageQueue(channelQueue);
+
+ JSONArray messages = oriMessage.getMessageBody().getJSONArray(SHUFFLE_MESSAGES);
+ if(messages==null){
+ return null;
+ }
+
+ String traceId = oriMessage.getMessageBody().getString(SHUFFLE_TRACE_ID);
+ if (!StringUtil.isEmpty(traceId)) {
+ TraceUtil.debug(traceId, "shuffle message in", "received message size:" + messages.size());
+ }
+
+ for (Object obj: messages) {
+ IMessage message = new Message((JSONObject) obj);
+ message.getHeader().setQueueId(queueId);
+
+ List<WindowInstance> windowInstances=window.queryOrCreateWindowInstance(message,queueId);
+ if(windowInstances==null||windowInstances.size()==0){
+ continue;
+ }
+ for(WindowInstance windowInstance:windowInstances){
+ String windowInstanceId = windowInstance.createWindowInstanceId();
+ if(!window.getWindowInstanceMap().containsKey(windowInstanceId)){
+ window.getWindowInstanceMap().putIfAbsent(windowInstanceId,windowInstance);
+ synchronized (this){
+ if(window.getFireMode()==2){
+ //这个模式窗口触发不会清理数据，需要额外的创建一个实例做最后的存储清理
+ Date endTime=DateUtil.parseTime(windowInstance.getEndTime());
+ Date lastFireTime=DateUtil.addDate(TimeUnit.SECONDS,endTime,window.getWaterMarkMinute()*window.getTimeUnitAdjust());
+ WindowInstance lastClearWindowInstance=window.createWindowInstance(windowInstance.getStartTime(),windowInstance.getEndTime(),DateUtil.format(lastFireTime),queueId);
+
+ addNeedFlushWindowInstance(lastClearWindowInstance);
+ }
+ }
+
+ }
+ if(windowInstance.isNewWindowInstance()){
+ addNeedFlushWindowInstance(windowInstance);
+ windowInstance.setNewWindowInstance(false);
+ ShufflePartitionManager.getInstance().setWindowInstanceFinished(windowInstance.createWindowInstanceId());
+ }
+ }
+
+ message.getMessageBody().put(WindowInstance.class.getSimpleName(), windowInstances);
+ message.getMessageBody().put(AbstractWindow.class.getSimpleName(), window);
+ beforeBatchAdd(oriMessage,message);
+ shuffleSink.batchAdd(message);
+ for(WindowInstance windowInstance:windowInstances){
+ window.getWindowFireSource().updateWindowInstanceLastUpdateTime(windowInstance,window.updateMaxEventTime(message));
+ }
+ }
+
+ return null;
+ }
+
+ @Override
+ protected Map<String, String> getChannelConfig() {
+ return channelConfig;
+ }
+
+ @Override
+ protected void putDynamicPropertyValue(Set<String> dynamiPropertySet,Properties properties){
+ String groupName="groupName";
+ if(!dynamiPropertySet.contains(groupName)){
+ properties.put(groupName,getDynamicPropertyValue());
+ }
+ if(!dynamiPropertySet.contains("tags")){
+ properties.put("tags",getDynamicPropertyValue());
+ }
+ }
+
+ /**
+ * 1个pipeline一个 shuffle topic
+ *
+ * @param topic
+ * @param pipeline
+ * @return
+ */
+ @Override
+ protected String createShuffleTopic(String topic, ChainPipeline pipeline) {
+ return "shuffle_" + topic + "_" + pipeline.getSource().getNameSpace().replaceAll("\\.", "_") + "_" + pipeline
+ .getConfigureName().replaceAll("\\.", "_").replaceAll(";", "_");
+ }
+
+ /**
+ * 一个window 一个channel
+ *
+ * @param pipeline
+ * @return
+ */
+ @Override
+ protected String createShuffleChannelName(ChainPipeline pipeline) {
+ return getDynamicPropertyValue();
+ }
+
+ /**
+ * 和pipeline namespace 相同
+ *
+ * @param pipeline
+ * @return
+ */
+ @Override
+ protected String createShuffleChannelNameSpace(ChainPipeline pipeline) {
+ return pipeline.getSource().getNameSpace();
+ }
+
+ /**
+ * 对于接收到新的window task消息后的处理器
+ *
+ * @return
+ */
+ protected ShuffleOutputDataSource createWindowTaskOutputDataSource() {
+
+ ShuffleOutputDataSource outputDataSource =new ShuffleOutputDataSource();
+ outputDataSource.init();
+ outputDataSource.setConfigureName("shuffleChannel");
+ return outputDataSource;
+ }
+
+ @Override
+ public void addNewSplit(IMessage message, AbstractContext context, NewSplitMessage newSplitMessage) {
+ this.currentQueueIds=newSplitMessage.getCurrentSplitIds();
+ List<WindowInstance> allWindowInstances=WindowInstance.queryAllWindowInstance(DateUtil.getCurrentTimeString(),window,newSplitMessage.getSplitIds());
+ if(CollectionUtil.isNotEmpty(allWindowInstances)){
+ Set<String> partitionNumKeys=new HashSet<>();
+ for(WindowInstance windowInstance:allWindowInstances){
+ windowInstance.setNewWindowInstance(false);
+ window.getWindowInstanceMap().putIfAbsent(windowInstance.createWindowInstanceId(),windowInstance);
+ window.getWindowFireSource().registFireWindowInstanceIfNotExist(windowInstance,window);
+ for(String queueId:newSplitMessage.getSplitIds()){
+ String key=window.getWindowMaxValueManager().createSplitNumberKey(windowInstance,queueId);
+ partitionNumKeys.add(key);
+ window.getStorage().loadSplitData2Local(queueId,windowInstance.createWindowInstanceId(),window.getWindowBaseValueClass(),new WindowRowOperator(windowInstance,queueId,window));
+ }
+ }
+ window.getWindowMaxValueManager().loadMaxSplitNum(partitionNumKeys);
+
+ }else {
+ for(String queueId:newSplitMessage.getSplitIds()){
+ ShufflePartitionManager.getInstance().setSplitFinished(queueId);
+ }
+ }
+
+ }
+
+ @Override
+ public void removeSplit(IMessage message, AbstractContext context, RemoveSplitMessage removeSplitMessage) {
+ this.currentQueueIds=removeSplitMessage.getCurrentSplitIds();
+ Set<String> queueIds=removeSplitMessage.getSplitIds();
+ if(queueIds!=null){
+ for(String queueId:queueIds){
+ ShufflePartitionManager.getInstance().setSplitInValidate(queueId);
+ window.clearCache(queueId);
+
+ }
+ window.getWindowMaxValueManager().removeKeyPrefixFromLocalCache(queueIds);
+ //window.getWindowFireSource().removeSplit(queueIds);
+ }
+ }
+
+ @Override
+ public void checkpoint(IMessage message, AbstractContext context, CheckPointMessage checkPointMessage) {
+ if(message.getHeader().isNeedFlush()){
+ this.flush(message.getHeader().getCheckpointQueueIds());
+ }
+ CheckPointState checkPointState= new CheckPointState();
+ checkPointState.setQueueIdAndOffset(this.shuffleSink.getFinishedQueueIdAndOffsets(checkPointMessage));
+ checkPointMessage.reply(checkPointState);
+ }
+
+ @Override
+ public String getConfigureName() {
+ return window.getConfigureName()+"_shuffle";
+ }
+
+ @Override
+ public String getNameSpace() {
+ return window.getNameSpace();
+ }
+
+ @Override
+ public String getType() {
+ return Pipeline.TYPE;
+ }
+
+ protected class ShuffleOutputDataSource extends WindowCache {
+ protected List<WindowInstance> notSaveWindowInstances=new ArrayList<>();//未保存的windowinstance
+
+ public ShuffleOutputDataSource() {
+ }
+
+ @Override
+ protected boolean batchInsert(List<IMessage> messageList) {
+ Map<Pair<String, String>, List<IMessage>> instance2Messages = new HashMap<>();
+ Map<String, WindowInstance> windowInstanceMap = new HashMap<>();
+ groupByWindowInstanceAndQueueId(messageList, instance2Messages, windowInstanceMap);
+ Iterator<Map.Entry<Pair<String, String>, List<IMessage>>> it = instance2Messages.entrySet().iterator();
+ while (it.hasNext()) {
+ Map.Entry<Pair<String, String>, List<IMessage>> entry = it.next();
+ Pair<String, String> queueIdAndInstanceKey = entry.getKey();
+ List<IMessage> messages = entry.getValue();
+ WindowInstance windowInstance = windowInstanceMap.get(queueIdAndInstanceKey.getRight());
+ window.shuffleCalculate(messages, windowInstance, queueIdAndInstanceKey.getLeft());
+ }
+ return true;
+ }
+
+ @Override
+ public boolean flushMessage(List<IMessage> messages) {
+ saveOtherState();
+ return super.flushMessage(messages);
+ }
+
+ protected void saveOtherState(){
+ if(notSaveWindowInstances.size()>0){
+ if(notSaveWindowInstances.size()>0){
+ List<WindowInstance> copy=null;
+ synchronized (this){
+ copy=this.notSaveWindowInstances;
+ this.notSaveWindowInstances=new ArrayList<>();
+ }
+ Set<String> existWindowInstaceIds=new HashSet<>();
+ List<WindowInstance> windowInstances=new ArrayList<>();
+ for(WindowInstance windowInstance:copy){
+ String windowInstanceId=windowInstance.createWindowInstanceId();
+ if(existWindowInstaceIds.contains(windowInstanceId)){
+ continue;
+ }
+ windowInstances.add(windowInstance);
+ existWindowInstaceIds.add(windowInstanceId);
+ }
+ ORMUtil.batchReplaceInto(windowInstances);
+ }
+
+ }
+ window.getWindowMaxValueManager().flush();
+ }
+
+ @Override
+ protected String generateShuffleKey(IMessage message) {
+ return null;
+ }
+ }
+
+ /**
+ * 根据message，把message分组到不同的group，分别处理
+ *
+ * @param messageList
+ * @param instance2Messages
+ * @param windowInstanceMap
+ */
+ protected void groupByWindowInstanceAndQueueId(List<IMessage> messageList, Map<Pair<String, String>, List<IMessage>> instance2Messages,
+ Map<String, WindowInstance> windowInstanceMap) {
+ for (IMessage message : messageList) {
+
+ List<WindowInstance> windowInstances = (List<WindowInstance>)message.getMessageBody().get(WindowInstance.class.getSimpleName());
+ String queueId = message.getHeader().getQueueId();
+ for(WindowInstance windowInstance:windowInstances){
+ String windowInstanceId = windowInstance.createWindowInstanceId();
+ Pair<String, String> queueIdAndInstanceKey = Pair.of(queueId, windowInstanceId);
+ List<IMessage> messages = instance2Messages.get(queueIdAndInstanceKey);
+ if (messages == null) {
+ messages = new ArrayList<>();
+ instance2Messages.put(queueIdAndInstanceKey, messages);
+ }
+ messages.add(message);
+ windowInstanceMap.put(windowInstanceId, windowInstance);
+ }
+
+ String oriQueueId = message.getMessageBody().getString(WindowCache.ORIGIN_QUEUE_ID);
+ String oriOffset = message.getMessageBody().getString(WindowCache.ORIGIN_OFFSET);
+ message.getHeader().setQueueId(oriQueueId);
+ message.getHeader().setOffset(oriOffset);
+
+ }
+ }
+
+ public ISplit getSplit(Integer index){
+ return queueList.get(index);
+ }
+
+ public JSONObject createMsg(JSONArray messages,ISplit split) {
+
+ JSONObject msg = new JSONObject();
+
+ msg.put(SHUFFLE_QUEUE_ID, split.getQueueId());//分片id
+ msg.put(SHUFFLE_MESSAGES, messages);//合并的消息
+ msg.put(MSG_OWNER,getDynamicPropertyValue());//消息owner
+
+ StringBuilder traceIds = new StringBuilder();
+ for (int i = 0; i < messages.size(); i++) {
+ JSONObject object = messages.getJSONObject(i);
+ if (object.containsKey(WindowCache.ORIGIN_MESSAGE_TRACE_ID)) {
+ traceIds.append(object.getString(WindowCache.ORIGIN_MESSAGE_TRACE_ID)).append(";");
+ }
+ }
+ msg.put(SHUFFLE_TRACE_ID, traceIds);
+ TraceUtil.debug(traceIds.toString(), "origin message out", split.getQueueId());
+
+ return msg;
+ }
+
+ public ISplit getChannelQueue(String key){
+ int index=hash(key);
+ ISplit targetQueue = queueList.get(index);
+ return targetQueue;
+ }
+
+ public int hash(Object key) {
+ int mValue=queueList.size();
+ int h = 0;
+ if (key != null) {
+ h = key.hashCode() ^ (h >>> 16);
+ if (h < 0) {
+ h = -h;
+ }
+ }
+ return h % mValue;
+ }
+
+ public void flush(Set<String> checkpointQueueIds) {
+ shuffleSink.flush(checkpointQueueIds);
+ }
+
+ /**
+ * 每次checkpoint的回调函数，默认是空实现，子类可以扩展实现
+ *
+ * @param oriMessage
+ * @param context
+ */
+ protected void afterFlushCallback(IMessage oriMessage, AbstractContext context) {
+ }
+
+ /**
+ * shuffle 获取数据，插入缓存前的回调函数，默认空实现，可以子类覆盖扩展
+ *
+ * @param oriMessage
+ * @param message
+ */
+ protected void beforeBatchAdd(IMessage oriMessage, IMessage message) {
+ }
+
+ /**
+ * 过滤掉不是这个window的消息
+ *
+ * @param oriMessage
+ * @return
+ */
+ protected boolean filterNotOwnerMessage(IMessage oriMessage) {
+ String owner=oriMessage.getMessageBody().getString(MSG_OWNER);
+ if(owner!=null&&owner.equals(getDynamicPropertyValue())){
+ return false;
+ }
+ return true;
+ }
+
+ public Set<String> getCurrentQueueIds() {
+ return currentQueueIds;
+ }
+
+ public List<ISplit> getQueueList() {
+ return queueList;
+ }
+
+ public synchronized void addNeedFlushWindowInstance(WindowInstance windowInstance){
+ if(!window.isLocalStorageOnly()){
+ this.shuffleSink.notSaveWindowInstances.add(windowInstance);
+ }
+ }
+
+ public synchronized void clearCache(WindowInstance windowInstance){
+ this.shuffleSink.notSaveWindowInstances.remove(windowInstance);
+ }
+
+}
\ No newline at end of file
diff --git a/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/source/WindowRireSource.java b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/source/WindowRireSource.java
new file mode 100644
index 0000000..c6495db
--- /dev/null
+++ b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/source/WindowRireSource.java
@@ -0,0 +1,336 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.window.source;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Date;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.rocketmq.streams.common.channel.sinkcache.IMessageCache;
+import org.apache.rocketmq.streams.common.channel.sinkcache.IMessageFlushCallBack;
+import org.apache.rocketmq.streams.common.channel.sinkcache.impl.AbstractMutilSplitMessageCache;
+import org.apache.rocketmq.streams.common.channel.source.AbstractSupportOffsetResetSource;
+import org.apache.rocketmq.streams.common.context.AbstractContext;
+import org.apache.rocketmq.streams.common.context.IMessage;
+import org.apache.rocketmq.streams.common.interfaces.IStreamOperator;
+import org.apache.rocketmq.streams.window.model.WindowInstance;
+import org.apache.rocketmq.streams.window.operator.AbstractWindow;
+
+public class WindowRireSource extends AbstractSupportOffsetResetSource implements IStreamOperator {
+ protected static final Log LOG = LogFactory.getLog(WindowRireSource.class);
+ private AbstractWindow window;
+ protected transient ConcurrentHashMap<String,Long> maxEventTimes=new ConcurrentHashMap();//max event time procced by window
+ protected transient ConcurrentHashMap<String,Long> eventTimeLastUpdateTimes=new ConcurrentHashMap<>();
+ protected transient ScheduledExecutorService fireCheckScheduler;//检查是否触发
+ protected transient ScheduledExecutorService checkpointScheduler;
+ protected transient ConcurrentHashMap<String,WindowInstance> windowInstances=new ConcurrentHashMap();
+
+ protected transient IMessageCache<WindowInstance> fireInstanceCache=new WindowInstanceCache();
+ //正在触发中的windowintance
+ protected transient ConcurrentHashMap<String,WindowInstance> firingWindowInstances=new ConcurrentHashMap<>();
+
+
+ //<windowinstanceId,<queueId，offset>>
+ protected transient ConcurrentHashMap<String,Map<String,String>> windowInstanceQueueOffsets=new ConcurrentHashMap<>();
+
+ public WindowRireSource(AbstractWindow window){
+ this.window=window;
+ }
+
+ @Override
+ protected boolean initConfigurable() {
+ fireCheckScheduler=new ScheduledThreadPoolExecutor(2);
+ checkpointScheduler=new ScheduledThreadPoolExecutor(3);
+ setReceiver(window.getFireReceiver());
+ fireInstanceCache.openAutoFlush();
+ return super.initConfigurable();
+ }
+ @Override
+ protected boolean startSource() {
+ //检查window instance，如果已经到了触发时间，且符合触发条件，直接触发，如果到了触发时间，还未符合触发条件。则放入触发列表。下次调度时间是下一个最近触发的时间
+
+ fireCheckScheduler.scheduleWithFixedDelay(new Runnable() {
+ // long startTime=System.currentTimeMillis();
+ @Override
+ public void run() {
+ try {
+ //System.out.println("fire schdule time is "+(System.currentTimeMillis()-startTime)+" windowinstance count is "+windowInstances.size());
+ //startTime=System.currentTimeMillis();
+ if(windowInstances.size()==0){
+ //if(eventTimeLastUpdateTime!=null){
+ // int gap=(int)(System.currentTimeMillis()-eventTimeLastUpdateTime);
+ // if(window.getMsgMaxGapSecond()!=null&&gap>window.getMsgMaxGapSecond()*1000){
+ // for(String key:fireCounts.keySet()){
+ // Integer count=fireCounts.get(key);
+ // if(count==0){
+ // System.out.println("===================== "+key+":"+count);
+ // }
+ // }
+ // }
+ //}
+
+ }
+ List<WindowInstance> windowInstanceList=new ArrayList<>();
+ windowInstanceList.addAll(windowInstances.values());
+ long fireStartTime=System.currentTimeMillis();
+ Collections.sort(windowInstanceList, new Comparator<WindowInstance>() {
+ @Override
+ public int compare(WindowInstance o1, WindowInstance o2) {
+ int value= o1.getFireTime().compareTo(o2.getFireTime());
+ if(value!=0){
+ return value;
+ }
+ return o2.getStartTime().compareTo(o1.getStartTime());
+ }
+ });
+ for(int i=0;i<windowInstanceList.size();i++){
+ WindowInstance windowInstance = windowInstanceList.get(i);
+
+ boolean success= executeFireTask(windowInstance,false);
+ if(!success){
+ continue;
+ }
+ windowInstances.remove(windowInstance.createWindowInstanceId());
+
+ }
+ } catch (Exception e) {
+ e.printStackTrace();
+ }
+ }
+
+ },0,1, TimeUnit.SECONDS);
+
+ //定时发送checkpoint，提交和保存数据。在pull模式会有用
+ //fireCheckScheduler.scheduleWithFixedDelay(new Runnable() {
+ //
+ // @Override
+ // public void run() {
+ // if(checkPointManager.getCurrentSplits()==null||checkPointManager.getCurrentSplits().size()==0){
+ // return;
+ // }
+ // for(WindowInstance windowInstance:firingWindowInstances.values()){
+ // Set<String> splits=checkPointManager.getCurrentSplits();
+ // Set<String> windowInstanceSplits=new HashSet<>();
+ // for(String splitId:splits){
+ // //String windowInstanceSpiltId=windowInstance.createWindowInstancePartitionId();
+ // windowInstanceSplits.add(splitId);
+ // }
+ // sendCheckpoint(windowInstanceSplits);
+ // }
+ //
+ // }
+ //},0,getCheckpointTime(), TimeUnit.MILLISECONDS);
+
+ return false;
+ }
+
+
+
+ /**
+ * 如果没有window instance，则注册，否则放弃
+ * @param windowInstance
+ */
+ public void registFireWindowInstanceIfNotExist(WindowInstance windowInstance, AbstractWindow window){
+ String windowInstanceId=windowInstance.createWindowInstanceId();
+ WindowInstance old= windowInstances.putIfAbsent(windowInstanceId,windowInstance);
+ if(old==null){
+ window.getWindowInstanceMap().put(windowInstanceId,windowInstance);
+ }
+ LOG.debug("register window instance into manager, instance key: " + windowInstance.createWindowInstanceId());
+ }
+ /**
+ * 注册一个window instance
+ * @param windowInstance
+ */
+ public void updateWindowInstanceLastUpdateTime(WindowInstance windowInstance,Long windowMaxEventTime){
+ String windowInstanceId=windowInstance.createWindowInstanceId();
+ if(windowMaxEventTime!=null){
+ this.maxEventTimes.put(windowInstanceId,windowMaxEventTime);
+ this.eventTimeLastUpdateTimes.put(windowInstanceId,System.currentTimeMillis());
+ }
+
+
+ windowInstances.putIfAbsent(windowInstanceId,windowInstance);
+
+
+
+ }
+ /**
+ * 触发窗口
+ * @param windowInstance
+ */
+ public boolean executeFireTask(WindowInstance windowInstance,boolean startNow) {
+ String windowInstanceId=windowInstance.createWindowInstanceId();
+ if (canFire(windowInstance)) {
+ //maybe firimg
+ if (firingWindowInstances.containsKey(windowInstanceId)) {
+ //System.out.println("has firing");
+
+ return true;
+ }
+ //maybe fired
+ //if (!window.getWindowInstanceMap().containsKey(windowInstanceId)) {
+ // return true;
+ //}
+ //start firing
+ firingWindowInstances.put(windowInstanceId, windowInstance);
+ if(startNow){
+ fireWindowInstance(windowInstance);
+ }else {
+ fireInstanceCache.addCache(windowInstance);
+ }
+ return true;
+ }
+ return false;
+ }
+ /**
+ * 触发窗口
+ * @param windowInstance
+ */
+ protected void fireWindowInstance(WindowInstance windowInstance) {
+ try {
+ if (windowInstance == null) {
+ LOG.error("window instance is null!");
+ return;
+ }
+ String windowInstanceId = windowInstance.createWindowInstanceId();
+ if (window == null) {
+ LOG.error(windowInstanceId + "'s window object have been removed!");
+ return;
+ }
+
+
+ if(windowInstance.getLastMaxUpdateTime()==null){
+ windowInstance.setLastMaxUpdateTime(this.maxEventTimes.get(windowInstanceId));
+ }
+ int fireCount=window.fireWindowInstance(windowInstance,windowInstanceQueueOffsets.get(windowInstanceId));
+ LOG.debug("fire instance("+windowInstance.createWindowInstanceId()+" fire count is "+fireCount);
+ firingWindowInstances.remove(windowInstanceId);
+ this.eventTimeLastUpdateTimes.remove(windowInstanceId);
+ this.maxEventTimes.remove(windowInstanceId);
+ } catch (Exception e) {
+ e.printStackTrace();
+ }
+ }
+
+
+ /**
+ * 是否符合触发条件
+ * @param windowInstance
+ * @return
+ */
+ protected boolean canFire(WindowInstance windowInstance) {
+ String windowInstanceId=windowInstance.createWindowInstanceId();
+ if(window == null){
+ LOG.warn(windowInstanceId + " can't find window!");
+ return false;
+ }
+ Date realFireTime=window.getRealFireTime(windowInstance);
+ /**
+ * 未到触发时间
+ */
+ Long maxEventTime=this.maxEventTimes.get(windowInstanceId);
+ if(maxEventTime==null){
+ return false;
+ }
+ if(maxEventTime-realFireTime.getTime()>=0){
+ return true;
+ }
+ if(maxEventTime-realFireTime.getTime()<0){
+ Long eventTimeLastUpdateTime=this.eventTimeLastUpdateTimes.get(windowInstanceId);
+ if(eventTimeLastUpdateTime==null){
+ return false;
+ }
+ int gap=(int)(System.currentTimeMillis()-eventTimeLastUpdateTime);
+ if(window.getMsgMaxGapSecond()!=null&&gap>window.getMsgMaxGapSecond()*1000){
+ LOG.warn("the fire reason is exceed the gap "+gap+" window instance id is "+windowInstanceId);
+ return true;
+ }
+ return false;
+ }
+
+ return true;
+ }
+
+ @Override
+ public Object doMessage(IMessage message, AbstractContext context) {
+ return null;
+ }
+
+
+ protected class WindowInstanceCache extends AbstractMutilSplitMessageCache<WindowInstance>{
+
+ public WindowInstanceCache() {
+ super(new IMessageFlushCallBack<WindowInstance>() {
+ @Override
+ public boolean flushMessage(List<WindowInstance> windowInstances) {
+
+ Collections.sort(windowInstances, new Comparator<WindowInstance>() {
+ @Override
+ public int compare(WindowInstance o1, WindowInstance o2) {
+ int value= o1.getFireTime().compareTo(o2.getFireTime());
+ if(value!=0){
+ return value;
+ }
+ return o2.getStartTime().compareTo(o1.getStartTime());
+ }
+ });
+
+ for(WindowInstance windowInstance:windowInstances){
+ fireWindowInstance(windowInstance);
+ }
+ return true;
+ }
+ });
+ }
+
+
+ @Override
+ protected String createSplitId(WindowInstance windowInstance) {
+ return windowInstance.getSplitId();
+ }
+ }
+
+ @Override
+ public boolean supportNewSplitFind() {
+ return true;
+ }
+
+ @Override
+ public boolean supportRemoveSplitFind() {
+ return false;
+ }
+
+ @Override
+ public boolean supportOffsetRest() {
+ return false;
+ }
+
+ @Override
+ protected boolean isNotDataSplit(String queueId) {
+ return false;
+ }
+}
diff --git a/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/state/WindowBaseValue.java b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/state/WindowBaseValue.java
new file mode 100644
index 0000000..d558c20
--- /dev/null
+++ b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/state/WindowBaseValue.java
@@ -0,0 +1,188 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.window.state;
+
+import java.io.Serializable;
+import java.util.Date;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.apache.rocketmq.streams.common.configurable.BasedConfigurable;
+import org.apache.rocketmq.streams.common.utils.DateUtil;
+import org.apache.rocketmq.streams.common.utils.MapKeyUtil;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.rocketmq.streams.common.utils.ReflectUtil;
+
+public class WindowBaseValue extends BasedConfigurable implements Serializable {
+
+ private static final Log LOG = LogFactory.getLog(WindowBaseValue.class);
+
+ private static final long serialVersionUID = -4985883726971532986L;
+
+ /**
+ * 唯一标识一个groupBy的窗口计算数据 创建唯一主键 内存及磁盘里使用(partition,windowNamespace,windowName,startTime,endOrFireTime,groupByValue)的拼接 数据库里用上面拼接字符串的MD5
+ */
+ protected String msgKey;
+
+ /**
+ * 唯一标识一个窗口 内存及磁盘使用(patitionId,windowNamespace,windowName,startTime,endOrFireTime) 数据库里使用上面字符串的MD5
+ */
+ protected String windowInstanceId;
+
+ /**
+ * 分片信息（metaQ里是queue）
+ */
+ protected String partition;
+
+ /**
+ * 同一分片同一窗口的自增数据（增加逻辑在业务里，为什么不使用id？）
+ */
+ protected long partitionNum;
+
+ /**
+ * 标识一个分片同一个窗口 内存及磁盘使用(partition,windowNamespace,windowName,windowinstanceName,startTime,endTime,partition) 数据库里使用上面字符串的MD5值
+ */
+ protected String windowInstancePartitionId;
+
+ /**
+ * 窗口实例的开始时间
+ */
+ protected String startTime;
+
+ /**
+ * 窗口实例的结束时间
+ */
+ protected String endTime;
+
+ /**
+ * 窗口实例的触发时间
+ */
+ protected String fireTime;
+
+ protected AtomicLong updateVersion = new AtomicLong(0);
+
+ public WindowBaseValue() {
+ setGmtCreate(DateUtil.getCurrentTime());
+ setGmtModified(DateUtil.getCurrentTime());
+ }
+
+ public long incrementUpdateVersion() {
+ return updateVersion.incrementAndGet();
+ }
+
+ @Override
+ public Date getGmtCreate() {
+ return gmtCreate;
+ }
+
+ @Override
+ public void setGmtCreate(Date gmtCreate) {
+ this.gmtCreate = gmtCreate;
+ }
+
+ @Override
+ public Date getGmtModified() {
+ return gmtModified;
+ }
+
+ @Override
+ public void setGmtModified(Date gmtModified) {
+ this.gmtModified = gmtModified;
+ }
+
+ public String getMsgKey() {
+ return msgKey;
+ }
+
+ public void setMsgKey(String msgKey) {
+ this.msgKey = msgKey;
+ }
+
+ public String getWindowInstanceId() {
+ return windowInstanceId;
+ }
+
+ public void setWindowInstanceId(String windowInstanceId) {
+ this.windowInstanceId = windowInstanceId;
+ }
+
+ public String getPartition() {
+ return partition;
+ }
+
+ public void setPartition(String partition) {
+ this.partition = partition;
+ }
+
+ public long getPartitionNum() {
+ return partitionNum;
+ }
+
+ public void setPartitionNum(long partitionNum) {
+ this.partitionNum = partitionNum;
+ }
+
+ public String getStartTime() {
+ return startTime;
+ }
+
+ public void setStartTime(String startTime) {
+ this.startTime = startTime;
+ }
+
+ public String getEndTime() {
+ return endTime;
+ }
+
+ public void setEndTime(String endTime) {
+ this.endTime = endTime;
+ }
+
+ public String getFireTime() {
+ return fireTime;
+ }
+
+ public void setFireTime(String fireTime) {
+ this.fireTime = fireTime;
+ }
+
+ public long getUpdateVersion() {
+ return updateVersion.get();
+ }
+
+ public void setUpdateVersion(long updateVersion) {
+ this.updateVersion.set(updateVersion);
+ }
+
+ public String getWindowInstancePartitionId() {
+ return windowInstancePartitionId;
+ }
+
+ public void setWindowInstancePartitionId(String windowInstancePartitionId) {
+ this.windowInstancePartitionId = windowInstancePartitionId;
+ }
+
+ @Override
+ public WindowBaseValue clone() {
+ String json = this.toJson();
+ WindowBaseValue clonedValue = ReflectUtil.forInstance(this.getClass());
+ clonedValue.toObject(json);
+ return clonedValue;
+ }
+
+}
+
diff --git a/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/state/impl/JoinLeftState.java b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/state/impl/JoinLeftState.java
new file mode 100644
index 0000000..63dfe07
--- /dev/null
+++ b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/state/impl/JoinLeftState.java
@@ -0,0 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.window.state.impl;
+
+public class JoinLeftState extends JoinState {
+}
diff --git a/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/state/impl/JoinRightState.java b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/state/impl/JoinRightState.java
new file mode 100644
index 0000000..eb69161
--- /dev/null
+++ b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/state/impl/JoinRightState.java
@@ -0,0 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.window.state.impl;
+
+public class JoinRightState extends JoinState {
+}
diff --git a/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/state/impl/JoinState.java b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/state/impl/JoinState.java
new file mode 100644
index 0000000..3a85ecd
--- /dev/null
+++ b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/state/impl/JoinState.java
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.window.state.impl;
+
+import java.util.Date;
+
+import org.apache.rocketmq.streams.window.state.WindowBaseValue;
+
+public class JoinState extends WindowBaseValue {
+
+ private long windowId;
+
+ private String windowName;
+
+ private String windowNameSpace;
+
+ private String messageId;
+
+ private String messageKey;
+
+ private Date messageTime;
+
+ private String messageBody;
+
+ public long getWindowId() {
+ return windowId;
+ }
+
+ public void setWindowId(long windowId) {
+ this.windowId = windowId;
+ }
+
+ public String getWindowName() {
+ return windowName;
+ }
+
+ public void setWindowName(String windowName) {
+ this.windowName = windowName;
+ }
+
+ public String getWindowNameSpace() {
+ return windowNameSpace;
+ }
+
+ public void setWindowNameSpace(String windowNameSpace) {
+ this.windowNameSpace = windowNameSpace;
+ }
+
+ public String getMessageId() {
+ return messageId;
+ }
+
+ public void setMessageId(String messageId) {
+ this.messageId = messageId;
+ }
+
+ public String getMessageKey() {
+ return messageKey;
+ }
+
+ public void setMessageKey(String messageKey) {
+ this.messageKey = messageKey;
+ }
+
+ public Date getMessageTime() {
+ return messageTime;
+ }
+
+ public void setMessageTime(Date messageTime) {
+ this.messageTime = messageTime;
+ }
+
+ public String getMessageBody() {
+ return messageBody;
+ }
+
+ public void setMessageBody(String messageBody) {
+ this.messageBody = messageBody;
+ }
+}
diff --git a/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/state/impl/WindowValue.java b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/state/impl/WindowValue.java
new file mode 100644
index 0000000..da6097d
--- /dev/null
+++ b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/state/impl/WindowValue.java
@@ -0,0 +1,548 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.window.state.impl;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+
+import com.alibaba.fastjson.JSON;
+import com.alibaba.fastjson.JSONArray;
+import com.alibaba.fastjson.JSONObject;
+
+import org.apache.rocketmq.streams.common.interfaces.IStreamOperator;
+import org.apache.rocketmq.streams.common.utils.Base64Utils;
+import org.apache.rocketmq.streams.window.model.FunctionExecutor;
+import org.apache.rocketmq.streams.window.model.WindowCache;
+import org.apache.rocketmq.streams.window.model.WindowInstance;
+import org.apache.rocketmq.streams.window.operator.AbstractWindow;
+import org.apache.rocketmq.streams.window.state.WindowBaseValue;
+import org.apache.rocketmq.streams.script.context.FunctionContext;
+import org.apache.rocketmq.streams.common.datatype.DataType;
+import org.apache.rocketmq.streams.common.datatype.NotSupportDataType;
+import org.apache.rocketmq.streams.common.context.IMessage;
+import org.apache.rocketmq.streams.common.context.Message;
+import org.apache.rocketmq.streams.common.utils.DataTypeUtil;
+import org.apache.rocketmq.streams.common.utils.DateUtil;
+import org.apache.rocketmq.streams.common.utils.MapKeyUtil;
+import org.apache.rocketmq.streams.common.utils.ReflectUtil;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+import org.apache.rocketmq.streams.common.utils.TraceUtil;
+import org.apache.rocketmq.streams.script.operator.impl.AggregationScript;
+import org.apache.rocketmq.streams.script.operator.impl.FunctionScript;
+import org.apache.rocketmq.streams.script.service.IAccumulator;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.rocketmq.streams.db.driver.orm.ORMUtil;
+
+public class WindowValue extends WindowBaseValue implements Serializable {
+
+ private static final long serialVersionUID = 1083444850264401338L;
+
+ private static final Log LOG = LogFactory.getLog(WindowValue.class);
+
+ /**
+ * 如果做分组，设置分组
+ */
+ protected String groupBy = "0";
+
+ /**
+ * split id和max offset的映射关系
+ */
+ protected ConcurrentHashMap<String, String> maxOffset = new ConcurrentHashMap<>(16);
+
+ /**
+ * the result of aggregation column
+ */
+ private Map<String, Object> aggColumnResult = new ConcurrentHashMap<>(16);
+
+ /**
+ * the result of select column
+ */
+ protected Map<String, Object> computedColumnResult = new HashMap<>(16);
+
+ protected transient Long lastUpdateTime;//used in session window，set last update time
+
+ public WindowValue() {
+ setGmtCreate(DateUtil.getCurrentTime());
+ setGmtModified(DateUtil.getCurrentTime());
+ }
+
+ @Override
+ protected void getJsonObject(JSONObject jsonObject) {
+ super.getJsonObject(jsonObject);
+ String result = jsonObject.getString("aggColumnResult");
+ setAggColumnResult(result);
+ }
+
+ @Override
+ protected void setJsonObject(JSONObject jsonObject) {
+ super.setJsonObject(jsonObject);
+ if (aggColumnResult == null) {
+ return;
+ }
+ jsonObject.put("aggColumnResult", getAggColumnResult());
+
+ }
+
+ public WindowValue(WindowValue theValue) {
+ this.startTime = theValue.getStartTime();
+ this.endTime = theValue.getEndTime();
+ this.fireTime = theValue.getFireTime();
+ this.groupBy = theValue.getGroupBy();
+ setNameSpace(theValue.getNameSpace());
+ setConfigureName(theValue.getConfigureName());
+ }
+
+ /**
+ * 计算结果序列化成json
+ *
+ * @return
+ */
+ public String getAggColumnResult() {
+ JSONArray jsonArray = new JSONArray();
+
+ Iterator<Entry<String, Object>> it = aggColumnResult.entrySet().iterator();
+ while (it.hasNext()) {
+ JSONObject jsonObject = new JSONObject();
+ Entry<String, Object> entry = it.next();
+ String functionName = entry.getKey();
+ Object value = entry.getValue();
+ jsonObject.put("function", functionName);
+ if (value == null) {
+ continue;
+ }
+ DataType dataType = DataTypeUtil.getDataTypeFromClass(value.getClass());
+ boolean isBasicType = false;
+ String jsonValue = null;
+ if (!NotSupportDataType.class.isInstance(dataType)) {
+ isBasicType = true;
+ jsonValue = dataType.toDataJson(value);
+ jsonObject.put("datatype", dataType.getDataTypeName());
+ } else {
+ isBasicType = false;
+ jsonValue = ReflectUtil.serializeObject(value).toJSONString();
+ }
+
+ jsonObject.put("isBasic", isBasicType);
+ jsonObject.put("result", jsonValue);
+ jsonArray.add(jsonObject);
+ }
+ return encodeSQLContent(jsonArray.toJSONString());
+ }
+
+ /**
+ * 还原计算结果
+ *
+ * @param jsonArrayStr
+ */
+ public void setAggColumnResult(String jsonArrayStr) {
+ jsonArrayStr = decodeSQLContent(jsonArrayStr);
+ JSONArray functionResultJson = JSONArray.parseArray(jsonArrayStr);
+ for (int i = 0; i < functionResultJson.size(); i++) {
+ JSONObject jsonObject = functionResultJson.getJSONObject(i);
+ String functionName = jsonObject.getString("function");
+ Boolean isBasic = jsonObject.getBoolean("isBasic");
+ String jsonValue = jsonObject.getString("result");
+ Object value = null;
+ if (isBasic) {
+ String datatype = jsonObject.getString("datatype");
+ DataType dataType = DataTypeUtil.getDataType(datatype);
+ value = dataType.getData(jsonValue);
+ } else {
+ JSONObject objectJson = JSONObject.parseObject(jsonValue);
+ value = ReflectUtil.deserializeObject(objectJson);
+ }
+ aggColumnResult.put(functionName, value);
+ }
+ }
+
+ public void setAggColumnMap(Map<String, Object> aggColumnResult) {
+ this.aggColumnResult = aggColumnResult;
+ }
+
+ public void setComputedColumnResult(String computedColumnResult) {
+ computedColumnResult = decodeSQLContent(computedColumnResult);
+ this.computedColumnResult = Message.parseObject(computedColumnResult);
+ }
+
+ public String getComputedColumnResult() {
+ JSONObject object = null;
+ if (JSONObject.class.isInstance(computedColumnResult)) {
+ object = (JSONObject)computedColumnResult;
+ } else {
+ object = new JSONObject(computedColumnResult);
+ }
+ return encodeSQLContent(object.toJSONString());
+ }
+
+ public void setMaxOffset(String theOffset) {
+ JSONObject object = JSONObject.parseObject(theOffset);
+ Iterator<String> iterator = object.keySet().iterator();
+ while (iterator.hasNext()) {
+ String key = iterator.next();
+ maxOffset.put(key, object.getString(key));
+ }
+ }
+
+ public String getMaxOffset() {
+ JSONObject object = new JSONObject();
+ object.putAll(maxOffset);
+ return object.toJSONString();
+ }
+
+ public Iterator<Entry<String, Object>> iteratorComputedColumnResult() {
+ return computedColumnResult.entrySet().iterator();
+ }
+
+ public Object getComputedColumnResultByKey(String fieldName) {
+ return computedColumnResult.get(fieldName);
+ }
+
+ public Map<String, Object> getcomputedResult() {
+ return this.computedColumnResult;
+ }
+
+ /**
+ * @param window all kinds of configurable information
+ * @param message the consumed data
+ */
+ public boolean calculate(AbstractWindow window, IMessage message) {
+ message.getMessageBody().put(AbstractWindow.WINDOW_START, startTime);
+ message.getMessageBody().put(AbstractWindow.WINDOW_END, endTime);
+ //每个计算节点对应一个consume split，如果多于一个queue的话，
+ String queueId = message.getHeader().getQueueId();
+ String offset = message.getHeader().getOffset();
+ if (StringUtil.isEmpty(offset)) {
+ offset = String.valueOf(System.currentTimeMillis());
+ }
+ String maxOffsetOfQueue = this.maxOffset.get(queueId);
+ if (StringUtil.isEmpty(maxOffsetOfQueue)) {
+ maxOffsetOfQueue = offset;
+ this.maxOffset.put(queueId, maxOffsetOfQueue);
+ } else {
+ if (message.getHeader().greateThan(maxOffsetOfQueue)) {
+ this.maxOffset.put(queueId, offset);
+ } else {
+ //如果比最大的offset 小或等于，则直接丢弃掉消息
+ System.out.println("!!!!!!!!!!!!!!!!!!! has outOfOrder data");
+ return false;
+ }
+ }
+ try {
+ this.lastUpdateTime = WindowInstance.getOccurTime(window, message);
+ if (window.getReduceSerializeValue() != null) {
+ JSONObject accumulator = null;
+ if (computedColumnResult != null && JSONObject.class.isInstance(computedColumnResult)) {
+ accumulator = (JSONObject)computedColumnResult;
+ }
+
+ JSONObject result = window.getReducer().reduce(accumulator, message.getMessageBody());
+ computedColumnResult = result;
+ return true;
+ }
+ calFunctionColumn(window, message);
+ calProjectColumn(window, message);
+ String traceId = message.getMessageBody().getString(WindowCache.ORIGIN_MESSAGE_TRACE_ID);
+ if (!StringUtil.isEmpty(traceId)) {
+ TraceUtil.debug(traceId, "window value result", getComputedColumnResult());
+ }
+ } catch (Exception e) {
+ LOG.error("failed in calculating the message", e);
+ }
+
+ //there is no need writing back to message
+
+ return true;
+ }
+
+ protected void calFunctionColumn(AbstractWindow window, IMessage message) {
+ for (Entry<String, List<FunctionExecutor>> entry : window.getColumnExecuteMap().entrySet()) {
+ String computedColumn = entry.getKey();
+ List<FunctionExecutor> fifoQueue = entry.getValue();
+ for (FunctionExecutor operator : fifoQueue) {
+ String executorName = operator.getColumn();
+ IStreamOperator<IMessage, List<IMessage>> executor = operator.getExecutor();
+ if (executor instanceof AggregationScript) {
+ AggregationScript originAccScript = (AggregationScript)executor;
+ AggregationScript windowAccScript = originAccScript.clone();
+ Object accumulator = null;
+ if (aggColumnResult.containsKey(executorName)) {
+ accumulator = aggColumnResult.get(executorName);
+ } else {
+ IAccumulator director = AggregationScript.getAggregationFunction(
+ windowAccScript.getFunctionName());
+ accumulator = director.createAccumulator();
+ aggColumnResult.put(executorName, accumulator);
+ }
+ windowAccScript.setAccumulator(accumulator);
+ message.getMessageBody().put(AggregationScript.INNER_AGGREGATION_COMPUTE_KEY,
+ AggregationScript.INNER_AGGREGATION_COMPUTE_SINGLE);
+ FunctionContext context = new FunctionContext(message);
+ windowAccScript.doMessage(message, context);
+ } else if (executor instanceof FunctionScript) {
+ FunctionContext context = new FunctionContext(message);
+ ((FunctionScript)executor).doMessage(message, context);
+ }
+ }
+ //
+ computedColumnResult.put(computedColumn, message.getMessageBody().get(computedColumn));
+ }
+ }
+
+ protected void calProjectColumn(AbstractWindow window, IMessage message) {
+ Map<String, String> constMap = window.getColumnProjectMap();
+ for (Entry<String, String> entry : constMap.entrySet()) {
+ String computedColumn = entry.getKey();
+ String originColumn = entry.getValue();
+ if (message.getMessageBody().containsKey(originColumn)) {
+ computedColumnResult.put(computedColumn, message.getMessageBody().get(originColumn));
+ } else {
+ if (LOG.isWarnEnabled()) {
+ LOG.warn("field:\t " + originColumn + " lost!");
+ }
+ }
+ }
+ }
+
+ /**
+ * merge different window values into one window value which have the same group by value
+ *
+ * @param window the window definition
+ * @param windowInstances all window instance which belong to same window and have different group by value
+ * @return
+ */
+ public static List<WindowValue> mergeWindowValues(AbstractWindow window, List<WindowInstance> windowInstances) {
+ if (windowInstances == null || windowInstances.size() == 0) {
+ return new ArrayList<>();
+ }
+ StringBuilder sb = new StringBuilder();
+ boolean isFirst = true;
+ String name = MapKeyUtil.createKey(window.getNameSpace(), window.getConfigureName());
+ for (WindowInstance windowInstance : windowInstances) {
+ if (isFirst) {
+ isFirst = false;
+ } else {
+ sb.append(",");
+ }
+ sb.append("('" + name + "','" + windowInstance.getStartTime() + "','" + windowInstance.getEndTime() + "')");
+ }
+ String inSQL = sb.toString();
+ /**
+ * 分批，内存撑暴 todo
+ */
+ String sql = "select * from " + ORMUtil
+ .getTableName(WindowValue.class) + " where status > 0 && (name, start_time, end_time) in (" + inSQL + ")";
+ Map<String, Object> paras = new HashMap<>(4);
+ List<WindowValue> windowValueList = ORMUtil.queryForList(sql, paras, WindowValue.class);
+ return queryMergeWindowValues(window, windowValueList);
+ }
+
+ public static List<WindowValue> queryMergeWindowValues(AbstractWindow window, List<WindowValue> windowValueList) {
+ Map<String, List<WindowValue>> groupWindowMap = new HashMap<>(64);
+ for (WindowValue value : windowValueList) {
+ String key = MapKeyUtil.createKeyBySign(value.getStartTime(), value.getEndTime(),
+ value.getGroupBy());
+ if (groupWindowMap.containsKey(key)) {
+ groupWindowMap.get(key).add(value);
+ } else {
+ groupWindowMap.put(key, new ArrayList<WindowValue>() {{
+ add(value);
+ }});
+ }
+ }
+ List<WindowValue> mergedValueList = new ArrayList<>();
+ for (Entry<String, List<WindowValue>> entry : groupWindowMap.entrySet()) {
+ mergedValueList.add(mergeWindowValue(window, entry.getValue()));
+ }
+ return mergedValueList;
+ }
+
+ /**
+ * merge the group which has the same group by value and different split id
+ */
+ private static WindowValue mergeWindowValue(AbstractWindow window, List<WindowValue> valueList) {
+ WindowValue lastWindowValue = new WindowValue(valueList.get(0));
+ lastWindowValue.computedColumnResult = (Map<String, Object>)JSON.parse(
+ valueList.get(0).getComputedColumnResult());
+ //
+ for (Entry<String, List<FunctionExecutor>> entry : window.getColumnExecuteMap().entrySet()) {
+ String computedColumn = entry.getKey();
+ IMessage message = new Message(new JSONObject());
+ FunctionContext context = new FunctionContext(message);
+ List<FunctionExecutor> executorList = entry.getValue();
+ //column outside of the aggregation function should be calculated again!
+ boolean needMergeComputation = false;
+ for (FunctionExecutor info : executorList) {
+ String column = info.getColumn();
+ IStreamOperator<IMessage, List<IMessage>> engine = info.getExecutor();
+ if (engine instanceof AggregationScript) {
+ AggregationScript origin = (AggregationScript)engine;
+ AggregationScript operator = origin.clone();
+ if (needMergeComputation) {
+ message.getMessageBody().put(AggregationScript.INNER_AGGREGATION_COMPUTE_KEY,
+ AggregationScript.INNER_AGGREGATION_COMPUTE_SINGLE);
+ operator.setAccumulator(operator.getDirector().createAccumulator());
+ operator.doMessage(message, context);
+ } else {
+ message.getMessageBody().put(AggregationScript.INNER_AGGREGATION_COMPUTE_KEY,
+ AggregationScript.INNER_AGGREGATION_COMPUTE_MULTI);
+ List actors = valueList.stream().map(
+ windowValue -> windowValue.getAccumulatorByColumn(column)).collect(
+ Collectors.toList());
+ operator.setAccumulator(operator.getDirector().createAccumulator());
+ operator.setAccumulators(actors);
+ operator.doMessage(message, context);
+ needMergeComputation = true;
+ }
+ } else if (engine instanceof FunctionScript) {
+ FunctionScript theScript = (FunctionScript)engine;
+ String[] parameters = theScript.getDependentParameters();
+ for (String parameter : parameters) {
+ if (!message.getMessageBody().containsKey(parameter) && lastWindowValue.computedColumnResult
+ .containsKey(parameter)) {
+ message.getMessageBody().put(parameter,
+ lastWindowValue.computedColumnResult.get(parameter));
+ }
+ }
+ if (needMergeComputation) {
+ engine.doMessage(message, context);
+ }
+ }
+ }
+ if (message.getMessageBody().containsKey(computedColumn)) {
+ lastWindowValue.computedColumnResult.put(computedColumn, message.getMessageBody().get(computedColumn));
+ } else if (!needMergeComputation) {
+ lastWindowValue.computedColumnResult.put(computedColumn,
+ valueList.get(0).computedColumnResult.get(computedColumn));
+ }
+ }
+ // valueList.stream().map(value -> lastWindowValue.count += value.getCount());
+ //
+ List<String> traceList = new ArrayList<>();
+ for (WindowValue value : valueList) {
+ if (value.computedColumnResult.containsKey(TraceUtil.TRACE_ID_FLAG)) {
+ String traceIds = String.valueOf(value.computedColumnResult.get(TraceUtil.TRACE_ID_FLAG));
+ traceList.addAll(Arrays.asList(traceIds.split(",")));
+ }
+ }
+ if (!traceList.isEmpty()) {
+ StringBuffer buffer = new StringBuffer();
+ for (int i = 0; i < traceList.size(); i++) {
+ buffer.append(traceList.get(i));
+ if (i != traceList.size() - 1) {
+ buffer.append(",");
+ }
+ }
+ lastWindowValue.computedColumnResult.put(TraceUtil.TRACE_ID_FLAG, buffer.toString());
+ }
+ return lastWindowValue;
+ }
+
+ public Object getAccumulatorByColumn(String column) {
+ return aggColumnResult.getOrDefault(column, null);
+ }
+
+ public String getGroupBy() {
+ return groupBy;
+ }
+
+ public void setGroupBy(String groupBy) {
+ this.groupBy = groupBy;
+ }
+
+ @Override
+ public WindowValue clone() {
+ WindowValue clonedValue = new WindowValue();
+ clonedValue.setId(getId());
+ clonedValue.setGmtModified(gmtModified);
+ clonedValue.setGmtCreate(gmtCreate);
+ clonedValue.setEndTime(endTime);
+ clonedValue.setStartTime(startTime);
+ clonedValue.setFireTime(fireTime);
+ clonedValue.setConfigureName(getConfigureName());
+ clonedValue.setNameSpace(getNameSpace());
+ clonedValue.setMsgKey(msgKey);
+ clonedValue.setAggColumnMap(aggColumnResult);
+ clonedValue.setMaxOffset(getMaxOffset());
+ clonedValue.setWindowInstancePartitionId(windowInstancePartitionId);
+ clonedValue.setWindowInstanceId(windowInstanceId);
+ clonedValue.setPartition(partition);
+ clonedValue.setPartitionNum(partitionNum);
+ clonedValue.setGroupBy(groupBy);
+ clonedValue.setAggColumnResult(getAggColumnResult());
+ clonedValue.setComputedColumnResult(getComputedColumnResult());
+ clonedValue.setUpdateVersion(getUpdateVersion());
+ clonedValue.setVersion(getVersion());
+ clonedValue.setUpdateFlag(getUpdateFlag());
+ return clonedValue;
+ }
+
+ public WindowValue toMd5Value() {
+ WindowValue clonedValue = clone();
+ String md5MsgKey = StringUtil.createMD5Str(getMsgKey());
+ clonedValue.setMsgKey(md5MsgKey);
+ clonedValue.setWindowInstanceId(StringUtil.createMD5Str(clonedValue.getWindowInstanceId()));
+ clonedValue.setWindowInstancePartitionId(
+ StringUtil.createMD5Str(clonedValue.getWindowInstancePartitionId()));
+ return clonedValue;
+ }
+ //
+ //public WindowValue toOriginValue(boolean supportOutDate) {
+ // WindowValue clonedValue = clone();
+ // String windowInstanceId = WindowInstance.getWindowInstanceId(getNameSpace(), getConfigureName(), getStartTime(),
+ // getEndTime(), getFireTime(), supportOutDate);
+ // clonedValue.setMsgKey(MapKeyUtil
+ // .createKey(getPartition(), windowInstanceId, getGroupBy()));
+ // clonedValue.setWindowInstanceId(windowInstanceId);
+ // clonedValue.setWindowInstancePartitionId(
+ // MapKeyUtil.createKey(windowInstanceId, getPartition()));
+ // return clonedValue;
+ //}
+
+ public Long getLastUpdateTime() {
+ return lastUpdateTime;
+ }
+
+ public void setLastUpdateTime(Long lastUpdateTime) {
+ this.lastUpdateTime = lastUpdateTime;
+ }
+
+ protected String encodeSQLContent(String content) {
+ try {
+ return Base64Utils.encode(content.getBytes("UTF-8"));
+ } catch (Exception e) {
+ throw new RuntimeException("encode sql content error " + content, e);
+ }
+ }
+
+ protected String decodeSQLContent(String sqlContent) {
+ try {
+ return new String(Base64Utils.decode(sqlContent), "UTF-8");
+ } catch (Exception e) {
+ throw new RuntimeException("decode sql content error " + sqlContent, e);
+ }
+ }
+}
+
diff --git a/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/storage/AbstractWindowStorage.java b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/storage/AbstractWindowStorage.java
new file mode 100644
index 0000000..80514cb
--- /dev/null
+++ b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/storage/AbstractWindowStorage.java
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.window.storage;
+
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+import org.apache.rocketmq.streams.db.driver.batchloader.BatchRowLoader;
+import org.apache.rocketmq.streams.db.driver.batchloader.IRowOperator;
+import org.apache.rocketmq.streams.db.driver.orm.ORMUtil;
+import org.apache.rocketmq.streams.window.state.WindowBaseValue;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+public abstract class AbstractWindowStorage<T extends WindowBaseValue> implements IWindowStorage<T> {
+ protected boolean isLocalStorageOnly = false;
+ protected transient ExecutorService dataLoaderExecutor = new ThreadPoolExecutor(10, 10,
+ 0L, TimeUnit.MILLISECONDS,
+ new LinkedBlockingQueue<Runnable>());
+ ;
+
+ @Override
+ public Long getMaxShuffleId(String queueId, String windowNameSpace, String windowName, Class<T> clazz) {
+ if (isLocalStorageOnly) {
+ return null;
+ }
+ String sql = "select max(partition_num) as partition_num from " + ORMUtil.getTableName(clazz)
+ + " where name_space ='" + windowNameSpace + "' and configure_name='" + windowName + "' and `partition`='" + queueId + "'";
+ WindowBaseValue windowBaseValue = ORMUtil.queryForObject(sql, new HashMap<>(4), clazz);
+ if (windowBaseValue == null) {
+ return null;
+ }
+ return windowBaseValue.getPartitionNum();
+ }
+
+ @Override
+ public void multiPut(Map<String, T> map, String windowInstanceId, String queueId) {
+ throw new RuntimeException("can not support this method");
+ }
+
+ @Override
+ public Map<String, T> multiGet(Class<T> clazz, List<String> keys, String windowInstanceId, String queueId) {
+ throw new RuntimeException("can not support this method");
+ }
+
+ @Override
+ public void loadSplitData2Local(String queueId, String windowInstanceId, Class<T> clazz, IRowOperator processor) {
+ if (isLocalStorageOnly) {
+ return;
+ }
+ String windowInstancePartitionId = StringUtil.createMD5Str(windowInstanceId);
+ dataLoaderExecutor.execute(new Runnable() {
+ @Override
+ public void run() {
+ long start = System.currentTimeMillis();
+ BatchRowLoader batchRowLoader = new BatchRowLoader("partition_num",
+ "select * from " + ORMUtil.getTableName(clazz) + " where window_instance_partition_id ='"
+ + windowInstancePartitionId + "'", processor);
+ batchRowLoader.startLoadData();
+ ShufflePartitionManager.getInstance().setWindowInstanceFinished(windowInstanceId);
+ System.out.println(System.currentTimeMillis() - start);
+ System.out.println("");
+ }
+ });
+
+ }
+
+ @Override
+ public void put(String key, T value) {
+ Map<String, T> map = new HashMap<>();
+ map.put(key, value);
+ multiPut(map);
+ }
+
+ @Override
+ public T get(Class<T> clazz, String key) {
+ Map<String, T> result = multiGet(clazz, key);
+ if (result == null) {
+ return null;
+ }
+ return result.values().iterator().next();
+ }
+
+ @Override
+ public Map<String, T> multiGet(Class<T> clazz, String... keys) {
+ List<String> keyList = new ArrayList<>();
+ for (String key : keys) {
+ keyList.add(key);
+ }
+ return multiGet(clazz, keyList);
+ }
+
+ public boolean isLocalStorageOnly() {
+ return isLocalStorageOnly;
+ }
+
+ public void setLocalStorageOnly(boolean localStorageOnly) {
+ isLocalStorageOnly = localStorageOnly;
+ }
+
+}
diff --git a/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/storage/ICommonStorage.java b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/storage/ICommonStorage.java
new file mode 100644
index 0000000..95003ac
--- /dev/null
+++ b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/storage/ICommonStorage.java
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.window.storage;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * 对存储的统一抽象,最底层的抽象 T必须是可识别的对象
+ */
+public interface ICommonStorage<T> {
+
+ /**
+ * 支持单个key value的存储
+ *
+ * @param key
+ * @param value
+ */
+ void put(String key, T value);
+
+ //多组key value批量存储
+ void multiPut(Map<String, T> map);
+
+ //获取多个key的值
+ Map<String, T> multiGet(Class<T> clazz, List<String> keys);
+
+ //获取单个key的值
+ T get(Class<T> clazz, String key);
+
+ //获取多个key的值
+ Map<String, T> multiGet(Class<T> clazz, String... keys);
+
+ void removeKeys(Collection<String> keys);
+
+}
diff --git a/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/storage/IKeyGenerator.java b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/storage/IKeyGenerator.java
new file mode 100644
index 0000000..cb01eff
--- /dev/null
+++ b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/storage/IKeyGenerator.java
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.window.storage;
+
+/**
+ * 为了适配不同存储，可以通过key generator的实现来做非kv系统的映射
+ */
+public interface IKeyGenerator {
+
+ /**
+ * @param
+ * @return
+ */
+ String createKey();
+}
diff --git a/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/storage/IShufflePartitionManager.java b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/storage/IShufflePartitionManager.java
new file mode 100644
index 0000000..6fd4639
--- /dev/null
+++ b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/storage/IShufflePartitionManager.java
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.window.storage;
+
+import org.apache.rocketmq.streams.common.channel.split.ISplit;
+
+import java.util.Collection;
+
+public interface IShufflePartitionManager {
+
+ /**
+ * 这个分片是否可用本地存储
+ *
+ * @param shuffleId
+ * @return
+ */
+ boolean isLocalStorage(String shuffleId, String windowInstanceId);
+
+ void setLocalStorageInvalid(ISplit channelQueue);
+
+ void setLocalStorageInvalid(ISplit channelQueue, String windowInstanceId);
+
+ /**
+ * setLocalStorageInvalid 如果 shuffle id不存在，且
+ *
+ * @param shuffleId
+ */
+ boolean setLocalStorageValdateIfNotExist(String shuffleId, String windowInstanceId);
+
+ /**
+ * 当窗口实例触发后，通过这个方法，回收资源
+ *
+ * @param windowInstanceId
+ * @param queueIds
+ */
+ void clearWindowInstanceStorageStatus(String windowInstanceId, Collection<String> queueIds);
+
+}
diff --git a/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/storage/IStorage.java b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/storage/IStorage.java
new file mode 100644
index 0000000..de7e566
--- /dev/null
+++ b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/storage/IStorage.java
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */package org.apache.rocketmq.streams.window.storage;
+
+import org.apache.rocketmq.streams.db.driver.batchloader.IRowOperator;
+
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+public interface IStorage<T> {
+
+
+ /**
+ * 支持单个key value的存储
+ * @param key
+ * @param value
+ */
+ void put(String key,T value);
+
+ //多组key value批量存储
+ void mutilPut(Map<String,T> keyValue);
+
+ //获取单个key的值
+ T get(Class<T> clazz,IKeyGenerator keyGenerator,String key);
+
+ //获取多个key的值
+ Map<String,T> mutilGet(Class<T> clazz,IKeyGenerator keyGenerator,String... keyValues);
+ //获取多个key的值
+ Map<String,T> mutilGet(Class< T> clazz,IKeyGenerator keyGenerator, List<String> keys);
+
+ /**
+ * remove keys
+ * @param keys
+ */
+ void removeKeys(IKeyGenerator keyGenerator, Collection<String> keys);
+
+ /**
+ * remove keys by prefix
+ * @param keyPrefix
+ */
+ void removeKeyPrefix(IKeyGenerator keyGenerator,String keyPrefix);
+
+ /*
+ create Iterator by key prefix
+ */
+ Iterator<T> iterateByPrefix(IKeyGenerator keyGenerator,String keyPrefix,Class<T> clazz);
+
+
+ T putIfAbsent(T t,Class<T> clazz);
+
+
+ int count(IKeyGenerator keyGenerator,String key);
+
+ int incrementAndGet(IKeyGenerator keyGenerator,String key);
+
+
+ Iterator<T> queryByPrefixBetweenOrderByValue(IKeyGenerator keyGenerator,String keyPrefix, Object startIndexValue,Object endIndexValue,Class<T> clazz);
+
+
+
+ void loadByPrefixBetweenOrderByValue(IKeyGenerator keyGenerator,String keyPrefix, Object startIndexValue,Object endIndexValue,
+ IRowOperator rowOperator,Class<T> clazz);
+
+}
diff --git a/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/storage/IWindowStorage.java b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/storage/IWindowStorage.java
new file mode 100644
index 0000000..4c55cd7
--- /dev/null
+++ b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/storage/IWindowStorage.java
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.window.storage;
+
+import org.apache.rocketmq.streams.common.channel.split.ISplit;
+import org.apache.rocketmq.streams.db.driver.batchloader.IRowOperator;
+import org.apache.rocketmq.streams.window.state.WindowBaseValue;
+import org.apache.rocketmq.streams.window.storage.WindowStorage.WindowBaseValueIterator;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+public interface IWindowStorage<T extends WindowBaseValue> extends ICommonStorage<T> {
+
+ //多组key value批量存储
+ void multiPut(Map<String, T> map, String windowInstanceId, String queueId);
+
+ //获取多个key的值
+ Map<String, T> multiGet(Class<T> clazz, List<String> keys, String windowInstanceId, String queueId);
+
+ /***
+ * 把queueId 前缀的数据全部失效掉
+ * @param channelQueue 必须6位，64001 1280001 128分片总数，001第一个分片
+ */
+
+ void clearCache(ISplit channelQueue, Class<T> clazz);
+
+ /**
+ * 删除一个窗口实例的数据，包括远程和本地存储
+ */
+ void delete(String windowInstanceId, Set<String> queueIds, Class<T> clazz);
+
+ /**
+ * 加载一个窗口实例的数据到本地存储
+ */
+ WindowBaseValueIterator<T> loadWindowInstanceSplitData(String localStorePrefix, String queueId, String windowInstanceId, String keyPrefix,
+ Class<T> clazz);
+
+ /**
+ * 这个窗口实例，最大的分片序列号，主要是通过db获取
+ *
+ * @return
+ */
+ Long getMaxShuffleId(String queueId, String windowNameSpace, String windowName, Class<T> clazz);
+
+ /**
+ * 批量加载数据，放入本地缓存
+ *
+ * @param splitNumer
+ * @param rowOperator
+ */
+ void loadSplitData2Local(String splitNumer, String windowInstanceId, Class<T> clazz, IRowOperator rowOperator);
+}
diff --git a/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/storage/ShufflePartitionManager.java b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/storage/ShufflePartitionManager.java
new file mode 100644
index 0000000..9bc48a6
--- /dev/null
+++ b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/storage/ShufflePartitionManager.java
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.window.storage;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+public class ShufflePartitionManager {
+ private static ShufflePartitionManager instance = new ShufflePartitionManager();
+ protected Map<String, Boolean> splitId2AllWindowInstanceFinishInit = new HashMap<>();//split是否有效，这个分片下所有的window instacne都完成了初始化
+ protected Map<String, Boolean> windowInstanceId2FinishInit = new HashMap<>();//window instance 是否完成初始化
+ private ExecutorService executorService;
+
+ private ShufflePartitionManager() {
+ executorService = new ThreadPoolExecutor(10, 10,
+ 0L, TimeUnit.MILLISECONDS,
+ new LinkedBlockingQueue<Runnable>());
+
+ }
+
+ public static ShufflePartitionManager getInstance() {
+ return instance;
+ }
+
+ /**
+ * if window instance finished init， return true else return false
+ *
+ * @param splitId
+ * @param windowInstanceId
+ * @return
+ */
+ public boolean isWindowInstanceFinishInit(String splitId, String windowInstanceId) {
+ Boolean allSpliltFinish = splitId2AllWindowInstanceFinishInit.get(splitId);
+ if (allSpliltFinish != null && allSpliltFinish) {
+ return true;
+ }
+ Boolean windowInstanceInitFinished = windowInstanceId2FinishInit.get(windowInstanceId);
+ if (windowInstanceInitFinished != null && windowInstanceInitFinished) {
+ return true;
+ }
+ return false;
+ }
+
+ public synchronized void setSplitFinished(String splitId) {
+ splitId2AllWindowInstanceFinishInit.put(splitId, true);
+ }
+
+ public synchronized void setSplitInValidate(String splitId) {
+ splitId2AllWindowInstanceFinishInit.put(splitId, false);
+ }
+
+ public synchronized void setWindowInstanceFinished(String windowInstanceId) {
+ windowInstanceId2FinishInit.put(windowInstanceId, true);
+ }
+
+ public synchronized void clearWindowInstance(String windowInstanceId) {
+ windowInstanceId2FinishInit.remove(windowInstanceId);
+ }
+
+ public synchronized void clearSplit(String queueId) {
+ splitId2AllWindowInstanceFinishInit.remove(queueId);
+ for (String windowInstanceId : this.windowInstanceId2FinishInit.keySet()) {
+ if (windowInstanceId.startsWith(queueId)) {
+ this.windowInstanceId2FinishInit.remove(windowInstanceId);
+ }
+ }
+ }
+}
diff --git a/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/storage/StorageManager.java b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/storage/StorageManager.java
new file mode 100644
index 0000000..a78c848
--- /dev/null
+++ b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/storage/StorageManager.java
@@ -0,0 +1,307 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.window.storage;
+
+/**
+ * 存储管理，根据分片本地存储是否有效，来选择对应的存储完成计算
+ */
+public class StorageManager {
+ //private ExecutorService executorService= Executors.newFixedThreadPool(10);
+ //
+ //private ExecutorService dbService = Executors.newSingleThreadExecutor();
+ //
+ //private static StorageManager storageManager=new StorageManager();
+ //private StorageManager(){}
+ //protected static IStorage rocksDBStorage=new RocksdbStorage();//本地存储
+ //protected static DBStorage dbStorage=new DBStorage();//jdbc 存储
+ ////0/null:false;1:true;2加载中
+ ////
+ //protected transient boolean isLocalOnly=false;//只支持本地存储
+ //protected ConcurrentHashMap<String,Integer> shuffleIdAndWindowInstance2IsLocal=new ConcurrentHashMap<>();//某个分片是否本地存储有效
+ //public static StorageManager getStorageManager(){
+ // return storageManager;
+ //}
+ //
+ //public static IShufflePartitionManager getShufflePartitionManager(){
+ // return storageManager;
+ //}
+ //
+ //public static IStorage getLocalStorage(){
+ // return rocksDBStorage;
+ //}
+ //
+ //public static IStorage getRemoteStorage(){
+ // return dbStorage;
+ //}
+ //
+ //@Override
+ //public void put(Map<String, WindowBaseValue> values, boolean onlyLocal) {
+ // if(onlyLocal){
+ // rocksDBStorage.put(values, true);
+ // return;
+ // }
+ // Map<String,WindowBaseValue> notLocalWindowBaseValues=new HashMap<>();
+ // Map<String,WindowBaseValue> localWindowBaseValues=new HashMap<>();
+ // Iterator<Entry<String, WindowBaseValue>> it = values.entrySet().iterator();
+ // while (it.hasNext()){
+ // Entry<String, WindowBaseValue>entry=it.next();
+ // boolean isLocal=isLocalStorage(entry.getValue().getPartition(),entry.getValue().getWindowInstanceId());
+ // if(isLocal){
+ // localWindowBaseValues.put(entry.getKey(),entry.getValue());
+ // }else {
+ // notLocalWindowBaseValues.put(entry.getKey(),entry.getValue());
+ // }
+ // }
+ // rocksDBStorage.put(values, false);
+ // if(isLocalOnly){
+ // return;
+ // }
+ // /**
+ // *
+ // */
+ // if (!CollectionUtil.isEmpty(localWindowBaseValues)) {
+ // //如果本地可用，可以用异步的方式写，提高写性能
+ // dbService.execute(new Runnable() {
+ // @Override
+ // public void run() {
+ // dbStorage.put(getMd5Value(localWindowBaseValues), false);
+ // }
+ // });
+ // } else if (!CollectionUtil.isEmpty(notLocalWindowBaseValues)) {
+ // //如果本地不可用，必须同步写
+ // dbStorage.put(getMd5Value(notLocalWindowBaseValues), false);
+ // }
+ //}
+ //
+ ///**
+ // * DB存储时用MD5，TODO 考虑使用aop
+ // */
+ //private Map<String, WindowBaseValue> getMd5Value(Map<String, WindowBaseValue> originMap) {
+ // Map<String, WindowBaseValue> valueMap = new HashMap<>(originMap.size());
+ // Iterator<Entry<String, WindowBaseValue>> iterator = originMap.entrySet().iterator();
+ // while (iterator.hasNext()) {
+ // Entry<String, WindowBaseValue> entry = iterator.next();
+ // WindowBaseValue value = entry.getValue();
+ // if (value instanceof WindowValue) {
+ // WindowValue md5Value = (WindowValue)entry.getValue();
+ // valueMap.put(entry.getKey(), md5Value.toMd5Value());
+ // } else {
+ // //TODO join的MD5计算逻辑
+ // valueMap.put(entry.getKey(), entry.getValue());
+ // }
+ // }
+ // return valueMap;
+ //}
+ //
+ //@Override
+ //public Map<String, WindowBaseValue> get(Collection<String> keys, Class<? extends WindowBaseValue> clazz) {
+ // Map<String, WindowBaseValue> result=new HashMap<>();
+ // if(isLocalOnly){
+ // result.putAll(rocksDBStorage.get(keys,clazz));
+ // return result;
+ // }
+ //
+ // List<String> notLocalKeys=new ArrayList<>();
+ // List<String> localKeys=new ArrayList<>();
+ // for(String key:keys){
+ // String[] values=MapKeyUtil.spliteKey(key);
+ // String shuffleId=values[0];
+ // boolean isLocal = isLocalStorage(shuffleId, WindowInstance.createWindowInstanceId(key));
+ // if(isLocal){
+ // localKeys.add(key);
+ // }else {
+ // notLocalKeys.add(key);
+ // }
+ // }
+ //
+ // result.putAll(rocksDBStorage.get(localKeys,clazz));
+ // result.putAll(dbStorage.get(notLocalKeys,clazz));
+ // return result;
+ //}
+ //
+ //@Override
+ //public void delete(String windowNameSpace, String windowName, String startTime, String endOrFireTime,
+ // Class<? extends WindowBaseValue> clazz) {
+ // executorService.execute(new Runnable() {
+ //
+ // @Override
+ // public void run() {
+ // rocksDBStorage.delete(windowNameSpace, windowName, startTime, endOrFireTime, clazz);
+ // if(!isLocalOnly){
+ // dbStorage.delete(windowNameSpace, windowName, startTime, endOrFireTime, clazz);
+ // }
+ //
+ // }
+ // });
+ //
+ //}
+ //
+ //@Override
+ //public void clearCache(ISplit channelQueue) {
+ // rocksDBStorage.clearCache(channelQueue);
+ //}
+ //
+ //@Override
+ //public Iterator<WindowBaseValue> loadWindowInstanceSplitData(String queueId, String windowNameSpace,
+ // String windowName, String startTime, String endOrFireTime, String key, Class<? extends WindowBaseValue> clazz) {
+ // boolean isLocal = isLocalStorage(queueId,
+ // WindowInstance.getWindowInstanceId(windowNameSpace, windowName, startTime, endOrFireTime));
+ // if(isLocal){
+ // return rocksDBStorage.loadWindowInstanceSplitData(queueId,windowNameSpace,windowName,startTime,endOrFireTime,key, clazz);
+ // }else {
+ // return dbStorage.loadWindowInstanceSplitData(queueId,windowNameSpace,windowName,startTime,endOrFireTime,key, clazz);
+ // }
+ //}
+ //
+ //@Override
+ //public long getMaxShuffleId(String queueId, String windowNameSpace, String windowName, String startTime,
+ // String endOrFireTime, Class<? extends WindowBaseValue> clazz) {
+ // if(isLocalOnly){
+ // return 0;
+ // }
+ // return dbStorage.getMaxShuffleId(queueId,windowNameSpace,windowName,startTime,endOrFireTime,clazz);
+ //}
+ //
+ //@Override
+ //public void loadSplitData2Local(String splitNumer, String windowNameSpace, String windowName,
+ // String startTime, String endOrFireTime, Class<? extends WindowBaseValue> clazz, IRowOperator rowOperator) {
+ // if(isLocalOnly){
+ // return;
+ // }
+ // String windowInstanceId = WindowInstance.getWindowInstanceId(windowNameSpace, windowName, startTime, endOrFireTime);
+ // Integer value=this.shuffleIdAndWindowInstance2IsLocal.get(MapKeyUtil.createKey(splitNumer,windowInstanceId));
+ // if(value!=null&&value!=0){
+ // return;
+ // }
+ // synchronized (this){
+ // value=this.shuffleIdAndWindowInstance2IsLocal.get(MapKeyUtil.createKey(splitNumer,windowInstanceId));
+ // if(value!=null&&value!=0){
+ // return;
+ // }
+ // shuffleIdAndWindowInstance2IsLocal.put(MapKeyUtil.createKey(splitNumer,windowInstanceId),2);
+ // executorService.execute(new Runnable() {
+ // @Override
+ // public void run() {
+ // if (rowOperator == null) {
+ // dbStorage.loadSplitData2Local(splitNumer, windowNameSpace, windowName, startTime, endOrFireTime,
+ // clazz,
+ // new IRowOperator() {
+ // @Override
+ // public void doProcess(Map<String, Object> row) {
+ // WindowBaseValue theValue = ORMUtil.convert(row, clazz);
+ // List<String> keys = new ArrayList<>();
+ // keys.add(theValue.getMsgKey());
+ // WindowBaseValue windowBaseValue = (WindowBaseValue)rocksDBStorage.get(keys, clazz);
+ // if (windowBaseValue == null) {
+ // Map<String, WindowBaseValue> map = new HashMap<>();
+ // map.put(theValue.getMsgKey(), theValue);
+ // rocksDBStorage.put(map, true);
+ // return;
+ // }
+ // if (theValue.getUpdateVersion() > windowBaseValue.getUpdateVersion()) {
+ // Map<String, WindowBaseValue> map = new HashMap<>();
+ // map.put(theValue.getMsgKey(), theValue);
+ // rocksDBStorage.put(map, true);
+ // }
+ // }
+ // });
+ // } else {
+ // dbStorage.loadSplitData2Local(splitNumer, windowNameSpace, windowName, startTime, endOrFireTime,
+ // clazz, rowOperator);
+ // }
+ // shuffleIdAndWindowInstance2IsLocal.put(MapKeyUtil.createKey(splitNumer, windowInstanceId), 1);
+ // }
+ // });
+ // }
+ //
+ //
+ //}
+ //
+ //
+ //@Override
+ //public boolean isLocalStorage(String shuffleId,String windowInstanceId) {
+ // Integer value=this.shuffleIdAndWindowInstance2IsLocal.get(MapKeyUtil.createKey(shuffleId,windowInstanceId));
+ // return isLocalStorage(value)||isLocalOnly;
+ //}
+ //@Override
+ //public void setLocalStorageInvalid(ISplit channelQueue,String windowInstanceId) {
+ // this.shuffleIdAndWindowInstance2IsLocal.remove(MapKeyUtil.createKey(channelQueue.getQueueId(),windowInstanceId),false);
+ //}
+ //@Override
+ //public void setLocalStorageInvalid(ISplit channelQueue) {
+ // Iterator<Entry<String, Integer>> it = this.shuffleIdAndWindowInstance2IsLocal.entrySet().iterator();
+ // List<String> keys=new ArrayList<>();
+ // while (it.hasNext()){
+ // Entry<String, Integer> entry=it.next();
+ // String key=entry.getKey();
+ // if(key.startsWith(channelQueue.getQueueId())){
+ // this.shuffleIdAndWindowInstance2IsLocal.put(key,0);
+ // keys.add(entry.getKey());
+ // }
+ //
+ // }
+ // executorService.execute(new Runnable() {
+ // @Override
+ // public void run() {
+ // clearCache(channelQueue);
+ // for(String key:keys){
+ // shuffleIdAndWindowInstance2IsLocal.remove(key);
+ // }
+ // }
+ // });
+ //}
+ //
+ //@Override
+ //public boolean setLocalStorageValdateIfNotExist(String shuffleId,String windowInstanceId) {
+ // Integer value = this.shuffleIdAndWindowInstance2IsLocal.get(
+ // MapKeyUtil.createKey(shuffleId, windowInstanceId));
+ // if (value != null) {
+ // return isLocalStorage(value);
+ // }
+ // this.shuffleIdAndWindowInstance2IsLocal.put( MapKeyUtil.createKey(shuffleId, windowInstanceId), 1);
+ // return true;
+ //}
+ //
+ //
+ //@Override
+ //public void clearWindowInstanceStorageStatus(String windowInstanceId,Collection<String> queueIds){
+ // if(queueIds!=null){
+ // for(String queueId:queueIds){
+ // this.shuffleIdAndWindowInstance2IsLocal.remove(MapKeyUtil.createKey(queueId,windowInstanceId));
+ // }
+ // }
+ //}
+ //
+ //
+ //public boolean isLocalStorage(Integer value){
+ // if(value!=null&&value==1){
+ // return true;
+ // }
+ // if(isLocalOnly){
+ // return true;
+ // }
+ // return false;
+ //}
+ //
+ //public boolean isLocalOnly() {
+ // return isLocalOnly;
+ //}
+ //
+ //public void setLocalOnly(boolean localOnly) {
+ // isLocalOnly = localOnly;
+ //}
+}
diff --git a/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/storage/WindowStorage.java b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/storage/WindowStorage.java
new file mode 100644
index 0000000..a3eff00
--- /dev/null
+++ b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/storage/WindowStorage.java
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.window.storage;
+
+import org.apache.rocketmq.streams.common.channel.split.ISplit;
+import org.apache.rocketmq.streams.common.utils.MapKeyUtil;
+import org.apache.rocketmq.streams.window.state.WindowBaseValue;
+import org.apache.rocketmq.streams.window.storage.db.DBStorage;
+import org.apache.rocketmq.streams.window.storage.rocksdb.RocksdbStorage;
+
+import java.util.*;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+public class WindowStorage<T extends WindowBaseValue> extends AbstractWindowStorage<T> {
+ protected transient ShufflePartitionManager shufflePartitionManager = ShufflePartitionManager.getInstance();
+ protected IWindowStorage localStorage;
+ protected IWindowStorage remoteStorage;
+
+ private ExecutorService executorService;
+
+ //private ExecutorService dbService;
+ public WindowStorage(boolean isLoaclStorageOnly) {
+ this();
+ this.isLocalStorageOnly = isLoaclStorageOnly;
+ }
+
+ public WindowStorage() {
+ localStorage = new RocksdbStorage();
+ remoteStorage = new DBStorage();
+ executorService = new ThreadPoolExecutor(10, 10,
+ 0L, TimeUnit.MILLISECONDS,
+ new LinkedBlockingQueue<Runnable>());
+ //dbService= new ThreadPoolExecutor(1, 1,
+ // 0L, TimeUnit.MILLISECONDS,
+ // new LinkedBlockingQueue<Runnable>());
+
+ }
+
+ @Override
+ public WindowBaseValueIterator<T> loadWindowInstanceSplitData(String localStorePrefix, String queueId, String windowInstanceId, String keyPrefix,
+ Class<T> clazz) {
+ if (isLocalStorageOnly) {
+ return localStorage.loadWindowInstanceSplitData(localStorePrefix, queueId, windowInstanceId, keyPrefix, clazz);
+ }
+ if (shufflePartitionManager.isWindowInstanceFinishInit(queueId, windowInstanceId)) {
+ return localStorage.loadWindowInstanceSplitData(localStorePrefix, queueId, windowInstanceId, keyPrefix, clazz);
+ }
+ return remoteStorage.loadWindowInstanceSplitData(localStorePrefix, queueId, windowInstanceId, keyPrefix
+ , clazz);
+ }
+
+ protected transient ExecutorService executor = new ThreadPoolExecutor(10, 10,
+ 0L, TimeUnit.MILLISECONDS,
+ new LinkedBlockingQueue<Runnable>(100));
+
+ @Override
+ public void multiPut(Map<String, T> values, String windowInstanceId, String queueId) {
+ localStorage.multiPut(values);
+ if (isLocalStorageOnly) {
+ return;
+ }
+ if (shufflePartitionManager.isWindowInstanceFinishInit(queueId, windowInstanceId)) {
+ //可以考虑异步
+
+ remoteStorage.multiPut(values);
+
+ return;
+ }
+ remoteStorage.multiPut(values);
+ }
+
+ @Override
+ public Map<String, T> multiGet(Class<T> clazz, List<String> keys, String windowInstanceId, String queueId) {
+ if (isLocalStorageOnly || shufflePartitionManager.isWindowInstanceFinishInit(queueId, windowInstanceId)) {
+ return localStorage.multiGet(clazz, keys);
+ }
+ return remoteStorage.multiGet(clazz, keys);
+ }
+
+ @Override
+ public void multiPut(Map<String, T> values) {
+ localStorage.multiPut(values);
+ if (isLocalStorageOnly) {
+ return;
+ }
+ remoteStorage.multiPut(values);
+
+ }
+
+ @Override
+ public Map<String, T> multiGet(Class<T> clazz, List<String> keys) {
+ Map<String, T> result = new HashMap<>();
+ if (isLocalStorageOnly) {
+ result.putAll(localStorage.multiGet(clazz, keys));
+ return result;
+ }
+
+ List<String> notLocalKeys = new ArrayList<>();
+ List<String> localKeys = new ArrayList<>();
+ for (String key : keys) {
+ String[] values = MapKeyUtil.spliteKey(key);
+ String shuffleId = values[0];
+ boolean isLocal = shufflePartitionManager.isWindowInstanceFinishInit(shuffleId, createWindowInstanceId(key));
+ if (isLocal) {
+ localKeys.add(key);
+ } else {
+ notLocalKeys.add(key);
+ }
+ }
+
+ result.putAll(localStorage.multiGet(clazz, localKeys));
+ result.putAll(remoteStorage.multiGet(clazz, notLocalKeys));
+ return result;
+ }
+
+ @Override
+ public void removeKeys(Collection<String> keys) {
+ localStorage.removeKeys(keys);
+ }
+
+ /**
+ * refer to: WindowMessageProcessor.createStoreKey
+ */
+ public static String createWindowInstanceId(String msgKey) {
+ String[] values = MapKeyUtil.spliteKey(msgKey);
+ String[] lastValues = Arrays.copyOfRange(values, 1, values.length - 2);
+ //values[4]: endTime or fireTime
+ return MapKeyUtil.createKey(lastValues);
+ }
+
+ @Override
+ public void delete(String windowInstanceId, Set<String> queueIds, Class<T> clazz) {
+ executorService.execute(new Runnable() {
+
+ @Override
+ public void run() {
+ localStorage.delete(windowInstanceId, queueIds, clazz);
+ if (!isLocalStorageOnly) {
+ remoteStorage.delete(windowInstanceId, queueIds, clazz);
+ }
+
+ }
+ });
+ }
+
+ public static abstract class WindowBaseValueIterator<T extends WindowBaseValue> implements Iterator<T> {
+ protected long partitionNum = -1;
+
+ public void setPartitionNum(long partitionNum) {
+ this.partitionNum = partitionNum;
+ }
+ }
+
+ @Override
+ public void clearCache(ISplit split, Class<T> clazz) {
+ localStorage.clearCache(split, clazz);
+ }
+
+ public IWindowStorage getLocalStorage() {
+ return localStorage;
+ }
+
+ public IWindowStorage getRemoteStorage() {
+ return remoteStorage;
+ }
+}
diff --git a/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/storage/db/DBStorage.java b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/storage/db/DBStorage.java
new file mode 100644
index 0000000..3d2b2e2
--- /dev/null
+++ b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/storage/db/DBStorage.java
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.window.storage.db;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.rocketmq.streams.common.channel.split.ISplit;
+import org.apache.rocketmq.streams.common.utils.CollectionUtil;
+import org.apache.rocketmq.streams.common.utils.MapKeyUtil;
+import org.apache.rocketmq.streams.common.utils.SQLUtil;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+import org.apache.rocketmq.streams.db.driver.orm.ORMUtil;
+import org.apache.rocketmq.streams.window.state.WindowBaseValue;
+import org.apache.rocketmq.streams.window.storage.AbstractWindowStorage;
+import org.apache.rocketmq.streams.window.storage.WindowStorage.WindowBaseValueIterator;
+
+/**
+ * database storage
+ */
+public class DBStorage<T extends WindowBaseValue> extends AbstractWindowStorage<T> {
+
+ @Override
+ public void multiPut(Map<String, T> values) {
+ if (CollectionUtil.isEmpty(values)) {
+ return;
+ }
+ ORMUtil.batchReplaceInto(values.values());
+ }
+
+ @Override
+ public Map<String, T> multiGet(Class<T> clazz, List<String> keys) {
+ if (CollectionUtil.isEmpty(keys)) {
+ return new HashMap<>(4);
+ }
+ Map<String, String> md5Key2Keys = new HashMap<>();
+ List<String> md5Keys = new ArrayList<>();
+ for (String key : keys) {
+ String md5Key = StringUtil.createMD5Str(key);
+ md5Keys.add(md5Key);
+ md5Key2Keys.put(md5Key, key);
+ }
+ List<T> values = ORMUtil.queryForList("select * from " + ORMUtil.getTableName(clazz) +
+ " where msg_key in (" + SQLUtil.createInSql(md5Keys) + " )", new HashMap<>(4), clazz);
+ Map<String, T> map = new HashMap<>(keys.size());
+ for (T value : values) {
+ String key = md5Key2Keys.get(value.getMsgKey());
+ map.put(key, value);
+ }
+ return map;
+ }
+
+ @Override
+ public void removeKeys(Collection<String> keys) {
+
+ }
+
+ @Override
+ public WindowBaseValueIterator<T> loadWindowInstanceSplitData(String localStorePrefix, String queueId, String windowInstanceId, String keyPrex, Class<T> clazz) {
+
+ //search max partition number in case of inserting fresh data [min,max)
+ long maxPartitionIndex = getPartitionNum(queueId, windowInstanceId, clazz, true) + 1;
+ long mimPartitionIndex = getPartitionNum(queueId, windowInstanceId, clazz, false) - 1;
+ if (maxPartitionIndex <= 1) {
+ return new WindowBaseValueIterator<T>() {
+ @Override
+ public boolean hasNext() {
+ return false;
+ }
+
+ @Override
+ public T next() {
+ return null;
+ }
+ };
+ }
+
+ DBIterator dbIterator = new DBIterator<T>(queueId, windowInstanceId, keyPrex, clazz, maxPartitionIndex);
+ dbIterator.setPartitionNum(mimPartitionIndex);
+ return dbIterator;
+ }
+
+ @Override
+ public void clearCache(ISplit channelQueue, Class<T> clazz) {
+ throw new RuntimeException("can not support this method");
+ }
+
+ @Override
+ public void delete(String windowInstanceId, Set<String> queueIds, Class<T> clazz) {
+ String sql = "delete from " + ORMUtil.getTableName(clazz) + " where window_instance_id = '" + StringUtil.createMD5Str(windowInstanceId) + "'";
+ ORMUtil.executeSQL(
+ sql,
+ new HashMap<>(4));
+ }
+
+ public static class DBIterator<T extends WindowBaseValue> extends WindowBaseValueIterator<T> {
+ private LinkedList<T> container = new LinkedList<>();
+ int batchSize = 1000;
+ private boolean exist = true;
+
+ private long maxPartitionIndex;
+ private Class<T> clazz;
+
+ String sql;
+
+ public DBIterator(String queueId, String windowInstanceId, String keyPrex, Class<T> clazz, long maxPartitionIndex) {
+ String windowInstancePartitionId = StringUtil.createMD5Str(windowInstanceId);
+
+ if (StringUtil.isEmpty(keyPrex)) {
+ sql = "select * from " + ORMUtil.getTableName(clazz)
+ + " where window_instance_partition_id = '" + windowInstancePartitionId
+ + "' and partition_num > #{partitionNum} order by window_instance_partition_id, partition_num limit "
+ + batchSize;
+ } else {
+ //join usage(different clazz)
+ String prefix = MapKeyUtil.createKey(queueId, windowInstanceId, keyPrex);
+ sql = "select * from " + ORMUtil.getTableName(clazz) + " where window_instance_partition_id ='"
+ + windowInstancePartitionId + "' " +
+ "and msg_key like '" + prefix
+ + "%' and partition_num > #{partitionNum} order by window_instance_partition_id, partition_num limit "
+ + batchSize;
+ }
+ this.maxPartitionIndex = maxPartitionIndex;
+ this.clazz = clazz;
+ }
+
+ @Override
+ public boolean hasNext() {
+ if (!container.isEmpty()) {
+ return true;
+ } else if (!exist) {
+ return false;
+ } else {
+ Map<String, Long> parameter = new HashMap<>(4);
+ parameter.put("partitionNum", partitionNum);
+ exist = partitionNum + batchSize <= maxPartitionIndex;
+ List<T> batchResult = ORMUtil.queryForList(sql, parameter, clazz);
+ if (CollectionUtil.isEmpty(batchResult)) {
+ return false;
+ } else {
+ partitionNum = batchResult.get(batchResult.size() - 1).getPartitionNum();
+ container.addAll(batchResult);
+ return true;
+ }
+ }
+ }
+
+ @Override
+ public T next() {
+ return container.poll();
+ }
+
+ }
+
+ protected Long getPartitionNum(String queueId, String windowInstanceId, Class<T> clazz, boolean isMax) {
+ String partitionNumSQL = isMax ? "max(partition_num)" : "min(partition_num)";
+ String windowInstancePartitionId = StringUtil.createMD5Str(windowInstanceId);
+ String sql = "select " + partitionNumSQL + " as partition_num from " + ORMUtil.getTableName(clazz)
+ + " where window_instance_partition_id ='" + windowInstancePartitionId + "'";
+ WindowBaseValue windowBaseValue = ORMUtil.queryForObject(sql, new HashMap<>(4), clazz);
+ if (windowBaseValue == null) {
+ return null;
+ }
+ return windowBaseValue.getPartitionNum();
+ }
+
+}
diff --git a/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/storage/file/FileStorage.java b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/storage/file/FileStorage.java
new file mode 100644
index 0000000..83dd299
--- /dev/null
+++ b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/storage/file/FileStorage.java
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.window.storage.file;
+
+import org.apache.rocketmq.streams.common.channel.split.ISplit;
+import org.apache.rocketmq.streams.common.utils.FileUtil;
+import org.apache.rocketmq.streams.common.utils.MapKeyUtil;
+import org.apache.rocketmq.streams.common.utils.ReflectUtil;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+import org.apache.rocketmq.streams.window.state.WindowBaseValue;
+import org.apache.rocketmq.streams.window.state.impl.WindowValue;
+import org.apache.rocketmq.streams.window.storage.AbstractWindowStorage;
+import org.apache.rocketmq.streams.window.storage.WindowStorage.WindowBaseValueIterator;
+
+import java.util.*;
+import java.util.Map.Entry;
+
+public class FileStorage<T extends WindowBaseValue> extends AbstractWindowStorage<T> {
+ private static final String SPLIT_SIGN = "############";
+ protected transient String filePath = "/tmp/storage/file.storage";
+ protected transient Map<String, String> cache = new HashMap<>();
+
+ @Override
+ public synchronized void clearCache(ISplit channelQueue, Class<T> clazz) {
+ String queueId = channelQueue.getQueueId();
+ deleteByKeyPrefix(queueId);
+ this.flush();
+ }
+
+ @Override
+ public synchronized void delete(String windowInstanceId, Set<String> queueIds, Class<T> clazz) {
+ Set<String> currentQueueIds = new HashSet<>(queueIds);
+ Iterator<String> it = currentQueueIds.iterator();
+ while (it.hasNext()) {
+ String queueId = it.next();
+ String firstKey = MapKeyUtil.createKey(queueId, windowInstanceId);
+ deleteByKeyPrefix(firstKey);
+ }
+ this.flush();
+ }
+
+ @Override
+ public synchronized WindowBaseValueIterator<T> loadWindowInstanceSplitData(String localStorePrefix, String queueId,
+ String windowInstanceId, String key, Class<T> clazz) {
+ String keyPrefix = MapKeyUtil.createKey(queueId, windowInstanceId, key);
+ if (StringUtil.isNotEmpty(localStorePrefix)) {
+ keyPrefix = localStorePrefix + keyPrefix;
+ }
+ final String keyPrefixFinnal = keyPrefix;
+ Map<String, String> copyCache = new HashMap<>();
+ copyCache.putAll(this.cache);
+ final Iterator<Entry<String, String>> iter = copyCache.entrySet().iterator();
+ return new WindowBaseValueIterator<T>() {
+
+ @Override
+ public boolean hasNext() {
+ return iter.hasNext();
+ }
+
+ @Override
+ public T next() {
+ while (iter.hasNext()) {
+ Entry<String, String> entry = iter.next();
+ String key = entry.getKey();
+ String value = entry.getValue();
+ if (key.startsWith(keyPrefixFinnal)) {
+ T jsonable = ReflectUtil.forInstance(clazz);
+ jsonable.toObject(value);
+ return jsonable;
+ }
+ }
+ return null;
+ }
+ };
+ }
+
+ @Override
+ public synchronized void multiPut(Map<String, T> map) {
+ for (Entry<String, T> entry : map.entrySet()) {
+ String key = entry.getKey();
+ String value = entry.getValue().toJson();
+ this.cache.put(key, value);
+ }
+ this.flush();
+ }
+
+ @Override
+ public synchronized Map<String, T> multiGet(Class<T> clazz, List<String> keys) {
+ Map<String, T> result = new HashMap<>();
+ for (String key : keys) {
+ String value = this.cache.get(key);
+ if (StringUtil.isNotEmpty(value)) {
+ T jsonable = ReflectUtil.forInstance(clazz);
+ jsonable.toObject(value);
+ result.put(key, jsonable);
+ }
+ }
+ return result;
+ }
+
+ @Override
+ public synchronized void removeKeys(Collection<String> keys) {
+ for (String key : keys) {
+ this.cache.remove(key);
+ }
+ this.flush();
+ }
+
+ protected synchronized void deleteByKeyPrefix(String keyPrefix) {
+ Map<String, String> copyCache = new HashMap<>();
+ copyCache.putAll(this.cache);
+ Iterator<Entry<String, String>> it = copyCache.entrySet().iterator();
+ while (it.hasNext()) {
+ Entry<String, String> entry = it.next();
+ String key = entry.getKey();
+ if (key.startsWith(keyPrefix)) {
+ cache.remove(key);
+ }
+ }
+ }
+
+ private synchronized void flush() {
+ List<String> buffer = new ArrayList<>();
+ Map<String, String> copyCache = new HashMap<>();
+ copyCache.putAll(this.cache);
+
+ Iterator<Entry<String, String>> it = copyCache.entrySet().iterator();
+ while (it.hasNext()) {
+ Entry<String, String> entry = it.next();
+ String key = entry.getKey();
+ String value = entry.getValue();
+ String line = MapKeyUtil.createKeyBySign(SPLIT_SIGN, key, value);
+ buffer.add(line);
+ }
+ FileUtil.write(filePath, buffer, false);
+ }
+
+ private void load() {
+ Map<String, String> cache = new HashMap<>();
+ List<String> buffer = FileUtil.loadFileLine(filePath);
+ for (String line : buffer) {
+ String[] values = line.split(SPLIT_SIGN);
+ cache.put(values[0], values[1]);
+ }
+ this.cache = cache;
+ }
+
+ //_order_by_split_num_1;1;namespace;name_window_10001;2021-07-13 15:07:40;2021-07-13 02:35:10;2021-07-13 02:35:15
+
+ public static void main(String[] args) {
+ FileStorage fileStorage = new FileStorage();
+ fileStorage.load();
+ fileStorage.deleteByKeyPrefix("_order_by_split_num_1;1;namespace;name_window_10001;2021-07-13 15:07:40;2021-07-13 02:35:10;2021-07-13 02:35:15");
+ fileStorage.deleteByKeyPrefix("1;1;namespace;name_window_10001;2021-07-13 15:07:40;2021-07-13 02:35:10;2021-07-13 02:35:15");
+ fileStorage.deleteByKeyPrefix("1;1;namespace;name_window_10001;2021-07-13 15:07:40;2021-07-13 02:35:05;2021-07-13 02:35:10");
+ fileStorage.deleteByKeyPrefix(" _order_by_split_num_1;1;namespace;name_window_10001;2021-07-13 15:07:40;2021-07-13 02:35:05;2021-07-13 02:35:10");
+ fileStorage.flush();
+ fileStorage.load();
+ WindowBaseValueIterator<WindowValue> fileIterator = fileStorage.loadWindowInstanceSplitData("_order_by_split_num_", "1", "1;namespace;name_window_10001",
+ null, WindowValue.class);
+ int sum = 0;
+ while (fileIterator.hasNext()) {
+ WindowValue windowValue = fileIterator.next();
+ if (windowValue == null) {
+ break;
+ }
+ sum++;
+ }
+
+ fileIterator = fileStorage.loadWindowInstanceSplitData(null, "1", "1;namespace;name_window_10001",
+ null, WindowValue.class);
+ while (fileIterator.hasNext()) {
+ WindowValue windowValue = fileIterator.next();
+ if (windowValue == null) {
+ break;
+ }
+ sum++;
+ }
+ System.out.println(sum);
+
+ }
+}
diff --git a/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/storage/rocksdb/RocksdbStorage.java b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/storage/rocksdb/RocksdbStorage.java
new file mode 100644
index 0000000..a603302
--- /dev/null
+++ b/rocketmq-streams-window/src/main/java/org/apache/rocketmq/streams/window/storage/rocksdb/RocksdbStorage.java
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.window.storage.rocksdb;
+
+import org.apache.rocketmq.streams.common.channel.split.ISplit;
+import org.apache.rocketmq.streams.common.utils.*;
+import org.apache.rocketmq.streams.window.state.WindowBaseValue;
+import org.apache.rocketmq.streams.window.storage.AbstractWindowStorage;
+import org.apache.rocketmq.streams.window.storage.WindowStorage.WindowBaseValueIterator;
+import org.rocksdb.*;
+
+import java.io.File;
+import java.io.UnsupportedEncodingException;
+import java.util.*;
+import java.util.Map.Entry;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+public class RocksdbStorage<T extends WindowBaseValue> extends AbstractWindowStorage<T> {
+ protected static String DB_PATH = "/tmp/rocksdb";
+ protected static String UTF8 = "UTF8";
+ protected static AtomicBoolean hasCreate = new AtomicBoolean(false);
+ protected static RocksDB rocksDB;
+ protected WriteOptions writeOptions = new WriteOptions();
+
+ static {
+ RocksDB.loadLibrary();
+ }
+
+ public RocksdbStorage() {
+ this(FileUtil.concatFilePath(StringUtil.isEmpty(FileUtil.getJarPath()) ? DB_PATH + File.separator + RuntimeUtil.getDipperInstanceId() : FileUtil.getJarPath() + File.separator + RuntimeUtil.getDipperInstanceId(), "rocksdb"));
+ }
+
+ public RocksdbStorage(String rocksdbFilePath) {
+ if (hasCreate.compareAndSet(false, true)) {
+ synchronized (RocksdbStorage.class) {
+ if (RocksdbStorage.rocksDB == null) {
+ synchronized (RocksdbStorage.class) {
+ if (RocksdbStorage.rocksDB == null) {
+ try (final Options options = new Options().setCreateIfMissing(true)) {
+
+ try {
+ File dir = new File(rocksdbFilePath);
+ if (dir.exists()) {
+ dir.delete();
+ }
+ dir.mkdirs();
+ final TtlDB db = TtlDB.open(options, rocksdbFilePath, 10800, false);
+ RocksdbStorage.rocksDB = db;
+ writeOptions.setSync(true);
+ } catch (RocksDBException e) {
+ throw new RuntimeException("create rocksdb error " + e.getMessage());
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+
+ }
+
+ @Override
+ public void removeKeys(Collection<String> keys) {
+
+ for (String key : keys) {
+ try {
+ rocksDB.delete(getKeyBytes(key));
+ } catch (RocksDBException e) {
+ throw new RuntimeException("delete error " + key);
+ }
+ }
+
+ }
+
+ @Override
+ public WindowBaseValueIterator<T> loadWindowInstanceSplitData(String localStorePrefix, String queueId, String windowInstanceId, String key, Class<T> clazz) {
+ String keyPrefix = MapKeyUtil.createKey(queueId, windowInstanceId, key);
+ if (StringUtil.isNotEmpty(localStorePrefix)) {
+ keyPrefix = localStorePrefix + keyPrefix;
+ }
+ return getByKeyPrefix(keyPrefix, clazz, false);
+ }
+
+ @Override
+ public void multiPut(Map<String, T> values) {
+ if (values == null) {
+ return;
... 6737 lines suppressed ...
[rocketmq-streams] 01/35: first commit
Posted by du...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
duhengforever pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/rocketmq-streams.git
commit 997f3dfc3b8ba4d3feaca97298e8da741be44d24
Author: duhenglucky <du...@apache.org>
AuthorDate: Tue Jul 13 21:07:47 2021 +0800
first commit
---
README.md | 1 +
1 file changed, 1 insertion(+)
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..49cd84d
--- /dev/null
+++ b/README.md
@@ -0,0 +1 @@
+# rocketmq-streams
[rocketmq-streams] 20/35: Merge pull request #6 from RonzL/main
Posted by du...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
duhengforever pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/rocketmq-streams.git
commit f349dd946ba29bdbc05ae958602c8b6936476d50
Merge: 6d89c9a 9c898ec
Author: Heng Du <du...@apache.org>
AuthorDate: Mon Aug 2 14:59:15 2021 +0800
Merge pull request #6 from RonzL/main
add module db-operator、transport-minio
rocketmq-streams-db-operator/pom.xml | 34 ++
.../rocketmq-streams-db-operator.iml | 16 +
.../streams/db/configuable/DBConfigureService.java | 282 ++++++++++++
.../DBSupportParentConfigureService.java | 37 ++
.../rocketmq/streams/db/driver/DriverBuilder.java | 111 +++++
.../rocketmq/streams/db/driver/IDriverBudiler.java | 36 ++
.../rocketmq/streams/db/driver/JDBCDriver.java | 277 ++++++++++++
.../db/driver/batchloader/BatchRowLoader.java | 179 ++++++++
.../db/driver/batchloader/IRowOperator.java | 33 ++
.../rocketmq/streams/db/driver/orm/ORMUtil.java | 490 +++++++++++++++++++++
.../rocketmq/streams/db/operator/SQLOperator.java | 178 ++++++++
.../org/apache/rocketmq/streams/db/Person.java | 110 +++++
.../DBSupportParentConfigureServiceTest.java | 74 ++++
.../streams/db/driver/orm/ORMUtilTest.java | 86 ++++
rocketmq-streams-transport-minio/pom.xml | 25 ++
.../rocketmq-streams-transport-minio.iml | 17 +
.../transport/minio/MinioFileTransport.java | 141 ++++++
.../yundun/dipper/configurable/DataTpyeTest.java | 70 +++
.../streams/configuable/model/DataTpyeTest.java | 68 +++
.../rocketmq/streams/configuable/model/Person.java | 97 ++++
.../streams/configurable/model/Person.java | 97 ++++
.../component/ConfigurableComponent.properties | 7 +
.../src/test/resources/log4j.xml | 20 +
.../src/test/resources/pro-function.txt | 11 +
.../src/test/resources/python_script.py | 22 +
25 files changed, 2518 insertions(+)
[rocketmq-streams] 29/35: Merge pull request #7 from
yuanxiaodong/main
Posted by du...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
duhengforever pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/rocketmq-streams.git
commit 22f9ab04fa9f0b684e988c7a99191d2144990e3d
Merge: 1b71925 60409d5
Author: rongtong <ji...@163.com>
AuthorDate: Mon Aug 2 19:49:18 2021 +0800
Merge pull request #7 from yuanxiaodong/main
add script module
rocketmq-streams-commons/pom.xml | 76 ++
.../streams/common/cache/CompressTable.java | 305 +++++++
.../common/cache/compress/AdditionStore.java | 161 ++++
.../streams/common/cache/compress/ByteArray.java | 103 +++
.../common/cache/compress/ByteArrayValueKV.java | 106 +++
.../streams/common/cache/compress/CacheKV.java | 371 ++++++++
.../streams/common/cache/compress/ICacheKV.java | 63 ++
.../streams/common/cache/compress/KVElement.java | 123 +++
.../cache/compress/impl/FixedLenRowCacheKV.java | 148 ++++
.../common/cache/compress/impl/IntValueKV.java | 195 +++++
.../streams/common/cache/compress/impl/KeySet.java | 109 +++
.../common/cache/compress/impl/ListValueKV.java | 66 ++
.../common/cache/compress/impl/MultiValueKV.java | 114 +++
.../common/cache/compress/impl/MutilValueKV.java | 116 +++
.../common/cache/compress/impl/SplitCache.java | 86 ++
.../common/cache/compress/impl/StringValueKV.java | 137 +++
.../streams/common/cache/softreference/ICache.java | 58 ++
.../cache/softreference/RebuildCacheElement.java | 26 +
.../softreference/impl/SoftReferenceCache.java | 127 +++
.../streams/common/calssscaner/AbstractScan.java | 257 ++++++
.../streams/common/channel/AbstractChannel.java | 220 +++++
.../rocketmq/streams/common/channel/IChannel.java | 34 +
.../AbstractSupportShuffleChannelBuilder.java | 31 +
.../common/channel/builder/IChannelBuilder.java | 70 ++
.../channel/builder/IShuffleChannelBuilder.java | 39 +
.../common/channel/impl/OutputPrintChannel.java | 47 +
.../common/channel/impl/PrintChannelBuilder.java | 47 +
.../channel/impl/file/FileChannelBuilder.java | 96 ++
.../streams/common/channel/impl/file/FileSink.java | 164 ++++
.../common/channel/impl/file/FileSource.java | 232 +++++
.../common/channel/impl/file/FileSplit.java | 64 ++
.../common/channel/impl/memory/MemoryCache.java | 83 ++
.../common/channel/impl/memory/MemoryChannel.java | 79 ++
.../channel/impl/memory/MemoryChannelBuilder.java | 71 ++
.../common/channel/impl/memory/MemorySink.java | 107 +++
.../common/channel/impl/memory/MemorySource.java | 92 ++
.../common/channel/impl/memory/MemorySplit.java | 52 ++
.../streams/common/channel/sink/AbstractSink.java | 243 +++++
.../channel/sink/AbstractSupportShuffleSink.java | 80 ++
.../streams/common/channel/sink/ISink.java | 104 +++
.../channel/sinkcache/DataSourceAutoFlushTask.java | 66 ++
.../common/channel/sinkcache/IMessageCache.java | 79 ++
.../channel/sinkcache/IMessageFlushCallBack.java | 31 +
.../impl/AbstractMutilSplitMessageCache.java | 137 +++
.../channel/sinkcache/impl/MessageCache.java | 142 +++
.../sinkcache/impl/MultiSplitMessageCache.java | 33 +
.../sinkcache/impl/MutilSplitMessageCache.java | 33 +
.../common/channel/source/AbstractBatchSource.java | 202 +++++
.../common/channel/source/AbstractSource.java | 536 ++++++++++++
.../source/AbstractSupportOffsetResetSource.java | 21 +
.../channel/source/AbstractUnreliableSource.java | 164 ++++
.../streams/common/channel/source/ISource.java | 90 ++
.../channel/source/ISplitChangedListener.java | 41 +
.../channel/source/systemmsg/NewSplitMessage.java | 26 +
.../source/systemmsg/RemoveSplitMessage.java | 26 +
.../source/systemmsg/SplitChangedMessage.java | 73 ++
.../streams/common/channel/split/ISplit.java | 49 ++
.../streams/common/checkpoint/CheckPoint.java | 78 ++
.../common/checkpoint/CheckPointManager.java | 279 ++++++
.../common/checkpoint/CheckPointMessage.java | 77 ++
.../streams/common/checkpoint/CheckPointState.java | 65 ++
.../streams/common/checkpoint/ICheckPoint.java | 31 +
.../common/classloader/ByteClassLoader.java | 30 +
.../common/classloader/FileClassLoader.java | 83 ++
.../common/classloader/IsolationClassLoader.java | 78 ++
.../common/compiler/CustomJavaCompiler.java | 248 ++++++
.../common/component/AbstractComponent.java | 208 +++++
.../streams/common/component/ComponentCreator.java | 293 +++++++
.../common/component/ConfigureDescriptor.java | 98 +++
.../streams/common/component/IComponent.java | 73 ++
.../streams/common/component/IgnoreNameSpace.java | 24 +
.../PropertyConfigureDescriptorManager.java | 104 +++
.../PropertyConfigureDiscriptorManager.java | 110 +++
.../common/configurable/AbstractConfigurable.java | 216 +++++
.../common/configurable/BasedConfigurable.java | 249 ++++++
.../IAfterConfiguableRefreshListerner.java | 25 +
.../streams/common/configurable/IConfigurable.java | 84 ++
.../configurable/IConfigurableIdentification.java | 40 +
.../configurable/IConfigurableListerner.java | 20 +
.../common/configurable/IConfigurableService.java | 136 +++
.../common/configurable/IFieldProcessor.java | 24 +
.../common/configurable/annotation/Changeable.java | 31 +
.../configurable/annotation/ENVDependence.java | 30 +
.../configurable/annotation/NoSerialized.java | 29 +
.../streams/common/configure/ConfigureFileKey.java | 142 +++
.../streams/common/context/AbstractContext.java | 364 ++++++++
.../streams/common/context/BatchMessageOffset.java | 123 +++
.../rocketmq/streams/common/context/Context.java | 33 +
.../rocketmq/streams/common/context/IMessage.java | 85 ++
.../streams/common/context/IgnoreMessage.java | 23 +
.../rocketmq/streams/common/context/Message.java | 129 +++
.../streams/common/context/MessageHeader.java | 360 ++++++++
.../streams/common/context/MessageOffset.java | 208 +++++
.../streams/common/context/UserDefinedMessage.java | 502 +++++++++++
.../streams/common/datatype/ArrayDataType.java | 146 ++++
.../streams/common/datatype/BaseDataType.java | 229 +++++
.../streams/common/datatype/BooleanDataType.java | 142 +++
.../streams/common/datatype/ByteDataType.java | 107 +++
.../common/datatype/ConfigurableDataType.java | 127 +++
.../streams/common/datatype/DataJsonable.java | 36 +
.../rocketmq/streams/common/datatype/DataType.java | 109 +++
.../streams/common/datatype/DateDataType.java | 153 ++++
.../streams/common/datatype/DoubleDataType.java | 156 ++++
.../streams/common/datatype/FloatDataType.java | 121 +++
.../common/datatype/GenericParameterDataType.java | 122 +++
.../common/datatype/GenericParamterDataType.java | 123 +++
.../streams/common/datatype/IJsonable.java | 34 +
.../streams/common/datatype/IntDataType.java | 122 +++
.../streams/common/datatype/JavaBeanDataType.java | 92 ++
.../streams/common/datatype/JsonableDataType.java | 104 +++
.../streams/common/datatype/ListDataType.java | 223 +++++
.../streams/common/datatype/LongDataType.java | 173 ++++
.../streams/common/datatype/MapDataType.java | 286 ++++++
.../common/datatype/NotSupportDataType.java | 74 ++
.../streams/common/datatype/NumberDataType.java | 84 ++
.../streams/common/datatype/SetDataType.java | 257 ++++++
.../streams/common/datatype/ShortDataType.java | 118 +++
.../streams/common/datatype/StringDataType.java | 83 ++
.../streams/common/dboperator/IDBDriver.java | 79 ++
.../common/disruptor/BufferFullFunction.java | 27 +
.../streams/common/disruptor/DisruptorEvent.java | 34 +
.../common/disruptor/DisruptorEventFactory.java | 26 +
.../common/disruptor/DisruptorProducer.java | 51 ++
.../streams/common/functions/FilterFunction.java | 24 +
.../streams/common/functions/ForEachFunction.java | 31 +
.../common/functions/ForEachMessageFunction.java | 30 +
.../streams/common/functions/Function.java | 19 +
.../streams/common/functions/MapFunction.java | 24 +
.../common/functions/MultiTableSplitFunction.java | 29 +
.../streams/common/functions/ReduceFunction.java | 24 +
.../streams/common/functions/SplitFunction.java | 30 +
.../common/functions/TableSplitFunction.java | 46 +
.../common/interfaces/IBaseStreamOperator.java | 33 +
.../streams/common/interfaces/IFilterService.java | 57 ++
.../streams/common/interfaces/IJDBCOperator.java | 77 ++
.../common/interfaces/IJsonobjectProcessor.java | 25 +
.../common/interfaces/ILineMessageProcessor.java | 32 +
.../streams/common/interfaces/IPropertyEnable.java | 39 +
.../common/interfaces/IScheduleExecutor.java | 27 +
.../streams/common/interfaces/IStreamOperator.java | 29 +
.../streams/common/interfaces/ISystemMessage.java | 20 +
.../common/interfaces/ISystemMessageProcessor.java | 23 +
.../streams/common/interfaces/ITransport.java | 32 +
.../streams/common/logger/LoggerCreator.java | 57 ++
.../streams/common/logger/LoggerOutputChannel.java | 79 ++
.../common/logger/MyDailyRollingFileAppender.java | 545 ++++++++++++
.../streams/common/metadata/AbstractMetaData.java | 247 ++++++
.../rocketmq/streams/common/metadata/MetaData.java | 227 +++++
.../streams/common/metadata/MetaDataAdapter.java | 48 +
.../streams/common/metadata/MetaDataField.java | 167 ++++
.../rocketmq/streams/common/model/BeanHolder.java | 36 +
.../rocketmq/streams/common/model/Entity.java | 49 ++
.../rocketmq/streams/common/model/NameCreator.java | 63 ++
.../rocketmq/streams/common/model/ServiceName.java | 28 +
.../streams/common/model/ThreadContext.java | 31 +
.../rocketmq/streams/common/monitor/IMonitor.java | 171 ++++
.../streams/common/monitor/MonitorFactory.java | 133 +++
.../common/monitor/TopologyFilterMonitor.java | 66 ++
.../common/monitor/group/GroupedMonitorInfo.java | 119 +++
.../common/monitor/group/MonitorCommander.java | 200 +++++
.../streams/common/monitor/impl/DipperMonitor.java | 510 +++++++++++
.../streams/common/monitor/impl/MonitorItem.java | 177 ++++
.../common/monitor/impl/NothingMonitorItem.java | 59 ++
.../common/monitor/impl/NothingMontior.java | 112 +++
.../optimization/CalculationResultCache.java | 64 ++
.../common/optimization/HyperscanRegex.java | 104 +++
.../streams/common/optimization/LikeRegex.java | 175 ++++
.../common/optimization/LogFingerprintFilter.java | 152 ++++
.../common/optimization/MessageGloableTrace.java | 80 ++
.../common/optimization/OptimizationRegex.java | 308 +++++++
.../optimization/SQLLogFingerprintFilter.java | 53 ++
.../AbstractMutilPipelineChainPipline.java | 193 ++++
.../streams/common/topology/ChainPipeline.java | 447 ++++++++++
.../streams/common/topology/ChainStage.java | 189 ++++
.../common/topology/builder/IStageBuilder.java | 39 +
.../common/topology/builder/PipelineBuilder.java | 370 ++++++++
.../common/topology/model/AbstractRule.java | 181 ++++
.../common/topology/model/AbstractScript.java | 110 +++
.../common/topology/model/AbstractStage.java | 298 +++++++
.../streams/common/topology/model/ILifeCycle.java | 33 +
.../common/topology/model/IStageHandle.java | 44 +
.../streams/common/topology/model/IWindow.java | 92 ++
.../streams/common/topology/model/Pipeline.java | 291 ++++++
.../topology/model/PipelineSourceJoiner.java | 48 +
.../streams/common/topology/model/Union.java | 63 ++
.../stages/AbstractStatelessChainStage.java | 41 +
.../topology/stages/AbstractWindowStage.java | 90 ++
.../common/topology/stages/FilterChainStage.java | 357 ++++++++
.../common/topology/stages/JoinChainStage.java | 160 ++++
.../common/topology/stages/NewSQLChainStage.java | 82 ++
.../common/topology/stages/OpenAPIChainStage.java | 80 ++
.../common/topology/stages/OutputChainStage.java | 231 +++++
.../common/topology/stages/PythonChainStage.java | 80 ++
.../topology/stages/RightJoinChainStage.java | 92 ++
.../common/topology/stages/SQLChainStage.java | 81 ++
.../common/topology/stages/ScriptChainStage.java | 124 +++
.../topology/stages/SubPiplineChainStage.java | 410 +++++++++
.../common/topology/stages/UnionChainStage.java | 51 ++
.../common/topology/stages/WindowChainStage.java | 83 ++
.../common/topology/stages/udf/FilterOperator.java | 37 +
.../topology/stages/udf/FlatMapOperator.java | 61 ++
.../common/topology/stages/udf/IReducer.java | 35 +
.../common/topology/stages/udf/IRedurce.java | 36 +
.../common/topology/stages/udf/MapOperator.java | 35 +
.../common/topology/stages/udf/StageBuilder.java | 92 ++
.../common/topology/stages/udf/UDFChainStage.java | 74 ++
.../topology/stages/udf/UDFUnionChainStage.java | 92 ++
.../common/transport/AbstractFileTransport.java | 41 +
.../streams/common/transport/IFileTransport.java | 68 ++
.../rocketmq/streams/common/utils/AESUtil.java | 108 +++
.../rocketmq/streams/common/utils/Base64Utils.java | 100 +++
.../rocketmq/streams/common/utils/ClassUtil.java | 157 ++++
.../streams/common/utils/CollectionUtil.java | 65 ++
.../rocketmq/streams/common/utils/CommandUtil.java | 184 ++++
.../streams/common/utils/CompressUtil.java | 88 ++
.../streams/common/utils/ConfigurableUtil.java | 248 ++++++
.../streams/common/utils/ContantsUtil.java | 355 ++++++++
.../rocketmq/streams/common/utils/DESUtils.java | 142 +++
.../streams/common/utils/DataTypeUtil.java | 504 +++++++++++
.../rocketmq/streams/common/utils/DateUtil.java | 603 +++++++++++++
.../common/utils/DipperThreadLocalUtil.java | 35 +
.../rocketmq/streams/common/utils/ENVUtile.java | 55 ++
.../rocketmq/streams/common/utils/FileUtil.java | 973 +++++++++++++++++++++
.../rocketmq/streams/common/utils/IOUtil.java | 99 +++
.../rocketmq/streams/common/utils/IPUtil.java | 201 +++++
.../streams/common/utils/InstantiationUtil.java | 118 +++
.../streams/common/utils/Ip2LongUtils.java | 296 +++++++
.../rocketmq/streams/common/utils/JarUtil.java | 147 ++++
.../streams/common/utils/JsonableUtil.java | 61 ++
.../streams/common/utils/LogParserUtil.java | 241 +++++
.../rocketmq/streams/common/utils/MapKeyUtil.java | 155 ++++
.../rocketmq/streams/common/utils/MessageUtil.java | 102 +++
.../streams/common/utils/NameCreatorUtil.java | 61 ++
.../rocketmq/streams/common/utils/NumberUtils.java | 179 ++++
.../rocketmq/streams/common/utils/PrintUtil.java | 205 +++++
.../streams/common/utils/PropertiesUtils.java | 144 +++
.../streams/common/utils/RandomStrUtil.java | 38 +
.../rocketmq/streams/common/utils/ReflectUtil.java | 882 +++++++++++++++++++
.../rocketmq/streams/common/utils/RegexUtil.java | 21 +
.../rocketmq/streams/common/utils/RuntimeUtil.java | 63 ++
.../streams/common/utils/SQLFormatterUtil.java | 362 ++++++++
.../rocketmq/streams/common/utils/SQLUtil.java | 413 +++++++++
.../streams/common/utils/ScheduleUtil.java | 52 ++
.../rocketmq/streams/common/utils/ShellUtil.java | 189 ++++
.../rocketmq/streams/common/utils/StringUtil.java | 369 ++++++++
.../streams/common/utils/SyslogParserUtil.java | 25 +
.../rocketmq/streams/common/utils/ThreadUtil.java | 29 +
.../rocketmq/streams/common/utils/TraceUtil.java | 118 +++
.../common/cache/compress/IntValueKVTest.java | 66 ++
.../softreference/SoftReferenceCacheTest.java | 55 ++
.../rocketmq/streams/common/channel/SinkTest.java | 73 ++
.../streams/common/channel/SourceTest.java | 75 ++
.../streams/common/classscan/ClassScanTest.java | 34 +
.../common/compiler/CustomJavaCompilerTest.java | 35 +
.../streams/common/datatype/DataTypeTest.java | 39 +
.../streams/common/stages/SelfChainStageTest.java | 36 +
.../src/test/resources/log4j.xml | 16 +
rocketmq-streams-script/pom.xml | 33 +
.../rocketmq/streams/script/ScriptComponent.java | 76 ++
.../streams/script/annotation/Function.java | 31 +
.../streams/script/annotation/FunctionMethod.java | 51 ++
.../script/annotation/FunctionMethodAilas.java | 37 +
.../script/annotation/FunctionParamter.java | 44 +
.../streams/script/annotation/UDAFFunction.java | 34 +
.../streams/script/context/FunctionContext.java | 87 ++
.../function/aggregation/AverageAccumulator.java | 98 +++
.../function/aggregation/ConcatAccumulator.java | 96 ++
.../aggregation/ConcatDistinctAccumulator.java | 98 +++
.../function/aggregation/CountAccumulator.java | 79 ++
.../function/aggregation/DistinctAccumulator.java | 75 ++
.../function/aggregation/MaxAccumulator.java | 111 +++
.../function/aggregation/MinAccumulator.java | 111 +++
.../function/aggregation/SumAccumulator.java | 82 ++
.../script/function/impl/bool/BooleanFunction.java | 39 +
.../script/function/impl/common/EchoFunction.java | 47 +
.../function/impl/condition/EqualsFunction.java | 155 ++++
.../impl/condition/ExecuteByConditionFuction.java | 36 +
.../impl/condition/GreateEqualsFunction.java | 74 ++
.../function/impl/condition/GreaterFunction.java | 69 ++
.../script/function/impl/condition/IFFunction.java | 39 +
.../impl/condition/InConditionFunction.java | 99 +++
.../impl/condition/LessEqualsFunction.java | 71 ++
.../function/impl/condition/LessFunction.java | 69 ++
.../function/impl/context/ContextFunction.java | 44 +
.../function/impl/date/AddMonthFunction.java | 73 ++
.../impl/date/CurrentTimestampFunction.java | 63 ++
.../script/function/impl/date/DateAddFunction.java | 146 ++++
.../function/impl/date/DateDiffFunction.java | 124 +++
.../function/impl/date/DateFormatFunction.java | 92 ++
.../function/impl/date/DatePartFunction.java | 218 +++++
.../function/impl/date/DateToCharFunction.java | 93 ++
.../function/impl/date/DateTruncFunction.java | 143 +++
.../function/impl/date/DateUtilFunction.java | 130 +++
.../function/impl/date/DateUtileFunction.java | 134 +++
.../script/function/impl/date/DayFunction.java | 62 ++
.../function/impl/date/FromUnixTimeFunction.java | 72 ++
.../script/function/impl/date/GetDateFunction.java | 43 +
.../script/function/impl/date/IsDateFunction.java | 54 ++
.../script/function/impl/date/LastDayFunction.java | 69 ++
.../script/function/impl/date/NextDayFunction.java | 100 +++
.../script/function/impl/date/NowFuntion.java | 46 +
.../script/function/impl/date/QuarterFuction.java | 75 ++
.../script/function/impl/date/QuarterFunction.java | 76 ++
.../function/impl/date/UnixTimeStampFunction.java | 77 ++
.../function/impl/date/WeekOfYearFunction.java | 59 ++
.../script/function/impl/date/WeekdayFunction.java | 60 ++
.../script/function/impl/env/ENVFunction.java | 47 +
.../impl/field/AdditionalFiledFunction.java | 318 +++++++
.../function/impl/field/CoalesceFunction.java | 51 ++
.../function/impl/field/ExtraFieldFunction.java | 67 ++
.../script/function/impl/field/FieldFunction.java | 155 ++++
.../function/impl/field/RemoveFieldFunction.java | 59 ++
.../function/impl/field/RetainFieldFunction.java | 51 ++
.../script/function/impl/filter/BreakFunction.java | 41 +
.../function/impl/filter/FilterFunction.java | 74 ++
.../function/impl/flatmap/SplitArrayFunction.java | 164 ++++
.../function/impl/header/HeaderFunction.java | 48 +
.../function/impl/json/JsonArrayFunction.java | 230 +++++
.../function/impl/json/JsonCreatorFunction.java | 218 +++++
.../function/impl/json/JsonValueFunction.java | 108 +++
.../script/function/impl/math/AbsFunction.java | 118 +++
.../script/function/impl/math/AcosFunction.java | 109 +++
.../function/impl/math/AdditionFunction.java | 62 ++
.../script/function/impl/math/AsinFunction.java | 109 +++
.../script/function/impl/math/AtanFunction.java | 88 ++
.../script/function/impl/math/BinFunction.java | 54 ++
.../script/function/impl/math/BitFunction.java | 201 +++++
.../script/function/impl/math/BroundFunction.java | 113 +++
.../function/impl/math/CardinalityFunction.java | 49 ++
.../script/function/impl/math/CbrtFunction.java | 67 ++
.../script/function/impl/math/CeilFunction.java | 109 +++
.../script/function/impl/math/ConvFunction.java | 244 ++++++
.../script/function/impl/math/CosFunction.java | 109 +++
.../script/function/impl/math/CosHFunction.java | 109 +++
.../script/function/impl/math/CotFunction.java | 109 +++
.../script/function/impl/math/DegreesFunction.java | 90 ++
.../function/impl/math/DivisionFunction.java | 62 ++
.../script/function/impl/math/EFunction.java | 42 +
.../script/function/impl/math/ExpFunction.java | 109 +++
.../function/impl/math/FactorialFunction.java | 79 ++
.../script/function/impl/math/FloorFunction.java | 115 +++
.../script/function/impl/math/HexFunction.java | 69 ++
.../script/function/impl/math/LnFunction.java | 109 +++
.../script/function/impl/math/Log10Function.java | 105 +++
.../script/function/impl/math/Log2Function.java | 105 +++
.../script/function/impl/math/LogFunction.java | 114 +++
.../script/function/impl/math/MathFunction.java | 132 +++
.../function/impl/math/MultiplicationFunction.java | 63 ++
.../function/impl/math/OperationFunction.java | 123 +++
.../script/function/impl/math/PIFunction.java | 42 +
.../script/function/impl/math/PowerFunction.java | 114 +++
.../script/function/impl/math/RadiansFunction.java | 48 +
.../script/function/impl/math/RandFunction.java | 62 ++
.../script/function/impl/math/RoundFunction.java | 117 +++
.../function/impl/math/ShiftLeftFunction.java | 52 ++
.../function/impl/math/ShiftRightFunction.java | 52 ++
.../impl/math/ShiftRightUnSignedFunction.java | 52 ++
.../script/function/impl/math/SignFunction.java | 50 ++
.../script/function/impl/math/SinFunction.java | 109 +++
.../script/function/impl/math/SinHFunction.java | 109 +++
.../script/function/impl/math/SqrtFunction.java | 109 +++
.../function/impl/math/SubtractionFunction.java | 62 ++
.../script/function/impl/math/TanFunction.java | 108 +++
.../script/function/impl/math/TanHFunction.java | 109 +++
.../script/function/impl/math/TruncFunction.java | 118 +++
.../script/function/impl/math/UnHexFunction.java | 64 ++
.../script/function/impl/mock/MockFunction.java | 39 +
.../function/impl/offset/OffsetFunction.java | 67 ++
.../script/function/impl/parser/GrokFunction.java | 91 ++
.../function/impl/parser/LogParserFunction.java | 85 ++
.../function/impl/parser/Paser2JsonFunction.java | 169 ++++
.../function/impl/parser/PaserBySplitFunction.java | 172 ++++
.../function/impl/parser/RegexParserFunction.java | 85 ++
.../function/impl/platform/BlinkUDFFunction.java | 33 +
.../function/impl/python/GroovyFunction.java | 52 ++
.../function/impl/python/PythonFunction.java | 49 ++
.../script/function/impl/router/RouteFunction.java | 37 +
.../script/function/impl/sql/SQLFunction.java | 56 ++
.../script/function/impl/string/AsciiFunction.java | 52 ++
.../function/impl/string/Base64Function.java | 86 ++
.../function/impl/string/BlinkStringFunction.java | 53 ++
.../impl/string/CharMatchCountFunction.java | 61 ++
.../script/function/impl/string/ChrFunction.java | 49 ++
.../function/impl/string/HashCodeFunction.java | 50 ++
.../function/impl/string/InitCapFunction.java | 53 ++
.../script/function/impl/string/InstrFunction.java | 136 +++
.../function/impl/string/InstrtFunction.java | 134 +++
.../function/impl/string/Ip2RegionFunction.java | 43 +
.../function/impl/string/IsEncodingFunction.java | 62 ++
.../function/impl/string/KeyValueFunction.java | 120 +++
.../script/function/impl/string/LTrimFunction.java | 49 ++
.../function/impl/string/LengthBFunction.java | 50 ++
.../function/impl/string/LengthFunction.java | 50 ++
.../script/function/impl/string/Md5Function.java | 84 ++
.../script/function/impl/string/NullFunction.java | 43 +
.../function/impl/string/ParseUrlFunction.java | 102 +++
.../script/function/impl/string/RTrimFunction.java | 50 ++
.../function/impl/string/RegexCountFunction.java | 113 +++
.../script/function/impl/string/RegexFunction.java | 159 ++++
.../function/impl/string/RegexInstrFunction.java | 180 ++++
.../function/impl/string/RegexReplaceFunction.java | 92 ++
.../function/impl/string/RegexSubStrFunction.java | 127 +++
.../function/impl/string/RepeatFunction.java | 60 ++
.../function/impl/string/ReplaceFunction.java | 56 ++
.../function/impl/string/ReverseFunction.java | 50 ++
.../function/impl/string/SoundxFunction.java | 112 +++
.../script/function/impl/string/SpaceFunction.java | 60 ++
.../function/impl/string/SplitPartFunction.java | 133 +++
.../impl/string/SubStringIndexFunction.java | 128 +++
.../function/impl/string/ToCharFunction.java | 119 +++
.../function/impl/string/ToLowerFunction.java | 49 ++
.../function/impl/string/ToUpperFunction.java | 49 ++
.../function/impl/string/TransLateFunction.java | 57 ++
.../script/function/impl/string/TrimFunction.java | 50 ++
.../script/function/impl/string/UUIDFunction.java | 39 +
.../function/impl/string/UrlDecodeFunction.java | 85 ++
.../function/impl/string/UrlEncodeFunction.java | 125 +++
.../script/function/impl/type/CastFunction.java | 73 ++
.../script/function/impl/udtf/UDTFFunction.java | 61 ++
.../function/impl/window/WindowFunction.java | 64 ++
.../script/function/model/FunctionConfigure.java | 573 ++++++++++++
.../function/model/FunctionConfigureMap.java | 109 +++
.../script/function/model/FunctionInfo.java | 101 +++
.../script/function/model/FunctionInfoMap.java | 84 ++
.../script/function/model/FunctionType.java | 45 +
.../function/service/IDipperInterfaceAdpater.java | 46 +
.../script/function/service/IFunctionService.java | 130 +++
.../service/impl/DefaultFunctionServiceImpl.java | 396 +++++++++
.../function/service/impl/ScanFunctionService.java | 96 ++
.../script/operator/AbstractScriptOperator.java | 82 ++
.../operator/expression/GroupScriptExpression.java | 239 +++++
.../operator/expression/ScriptExpression.java | 327 +++++++
.../operator/expression/ScriptParameter.java | 135 +++
.../script/operator/impl/AggregationScript.java | 211 +++++
.../operator/impl/FileGroovyScriptOperator.java | 33 +
.../operator/impl/FileJPythonScriptOperatore.java | 32 +
.../script/operator/impl/FunctionScript.java | 247 ++++++
.../script/operator/impl/GroovyScriptOperator.java | 108 +++
.../operator/impl/JPythonScriptOperator.java | 95 ++
.../script/operator/impl/JavaScriptOperator.java | 63 ++
.../script/operator/impl/ScriptOperator.java | 29 +
.../optimization/AbstractFunctionOptimization.java | 47 +
.../script/optimization/CompileParameter.java | 132 +++
.../optimization/CompileScriptExpression.java | 121 +++
.../script/optimization/EqualsOptimization.java | 44 +
.../script/optimization/IFunctionOptimization.java | 38 +
.../optimization/OptimizationScriptExpression.java | 67 ++
.../script/optimization/RegexOptimization.java | 43 +
.../script/optimization/ScriptOptimization.java | 407 +++++++++
.../script/parser/IScriptExpressionParser.java | 41 +
.../parser/ScriptExpressionParserFactory.java | 70 ++
.../parser/imp/ConditionExpressionParser.java | 227 +++++
.../parser/imp/FunctionExpressionParser.java | 361 ++++++++
.../streams/script/parser/imp/FunctionParser.java | 166 ++++
.../streams/script/service/IAccumulator.java | 67 ++
.../script/service/IFunctionFieldDependent.java | 40 +
.../streams/script/service/IScriptExpression.java | 59 ++
.../streams/script/service/IScriptParamter.java | 43 +
.../streams/script/service/IScriptService.java | 84 ++
.../streams/script/service/IScriptUDFInit.java | 37 +
.../script/service/impl/ScriptServiceImpl.java | 121 +++
.../streams/script/service/udf/UDAFScript.java | 230 +++++
.../streams/script/service/udf/UDFScript.java | 296 +++++++
.../streams/script/service/udf/UDTFScript.java | 113 +++
.../streams/script/utils/FunctionUtils.java | 254 ++++++
.../rocketmq/streams/script/utils/MatchUtil.java | 82 ++
.../streams/script/utils/RandomStrUtil.java | 38 +
.../streams/script/utils/ScriptParserUtil.java | 23 +
.../streams/script/utils/StringMatchUtil.java | 87 ++
.../rocketmq/streams/script/utils/UniqID.java | 66 ++
.../src/main/resources/log4j.xml | 20 +
.../streams/script/function/FunctionTest.java | 89 ++
.../streams/script/function/GrokFunctionTest.java | 44 +
.../script/function/GroovyFunctionTest.java | 52 ++
.../script/function/JPythonFunctionTest.java | 48 +
.../function/aggregation/AccumulatorTest.java | 235 +++++
.../script/operator/ScriptOperatorTest.java | 105 +++
.../streams/script/parser/ScriptParserTest.java | 43 +
.../src/test/resources/log4j.xml | 20 +
.../src/test/resources/python_script.py | 22 +
480 files changed, 54314 insertions(+)
[rocketmq-streams] 19/35: Merge branch 'apache:main' into main
Posted by du...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
duhengforever pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/rocketmq-streams.git
commit c41c8250b1663f0b9ef9e63305017f03375a0716
Merge: b3c4242 6d89c9a
Author: Arthur Liang <wr...@vip.qq.com>
AuthorDate: Mon Aug 2 13:49:21 2021 +0800
Merge branch 'apache:main' into main
.gitignore | 25 ++
README.md | 108 ++++-
pom.xml | 410 +++++++++++++++++++
rocketmq-streams-clients/pom.xml | 46 +++
.../rocketmq/streams/client/DataStreamAction.java | 101 +++++
.../rocketmq/streams/client/StreamBuilder.java | 28 ++
.../streams/client/source/DataStreamSource.java | 76 ++++
.../client/strategy/CheckpointStrategy.java | 69 ++++
.../streams/client/strategy/StateStrategy.java | 37 ++
.../rocketmq/streams/client/strategy/Strategy.java | 25 ++
.../streams/client/transform/DataStream.java | 437 +++++++++++++++++++++
.../streams/client/transform/JoinStream.java | 212 ++++++++++
.../streams/client/transform/SplitStream.java | 61 +++
.../streams/client/transform/WindowStream.java | 210 ++++++++++
.../client/transform/window/HoppingWindow.java | 32 ++
.../client/transform/window/SessionWindow.java | 32 ++
.../streams/client/transform/window/Time.java | 45 +++
.../client/transform/window/TumblingWindow.java | 33 ++
.../client/transform/window/WindowInfo.java | 83 ++++
.../rocketmq/streams/client/DBDriverTest.java | 75 ++++
.../rocketmq/streams/client/DataStreamTest.java | 107 +++++
.../apache/rocketmq/streams/client/FilterTest.java | 49 +++
.../apache/rocketmq/streams/client/JoinTest.java | 89 +++++
.../apache/rocketmq/streams/client/LeaseTest.java | 98 +++++
.../rocketmq/streams/client/ORMUtilTest.java | 172 ++++++++
.../apache/rocketmq/streams/client/SplitTest.java | 86 ++++
.../apache/rocketmq/streams/client/UnionTest.java | 82 ++++
.../apache/rocketmq/streams/client/WindowTest.java | 86 ++++
.../client/windows/AbstractWindowFireModeTest.java | 189 +++++++++
.../streams/client/windows/WindowFromFileTest.java | 158 ++++++++
.../streams/client/windows/WindowFromMetaq.java | 47 +++
.../client/windows/WindowHighAvailabilityTest.java | 131 ++++++
.../src/test/resources/log4j.xml | 36 ++
rocketmq-streams-dim/pom.xml | 47 +++
.../apache/rocketmq/streams/dim/DimComponent.java | 63 +++
.../rocketmq/streams/dim/builder/DimBuilder.java | 94 +++++
.../function/expression/InExpressionResource.java | 80 ++++
.../expression/NotInExpressionResource.java | 45 +++
.../dim/function/script/IntelligenceFunction.java | 81 ++++
.../script/IntelligenceNameListFunction.java | 24 ++
.../dim/function/script/NameListFunction.java | 203 ++++++++++
.../rocketmq/streams/dim/index/DimIndex.java | 319 +++++++++++++++
.../rocketmq/streams/dim/index/IndexExecutor.java | 258 ++++++++++++
.../intelligence/AbstractIntelligenceCache.java | 395 +++++++++++++++++++
.../dim/intelligence/AccountIntelligenceCache.java | 77 ++++
.../dim/intelligence/DomainIntelligenceCache.java | 83 ++++
.../dim/intelligence/IPIntelligenceCache.java | 108 +++++
.../dim/intelligence/URLIntelligenceCache.java | 80 ++++
.../rocketmq/streams/dim/model/AbstractDim.java | 312 +++++++++++++++
.../streams/dim/model/BooleanFieldDBDim.java | 55 +++
.../apache/rocketmq/streams/dim/model/DBDim.java | 140 +++++++
.../rocketmq/streams/dim/service/IDimService.java | 65 +++
.../streams/dim/service/impl/DimServiceImpl.java | 92 +++++
.../com/aliyun/service/ConfigureLoaderTest.java | 37 ++
.../com/aliyun/service/ExpressionExecutorTest.java | 80 ++++
.../java/com/aliyun/service/JsonParserTest.java | 40 ++
.../com/aliyun/service/NameListFunctionTest.java | 90 +++++
.../java/com/aliyun/service/TableCompressTest.java | 26 ++
rocketmq-streams-lease/pom.xml | 25 ++
.../rocketmq/streams/lease/LeaseComponent.java | 103 +++++
.../rocketmq/streams/lease/model/LeaseInfo.java | 127 ++++++
.../streams/lease/service/ILeaseGetCallback.java | 30 ++
.../streams/lease/service/ILeaseService.java | 136 +++++++
.../streams/lease/service/ILeaseStorage.java | 73 ++++
.../streams/lease/service/ILeaseStorasge.java | 63 +++
.../lease/service/impl/BasedLesaseImpl.java | 404 +++++++++++++++++++
.../lease/service/impl/LeaseServiceImpl.java | 275 +++++++++++++
.../streams/lease/service/impl/MockLeaseImpl.java | 95 +++++
.../lease/service/storages/DBLeaseStorage.java | 229 +++++++++++
.../rocketmq/streams/lease/LeaseComponentTest.java | 119 ++++++
.../src/test/resources/log4j.xml | 20 +
71 files changed, 8067 insertions(+), 1 deletion(-)
[rocketmq-streams] 17/35: modify the README.md
Posted by du...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
duhengforever pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/rocketmq-streams.git
commit 5555b67eba44b45356a53082ec8b7b496b5fbf9e
Author: 刈刀 <je...@gmail.com>
AuthorDate: Mon Aug 2 12:48:41 2021 +0800
modify the README.md
---
README.md | 30 ++++++++++++++++--------------
1 file changed, 16 insertions(+), 14 deletions(-)
diff --git a/README.md b/README.md
index 649319c..92a9b2f 100644
--- a/README.md
+++ b/README.md
@@ -44,6 +44,7 @@ StreamBuilder 用于构建流任务的源； 内部包含```dataStream()```和``
## DataStream API
### Source
+
DataStreamSource 是分段式编程的源头类，用于对接各种数据源， 从各大消息队列中获取数据；
+ ```fromFile``` 从文件中读取数据， 该方法包含俩个参数
@@ -56,13 +57,15 @@ DataStreamSource 是分段式编程的源头类，用于对接各种数据源，
+ ```groupName``` 消费者组的名称，必填参数
+ ```isJson``` 是否json格式，非必填参数
+ ```tags``` rocketmq消费的tags值，用于过滤消息，非必填参数
-
+
+ ```from``` 自定义的数据源， 通过实现ISource接口实现自己的数据源
### transform
+
transform 允许在流计算过程中对输入源的数据进行修改，进行下一步的操作；DataStream API中包括```DataStream```,```JoinStream```, ```SplitStream```,```WindowStream```等多个transform类；
#### DataStream
+
DataStream实现了一系列常见的流计算算子
+ ```map``` 通过将源的每个记录传递给函数func来返回一个新的DataStream
@@ -76,30 +79,29 @@ DataStream实现了一系列常见的流计算算子
+ ```toFile``` 将结果保存为文件，生成一个新的DataStreamAction实例
+ ```toDB``` 将结果保存到数据库
+ ```toRocketmq``` 将结果输出到rocketmq
-+ ```toSls``` 将结果输出到sls
+ ```to``` 将结果经过自定义的ISink接口输出到指定的存储
+ ```window``` 在窗口内进行相关的统计分析，一般会与```groupBy```连用， ```window()```用来定义窗口的大小， ```groupBy()```用来定义统计分析的主key，可以指定多个
- + ```count``` 在窗口内计数
- + ```min``` 获取窗口内统计值的最小值
- + ```max``` 获取窗口内统计值得最大值
- + ```avg``` 获取窗口内统计值的平均值
- + ```sum``` 获取窗口内统计值的加和值
- + ```reduce``` 在窗口内进行自定义的汇总运算
+ + ```count``` 在窗口内计数
+ + ```min``` 获取窗口内统计值的最小值
+ + ```max``` 获取窗口内统计值得最大值
+ + ```avg``` 获取窗口内统计值的平均值
+ + ```sum``` 获取窗口内统计值的加和值
+ + ```reduce``` 在窗口内进行自定义的汇总运算
+ ```join``` 根据条件将将俩个流进行关联， 合并为一个大流进行相关的运算
+ ```union``` 将俩个流进行合并
+ ```split``` 将一个数据流按照标签进行拆分，分为不同的数据流供下游进行分析计算
+ ```with``` with算子用来指定计算过程中的相关策略，包括checkpoint的存储策略，state的存储策略等
+# Strategy
-# Strategy
策略机制主要用来控制计算引擎运行过程中的底层逻辑，如checkpoint，state的存储方式等，后续还会增加对窗口、双流join等的控制；所有的控制策略通过```with```算子传入，可以同时传入多个策略类型；
```java
//指定checkpoint的存储策略
source
-.fromRocketmq("TSG_META_INFO", "")
-.map(message -> message + "--")
-.toPrint(1)
-.with(CheckpointStrategy.db("jdbc:mysql://XXXXX:3306/XXXXX", "", "", 0L))
-.start();
+ .fromRocketmq("TSG_META_INFO","")
+ .map(message->message+"--")
+ .toPrint(1)
+ .with(CheckpointStrategy.db("jdbc:mysql://XXXXX:3306/XXXXX","","",0L))
+ .start();
```
\ No newline at end of file
[rocketmq-streams] 08/35: add channel-configurable
Posted by du...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
duhengforever pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/rocketmq-streams.git
commit 1992a3514e723f452b59878be8898d855eab2c7f
Author: vv <ze...@alibaba-inc.com>
AuthorDate: Mon Aug 2 12:04:58 2021 +0800
add channel-configurable
---
rocketmq-streams-configurable/pom.xml | 21 +
.../streams/configuable/ConfigurableComponent.java | 189 +++++++
.../streams/configuable/model/Configure.java | 130 +++++
.../service/AbstractConfigurableService.java | 561 +++++++++++++++++++++
.../AbstractSupportParentConfigureService.java | 263 ++++++++++
.../service/ConfigurableServcieType.java | 31 ++
.../service/ConfigurableServiceFactory.java | 58 +++
.../service/impl/FileConfigureService.java | 250 +++++++++
.../impl/FileSupportParentConfigureService.java | 38 ++
.../service/impl/MemoryConfigureService.java | 122 +++++
.../impl/MemorySupportParentConfigureService.java | 39 ++
.../configurable/ConfigurableComponent.java | 188 +++++++
.../streams/configurable/model/Configure.java | 127 +++++
.../service/AbstractConfigurableService.java | 553 ++++++++++++++++++++
.../AbstractSupportParentConfigureService.java | 255 ++++++++++
.../service/ConfigurableServcieType.java | 29 ++
.../service/ConfigurableServiceFactory.java | 59 +++
.../service/impl/FileConfigureService.java | 249 +++++++++
.../impl/FileSupportParentConfigureService.java | 37 ++
.../service/impl/MemoryConfigureService.java | 121 +++++
.../impl/MemorySupportParentConfigureService.java | 37 ++
.../src/main/resources/log4j.xml | 20 +
.../configuable/ConfiguableComponentTest.java | 112 ++++
.../rocketmq/streams/configuable/model/Person.java | 110 ++++
.../configurable/ConfigurableComponentTest.java | 108 ++++
.../streams/configurable/model/Person.java | 110 ++++
.../src/test/resources/log4j.xml | 20 +
27 files changed, 3837 insertions(+)
diff --git a/rocketmq-streams-configurable/pom.xml b/rocketmq-streams-configurable/pom.xml
new file mode 100755
index 0000000..3f837d8
--- /dev/null
+++ b/rocketmq-streams-configurable/pom.xml
@@ -0,0 +1,21 @@
+<?xml version="1.0" encoding="utf-8"?>
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams</artifactId>
+ <version>2.0.0-SNAPSHOT</version>
+ </parent>
+ <artifactId>rocketmq-streams-configurable</artifactId>
+ <name>ROCKETMQ STREAMS :: configurable</name>
+ <packaging>jar</packaging>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams-serviceloader</artifactId>
+ </dependency>
+ </dependencies>
+</project>
diff --git a/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configuable/ConfigurableComponent.java b/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configuable/ConfigurableComponent.java
new file mode 100644
index 0000000..fbb35ca
--- /dev/null
+++ b/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configuable/ConfigurableComponent.java
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.streams.configuable;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.rocketmq.streams.common.component.AbstractComponent;
+import org.apache.rocketmq.streams.common.component.ComponentCreator;
+import org.apache.rocketmq.streams.common.component.ConfigureDescriptor;
+import org.apache.rocketmq.streams.common.configurable.IConfigurable;
+import org.apache.rocketmq.streams.common.configurable.IConfigurableService;
+import org.apache.rocketmq.streams.common.utils.ConfigurableUtil;
+import org.apache.rocketmq.streams.configuable.service.AbstractConfigurableService;
+import org.apache.rocketmq.streams.configuable.service.ConfigurableServcieType;
+import org.apache.rocketmq.streams.configuable.service.ConfigurableServiceFactory;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+/**
+ * 对Configurable对象，做统一的管理，统一查询，插入和更新。 insert/update 把configuabel对象写入存储，支持文件存储（file），内存存储（memory）和db存储（DB）。可以在配置通过这个ConfigureFileKey.CONNECT_TYPE key 配置 query 是基于内存的查询，对象定时load到内存，可以在属性文件通过这个ConfigureFileKey.POLLING_TIME key配置加载周期，单位是秒 新对象加载后生效，已经存在的对象只有updateFlag发生变化才会被替换
+ */
+public class ConfigurableComponent extends AbstractComponent<IConfigurableService>
+ implements IConfigurableService {
+
+ private static final Log LOG = LogFactory.getLog(ConfigurableComponent.class);
+
+ protected volatile IConfigurableService configureService = null;
+
+ protected transient String namespace;
+
+ public ConfigurableComponent() {
+ initConfigurableServiceDescriptor();
+ addConfigureDescriptor(
+ new ConfigureDescriptor(CONNECT_TYPE, false, ConfigurableServcieType.DEFAULT_SERVICE_NAME));
+ }
+
+ public static ConfigurableComponent getInstance(String namespace) {
+ return ComponentCreator.getComponent(namespace, ConfigurableComponent.class);
+ }
+
+ @Override
+ protected boolean initProperties(Properties properties) {
+ try {
+ if (configureService != null) {
+ return true;
+ }
+ this.configureService = ConfigurableServiceFactory.createConfigurableService(properties);
+ return true;
+ } catch (Exception e) {
+ LOG.error("ConfigurableComponent create error,properties= " + properties, e);
+ return false;
+ }
+
+ }
+
+ @Override
+ public boolean startComponent(String namespace) {
+ try {
+ this.namespace = namespace;
+ configureService.initConfigurables(namespace);
+ return true;
+ } catch (Exception e) {
+ LOG.error("ConfigurableComponent init error, namespace is " + namespace, e);
+ return false;
+ }
+
+ }
+
+ /**
+ * 启动测试模式，用内存数据库存储和加载configurable数据
+ */
+ public static void begineTestMode() {
+ System.setProperty(ConfigurableComponent.CONNECT_TYPE, ConfigurableServcieType.MEMORY_SERVICE_NAME);
+ }
+
+ /**
+ * 关闭测试模式，用配置文件中配置的属性加载configuable数据
+ */
+ public static void endTestMode() {
+ System.clearProperty(ConfigurableComponent.CONNECT_TYPE);
+ }
+
+ @Override
+ public boolean stop() {
+ return true;
+ }
+
+ @Override
+ public IConfigurableService getService() {
+ return configureService;
+ }
+
+ @Override
+ public void initConfigurables(String namespace) {
+ configureService.initConfigurables(namespace);
+ }
+
+ @Override
+ public boolean refreshConfigurable(String namespace) {
+ return configureService.refreshConfigurable(namespace);
+ }
+
+ public void mockConfigurable(String namespace) {
+ refreshConfigurable(namespace);
+
+ }
+
+ @Override
+ public List<IConfigurable> queryConfigurable(String type) {
+ return configureService.queryConfigurable(type);
+ }
+
+ @Override
+ public <T extends IConfigurable> List<T> queryConfigurableByType(String type) {
+ return configureService.queryConfigurableByType(type);
+ }
+
+ @Override
+ public IConfigurable queryConfigurableByIdent(String type, String name) {
+ return configureService.queryConfigurableByIdent(type, name);
+ }
+
+ @Override
+ public IConfigurable queryConfigurableByIdent(String identification) {
+ return configureService.queryConfigurableByIdent(identification);
+ }
+
+ @Override
+ public void insert(IConfigurable configurable) {
+ configureService.insert(configurable);
+ ConfigurableUtil.refreshMock(configurable);
+ }
+
+ @Override
+ public void update(IConfigurable configurable) {
+ configureService.update(configurable);
+ }
+
+ @Override
+ public <T> Map<String, T> queryConfigurableMapByType(String type) {
+ return configureService.queryConfigurableMapByType(type);
+ }
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public <T> T queryConfigurable(String configurableType, String name) {
+ return (T)queryConfigurableByIdent(configurableType, name);
+ }
+
+ //protected void insertConfigurable(JSONObject message, IConfigurable configurable) {
+ // ConfigurableUtil.insertConfigurable(message, configurable, this.configureService);
+ //}
+
+ @Override
+ public String getNamespace() {
+ if (AbstractConfigurableService.class.isInstance(configureService)) {
+ return ((AbstractConfigurableService)configureService).getNamespace();
+ }
+ return namespace;
+ }
+
+ @Override
+ public Collection<IConfigurable> findAll() {
+ return configureService.findAll();
+ }
+
+ @Override
+ public <T extends IConfigurable> List<T> loadConfigurableFromStorage(String type) {
+ return configureService.loadConfigurableFromStorage(type);
+ }
+}
diff --git a/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configuable/model/Configure.java b/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configuable/model/Configure.java
new file mode 100644
index 0000000..121233d
--- /dev/null
+++ b/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configuable/model/Configure.java
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.streams.configuable.model;
+
+import org.apache.rocketmq.streams.common.model.Entity;
+
+/**
+ *
+ * configuable如果存储在db，这个是db表的映射对象
+ */
+public class Configure extends Entity {
+
+ private static final long serialVersionUID = 5668017348345235669L;
+
+ private String nameSpace;
+ private String type;
+ private String name;
+ // private String identification;
+ private String jsonValue;
+ private String modifyTime;
+ private String remark;
+ private int openRange;
+
+
+ public static String createTableSQL(String tableName){
+ return "/******************************************/\n"
+ + "/* TableName = dipper_configure */\n"
+ + "/******************************************/\n"
+ + "CREATE TABLE IF NOT EXISTS `"+tableName+"` (\n"
+ + " `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT COMMENT '主键',\n"
+ + " `gmt_create` datetime NOT NULL COMMENT '创建时间',\n"
+ + " `gmt_modified` datetime NOT NULL COMMENT '修改时间',\n"
+ + " `namespace` varchar(32) NOT NULL COMMENT '项目标识',\n"
+ + " `type` varchar(32) NOT NULL COMMENT '配置类型',\n"
+ + " `name` varchar(128) NOT NULL COMMENT '配置名称',\n"
+ + " `json_value` text NOT NULL COMMENT '配置内容',\n"
+ + " `status` tinyint(3) unsigned NOT NULL DEFAULT '0' COMMENT '1:正在使用 0:已失效',\n"
+ + " PRIMARY KEY (`id`),\n"
+ + " UNIQUE KEY `uk_namespace_type_name` (`namespace`,`type`,`name`),\n"
+ + " KEY `idx_namespace` (`namespace`)\n"
+ + ") ENGINE=InnoDB AUTO_INCREMENT=1814834 DEFAULT CHARSET=utf8 COMMENT='统一接入配置项'\n"
+ + ";";
+ }
+
+ public String getNameSpace() {
+ return nameSpace;
+ }
+
+ public void setNameSpace(String nameSpace) {
+ this.nameSpace = nameSpace;
+ }
+
+ public String getType() {
+ return type;
+ }
+
+ public void setType(String type) {
+ this.type = type;
+ }
+
+ public String getName() {
+ return name;
+ }
+
+ public void setName(String name) {
+ this.name = name;
+ }
+
+ // public String getIdentification() {
+ // return identification;
+ // }
+
+ // public void createIdentification() {
+ // this.identification = MapKeyUtil.createKey(nameSpace, type, name);
+ // }
+
+ public String getJsonValue() {
+ return jsonValue;
+ }
+
+ public void setJsonValue(String jsonValue) {
+ this.jsonValue = jsonValue;
+ }
+
+ public String getModifyTime() {
+ return modifyTime;
+ }
+
+ public void setModifyTime(String modifyTime) {
+ this.modifyTime = modifyTime;
+ }
+
+ public String getRemark() {
+ return remark;
+ }
+
+ public void setRemark(String remark) {
+ this.remark = remark;
+ }
+
+ public int getOpenRange() {
+ return openRange;
+ }
+
+ public void setOpenRange(int openRange) {
+ this.openRange = openRange;
+ }
+
+ @Override
+ public String toString() {
+ return "Configure{" + "nameSpace='" + nameSpace + '\'' + ", type='" + type + '\'' + ", name='" + name + '\''
+ + ", jsonValue='" + jsonValue + '\'' + ", modifyTime='" + modifyTime + '\'' + ", remark='" + remark + '\''
+ + ", openRange=" + openRange + '}';
+ }
+}
diff --git a/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configuable/service/AbstractConfigurableService.java b/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configuable/service/AbstractConfigurableService.java
new file mode 100644
index 0000000..9f55da8
--- /dev/null
+++ b/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configuable/service/AbstractConfigurableService.java
@@ -0,0 +1,561 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.streams.configuable.service;
+
+import com.alibaba.fastjson.JSONObject;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.rocketmq.streams.common.component.AbstractComponent;
+import org.apache.rocketmq.streams.common.configurable.AbstractConfigurable;
+import org.apache.rocketmq.streams.common.configurable.IAfterConfiguableRefreshListerner;
+import org.apache.rocketmq.streams.common.configurable.IConfigurable;
+import org.apache.rocketmq.streams.common.configurable.IConfigurableService;
+import org.apache.rocketmq.streams.common.datatype.DataType;
+import org.apache.rocketmq.streams.common.model.Entity;
+import org.apache.rocketmq.streams.common.utils.ConfigurableUtil;
+import org.apache.rocketmq.streams.common.utils.DataTypeUtil;
+import org.apache.rocketmq.streams.common.utils.MapKeyUtil;
+import org.apache.rocketmq.streams.common.utils.ReflectUtil;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+import org.apache.rocketmq.streams.configuable.model.Configure;
+
+public abstract class AbstractConfigurableService implements IConfigurableService {
+
+ private static final Log LOG = LogFactory.getLog(AbstractConfigurableService.class);
+
+ private static final String CLASS_NAME = IConfigurableService.CLASS_NAME;
+
+ protected Map<String, List<IConfigurable>> type2ConfigurableMap = new HashMap<>();
+
+ protected Map<String, IConfigurable> name2ConfigurableMap = new HashMap<>();
+
+ protected Map<String, IConfigurable> configurableMap = new HashMap<>();
+
+ protected Properties properties;
+
+ protected transient String namespace;
+
+ public AbstractConfigurableService(Properties properties) {
+ this.properties = properties;
+ }
+
+ public AbstractConfigurableService() {
+ }
+
+ @Override
+ public IConfigurable queryConfigurableByIdent(String identification) {
+ return name2ConfigurableMap.get(identification);
+ }
+
+ protected String getConfigureKey(String nameSpace, String type, String name) {
+ return MapKeyUtil.createKey(nameSpace, type, name);
+ }
+
+ protected void updateConfiguresCache(IConfigurable configurable) {
+ if (configurable == null) {
+ return;
+ }
+ configurable.toJson();
+ String key = getConfigureKey(configurable.getNameSpace(), configurable.getType(), configurable.getConfigureName());
+ configurableMap.put(key, configurable);
+ }
+
+ protected void updateConfiguresCache(List<IConfigurable> configureList) {
+ for (IConfigurable iConfigurable : configureList) {
+ updateConfiguresCache(iConfigurable);
+ }
+ }
+
+ protected boolean equals(String key, List<?> newConfigureList) {
+ for (Object o : newConfigureList) {
+ IConfigurable configure = (IConfigurable)o;
+ String tempKey = getConfigureKey(configure.getNameSpace(), configure.getType(), configure.getConfigureName());
+ if (key.equals(tempKey)) {
+ IConfigurable oldConfigure = configurableMap.get(key);
+ if (oldConfigure == null) {
+ continue;
+ }
+ return ConfigurableUtil.compare(oldConfigure, configure);
+ }
+ }
+ return false;
+ }
+
+ @Override
+ public <T extends IConfigurable> List<T> queryConfigurableByType(String type) {
+ List<IConfigurable> list = queryConfigurable(type);
+ if (list == null) {
+ return new ArrayList<T>();
+ }
+ List<T> result = new ArrayList<T>();
+ for (IConfigurable configurable : list) {
+ result.add((T)configurable);
+ }
+ return result;
+ }
+
+ @Override
+ public boolean refreshConfigurable(String namespace) {
+ //每次刷新，重新刷新配置文件
+ //if(ComponentCreator.propertiesPath!=null){
+ // ComponentCreator.setProperties(ComponentCreator.propertiesPath);
+ //}
+ this.namespace = namespace;
+ // Map<String, List<IConfigurable>> namespace2ConfigurableMap = new HashMap<>();
+ Map<String, List<IConfigurable>> tempType2ConfigurableMap = new HashMap<>();
+ Map<String, IConfigurable> tempName2ConfigurableMap = new HashMap<>();
+ GetConfigureResult configures = loadConfigurable(namespace);
+ // updateConfiguresCache(configures.getConfigure());
+ if (configures != null && configures.isQuerySuccess() && configures.getConfigurables() != null) {
+ // List<Configure> configureList = filterConfigure(configures.getConfigure());
+ List<IConfigurable> configurables = configures.getConfigurables();
+ List<IConfigurable> configurableList = checkAndUpdateConfigurables(namespace, configurables,
+ tempType2ConfigurableMap, tempName2ConfigurableMap,
+ configures.getConfigurables());
+ // this.namespace2ConfigurableMap = namespace2ConfigurableMap;
+ for (IConfigurable configurable : configurableList) {
+ if (configurable instanceof IAfterConfiguableRefreshListerner) {
+ ((IAfterConfiguableRefreshListerner)configurable).doProcessAfterRefreshConfigurable(this);
+ }
+ }
+ return true;
+ }
+ return false;
+ }
+
+ @Override
+ public <T> T queryConfigurable(String configurableType, String name) {
+ return (T)queryConfigurableByIdent(configurableType, name);
+ }
+
+ protected List<IConfigurable> checkAndUpdateConfigurables(String namespace, List<IConfigurable> configurables,
+ Map<String, List<IConfigurable>> tempType2ConfigurableMap,
+ Map<String, IConfigurable> tempName2ConfigurableMap,
+ List configures) {
+ List<IConfigurable> configurableList = new ArrayList<>();
+ for (IConfigurable configurable : configurables) {
+ try {
+ boolean isUpdate = update(configurable, tempName2ConfigurableMap, tempType2ConfigurableMap);
+ if (isUpdate) {
+ configurableList.add(configurable);
+ }
+ } catch (Exception e) {
+ LOG.error("组件初始化异常：" + e.getMessage() + ",name=" + configurable.getConfigureName(), e);
+ }
+ }
+ destroyOldConfigurables(tempName2ConfigurableMap);
+ this.name2ConfigurableMap = tempName2ConfigurableMap;
+ this.type2ConfigurableMap = tempType2ConfigurableMap;
+ return configurableList;
+ }
+
+ private void destroyOldConfigurables(Map<String, IConfigurable> tempName2ConfigurableMap) {
+ Iterator<Map.Entry<String, IConfigurable>> it = this.name2ConfigurableMap.entrySet().iterator();
+ while (it.hasNext()) {
+ Map.Entry<String, IConfigurable> entry = it.next();
+ String key = entry.getKey();
+ IConfigurable value = entry.getValue();
+ if (!tempName2ConfigurableMap.containsKey(key)) {
+ destroyOldConfigurable(value);
+ }
+ }
+
+ }
+
+ private void destroyOldConfigurable(IConfigurable oldConfigurable) {
+ if (AbstractConfigurable.class.isInstance(oldConfigurable)) {
+ ((AbstractConfigurable)oldConfigurable).destroy();
+ }
+ String key = getConfigureKey(oldConfigurable.getNameSpace(), oldConfigurable.getType(),
+ oldConfigurable.getConfigureName());
+ configurableMap.remove(key);
+ }
+
+ protected void initConfigurable(IConfigurable configurable) {
+ if (AbstractConfigurable.class.isInstance(configurable)) {
+ AbstractConfigurable abstractConfigurable = (AbstractConfigurable)configurable;
+ abstractConfigurable.setConfigurableService(this);
+ }
+
+ configurable.init();
+
+ }
+
+ /**
+ * 内部使用
+ */
+ private ScheduledExecutorService scheduledExecutorService;
+
+ @Override
+ public void initConfigurables(final String namespace) {
+ refreshConfigurable(namespace);
+ long polingTime = -1;
+ if (this.properties != null) {
+ String pollingTimeStr = this.properties.getProperty(AbstractComponent.POLLING_TIME);
+ if (StringUtil.isNotEmpty(pollingTimeStr)) {
+ polingTime = Long.valueOf(pollingTimeStr);
+ }
+ }
+ if (polingTime > 0) {
+ scheduledExecutorService = new ScheduledThreadPoolExecutor(3);
+ scheduledExecutorService.scheduleWithFixedDelay(new Runnable() {
+
+ @Override
+ public void run() {
+ try {
+ refreshConfigurable(namespace);
+ } catch (Exception e) {
+ LOG.error("Load configurables error:" + e.getMessage(), e);
+ }
+ }
+ }, polingTime, polingTime, TimeUnit.SECONDS);
+ }
+ }
+ // @Override
+ // public List<IConfigurable> queryConfigurable(String nameSpace) {
+ // return namespace2ConfigurableMap.get(nameSpace);
+ // }
+
+ @Override
+ public List<IConfigurable> queryConfigurable(String type) {
+ String key = MapKeyUtil.createKey(type);
+ return type2ConfigurableMap.get(key);
+ }
+
+ @Override
+ public IConfigurable queryConfigurableByIdent(String type, String name) {
+ String key = MapKeyUtil.createKey(type, name);
+ return name2ConfigurableMap.get(key);
+ }
+
+ /**
+ * 根据namespace加载配置信息
+ *
+ * @param namespace
+ * @return
+ * @throws Exception
+ */
+ protected abstract GetConfigureResult loadConfigurable(String namespace);
+
+ @Override
+ public void update(IConfigurable configurable) {
+ // update(configurable,name2ConfigurableMap,type2ConfigurableMap);
+ updateConfigurable(configurable);
+ }
+
+ protected abstract void updateConfigurable(IConfigurable configurable);
+
+ protected abstract void insertConfigurable(IConfigurable configurable);
+
+ protected boolean update(IConfigurable configurable, Map<String, IConfigurable> name2ConfigurableMap,
+ Map<String, List<IConfigurable>> type2ConfigurableMap) {
+ if (configurable == null) {
+ return false;
+ }
+
+ boolean isUpdate = false;
+ List<IConfigurable> configurableList = new ArrayList<>();
+ configurableList.add(configurable);
+
+ String nameKey = MapKeyUtil.createKey(configurable.getType(), configurable.getConfigureName());
+ if (this.name2ConfigurableMap.containsKey(nameKey)) {
+ String configureKey = getConfigureKey(namespace, configurable.getType(), configurable.getConfigureName());
+ IConfigurable oldConfigurable = this.name2ConfigurableMap.get(nameKey);
+ if (equals(configureKey, configurableList)) {
+ configurable = oldConfigurable;
+ // name2ConfigurableMap.put(nameKey, name2ConfigurableMap.get(nameKey));
+ } else {
+ destroyOldConfigurable(oldConfigurable);
+ initConfigurable(configurable);
+ isUpdate = true;
+ }
+ } else {
+ initConfigurable(configurable);
+ isUpdate = true;
+ }
+ updateConfiguresCache(configurable);
+ name2ConfigurableMap.put(nameKey, configurable);
+ String typeKey = MapKeyUtil.createKey(configurable.getType());
+ // put2Map(namespace2ConfigurableMap, namespace, configurable);
+ put2Map(type2ConfigurableMap, typeKey, configurable);
+ return isUpdate;
+ }
+
+ @Override
+ public void insert(IConfigurable configurable) {
+ // update(configurable,name2ConfigurableMap,type2ConfigurableMap);
+ insertConfigurable(configurable);
+ }
+
+ /**
+ * 给一个扣，可以跨命名空间查询数据
+ *
+ * @param namespaces
+ * @return
+ */
+ public List<IConfigurable> queryConfiguableByNamespace(String... namespaces) {
+ List<IConfigurable> configurables = new ArrayList<>();
+ if (namespaces == null || namespaces.length == 0) {
+ return configurables;
+ }
+ for (String namespace : namespaces) {
+ GetConfigureResult result = loadConfigurable(namespace);
+ if (result.querySuccess) {
+ if (result.configurables != null && result.configurables.size() > 0) {
+ configurables.addAll(result.configurables);
+ }
+ } else {
+ throw new RuntimeException("Load configurable error, the namespace is " + namespace);
+ }
+ }
+ return configurables;
+
+ }
+
+ /**
+ * 往一个value是list的map中添加数据，如果list是空创建，否则直接插入
+ *
+ * @param map
+ * @param key
+ * @param configurable
+ */
+ protected void put2Map(Map<String, List<IConfigurable>> map, String key, IConfigurable configurable) {
+ List<IConfigurable> list = map.computeIfAbsent(key, k -> new ArrayList<IConfigurable>());
+ list.add(configurable);
+ }
+
+ @Override
+ public Collection<IConfigurable> findAll() {
+ return name2ConfigurableMap.values();
+ }
+
+ /**
+ * 把configurable转换成configure
+ *
+ * @param configurable
+ * @return
+ */
+ protected Configure createConfigure(IConfigurable configurable) {
+ Configure configure = new Configure();
+ configure.setType(configurable.getType());
+ configure.setName(configurable.getConfigureName());
+ configure.setNameSpace(configurable.getNameSpace());
+ String jsonString = configurable.toJson();
+ if (!StringUtil.isEmpty(jsonString)) {
+ JSONObject jsonObject = JSONObject.parseObject(jsonString);
+ jsonObject.put(CLASS_NAME, configurable.getClass().getName());
+ configure.setJsonValue(jsonObject.toJSONString());
+ }
+ // configure.createIdentification();
+ return configure;
+ }
+
+ @Override
+ public <T> Map<String, T> queryConfigurableMapByType(String type) {
+ List<IConfigurable> configurables = queryConfigurable(type);
+ if (configurables == null) {
+ return new HashMap<String, T>();
+ }
+ Map<String, T> result = new HashMap<String, T>();
+ for (IConfigurable configurable : configurables) {
+ result.put(configurable.getConfigureName(), (T)configurable);
+ }
+ return result;
+ }
+
+ /**
+ * 把configure转换成configurable
+ *
+ * @param configures
+ * @return
+ */
+ protected List<IConfigurable> convert(List<Configure> configures) {
+ if (configures == null) {
+ return new ArrayList<IConfigurable>();
+ }
+ List<IConfigurable> configurables = new ArrayList<IConfigurable>();
+ for (Configure configure : configures) {
+ IConfigurable configurable = convert(configure);
+ if (configurable != null) {
+ configurables.add(configurable);
+ }
+
+ }
+ return configurables;
+ }
+
+ protected IConfigurable createConfigurableFromJson(String namespace, String type, String name, String jsonValue) {
+ if (StringUtil.isEmpty(jsonValue)) {
+ return null;
+ }
+ JSONObject jsonObject = JSONObject.parseObject(jsonValue);
+ String className = jsonObject.getString(CLASS_NAME);
+ IConfigurable configurable = createConfigurable(className);
+ if (configurable == null) {
+ return null;
+ }
+ configurable.setConfigureName(name);
+ configurable.setNameSpace(namespace);
+ configurable.setType(type);
+ if (AbstractConfigurable.class.isInstance(configurable)) {
+ AbstractConfigurable abstractConfigurable = (AbstractConfigurable)configurable;
+ abstractConfigurable.setConfigurableService(this);
+ }
+ configurable.toObject(jsonValue);
+ return configurable;
+ }
+
+ /**
+ * 提供一个入口，可以让外部用户改变configure对应的configurable的值
+ *
+ * @param configure
+ * @return
+ */
+ protected IConfigurable convert(Configure configure) {
+
+ return convertConfigurable(configure);
+ }
+
+ protected IConfigurable convertConfigurable(Configure configure) {
+ String className = null;
+ try {
+ String jsonString = configure.getJsonValue();
+ IConfigurable configurable =
+ createConfigurableFromJson(configure.getNameSpace(), configure.getType(), configure.getName(),
+ jsonString);
+ if (configurable instanceof Entity) {
+ // add by wangtl 20171110 Configurable接口第三方包也在用，故不能Configurable里加接口，只能加到抽象类里，这里强转下
+ Entity abs = (Entity)configurable;
+ abs.setId(configure.getId());
+ abs.setGmtCreate(configure.getGmtCreate());
+ abs.setGmtModified(configure.getGmtModified());
+ /*
+ * abs.setTempKey((configurable.getNameSpace() + configurable.getType() +
+ * configurable.getConfigureName() + jsonString).hashCode());
+ */
+ }
+ convertPost(configurable);
+ return configurable;
+ } catch (Exception e) {
+ LOG.error("转换异常：" + configure.toString(), e);
+ return null;
+ }
+ }
+
+ /**
+ * 如果需要改变configurable的属性，可以再这里设置
+ *
+ * @param configurable
+ */
+ @SuppressWarnings("rawtypes")
+ protected void convertPost(IConfigurable configurable) {
+ if (this.properties == null) {
+ return;
+ }
+ String identification =
+ MapKeyUtil.createKey(configurable.getNameSpace(), configurable.getType(), configurable.getConfigureName());
+ String propertyValue = this.properties.getProperty(identification);
+ if (StringUtil.isEmpty(propertyValue)) {
+ return;
+ }
+ String[] fieldName2Values = propertyValue.split(",");
+ if (fieldName2Values.length == 0) {
+ return;
+ }
+ for (String fieldName2Value : fieldName2Values) {
+ try {
+ String[] fieldName2ValueArray = fieldName2Value.split(":");
+ if (fieldName2ValueArray.length != 2) {
+ continue;
+ }
+ String fieldName = fieldName2ValueArray[0];
+ String value = fieldName2ValueArray[1];
+ Class clazz = ReflectUtil.getBeanFieldType(configurable.getClass(), fieldName);
+ DataType dataType = DataTypeUtil.createDataType(clazz, null);
+ if (dataType == null) {
+ continue;
+ }
+ Object fieldValue = dataType.getData(value);
+ ReflectUtil.setBeanFieldValue(configurable, fieldName, fieldValue);
+
+ } catch (Exception e) {
+ LOG.error("convert post error " + fieldName2Value, e);
+ continue;
+ }
+
+ }
+ }
+
+ /**
+ * 创建configurable对象
+ *
+ * @param className class name
+ * @return
+ */
+ @SuppressWarnings("rawtypes")
+ protected IConfigurable createConfigurable(String className) {
+ return ReflectUtil.forInstance(className);
+ }
+
+ public class GetConfigureResult {
+
+ private boolean querySuccess;
+ private List<IConfigurable> configurables;
+
+ public boolean isQuerySuccess() {
+ return querySuccess;
+ }
+
+ public void setQuerySuccess(boolean querySuccess) {
+ this.querySuccess = querySuccess;
+ }
+
+ public List<IConfigurable> getConfigurables() {
+ return configurables;
+ }
+
+ public void setConfigurables(List<IConfigurable> configurables) {
+ this.configurables = configurables;
+ }
+ }
+
+ @Override
+ public String getNamespace() {
+ return namespace;
+ }
+
+ public Properties getProperties() {
+ return properties;
+ }
+
+ public void setProperties(Properties properties) {
+ this.properties = properties;
+ }
+}
diff --git a/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configuable/service/AbstractSupportParentConfigureService.java b/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configuable/service/AbstractSupportParentConfigureService.java
new file mode 100644
index 0000000..390a119
--- /dev/null
+++ b/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configuable/service/AbstractSupportParentConfigureService.java
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.streams.configuable.service;
+
+import org.apache.rocketmq.streams.common.configurable.IConfigurable;
+import org.apache.rocketmq.streams.common.configurable.IConfigurableListerner;
+import org.apache.rocketmq.streams.common.configurable.IConfigurableService;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.Properties;
+
+/**
+ * namespace 分层，支持顶级命名空间，顶级命名空间的对象，所有命名空间都可见。顶级命名空间是固定值IConfigurableService.PARENT_CHANNEL_NAME_SPACE
+ */
+public abstract class AbstractSupportParentConfigureService extends AbstractConfigurableService
+ implements IConfigurableService {
+
+ private static final Log LOG = LogFactory.getLog(AbstractSupportParentConfigureService.class);
+ protected IConfigurableService configureService = null;
+ protected IConfigurableService parentConfigureService = null;
+ //protected IConfigurableService shareConfigureService = null;
+ protected Properties properties;
+
+ public AbstractSupportParentConfigureService() {
+ super(null);
+ }
+
+ public void initMethod(Properties property) {
+ this.properties = property;
+ initBeforeInitConfigurable(property);
+ }
+
+ protected abstract void initBeforeInitConfigurable(Properties property);
+
+
+
+
+ @Override
+ public void initConfigurables(String namespace) {
+
+ if (!IConfigurableService.PARENT_CHANNEL_NAME_SPACE.equals(namespace)) {
+ parentConfigureService.initConfigurables(IConfigurableService.PARENT_CHANNEL_NAME_SPACE);
+ } else {
+ parentConfigureService = null;
+ }
+ configureService.initConfigurables(namespace);
+ }
+
+ @Override
+ public boolean refreshConfigurable(String namespace) {
+
+ if (!IConfigurableService.PARENT_CHANNEL_NAME_SPACE.equals(namespace)) {
+ parentConfigureService.refreshConfigurable(IConfigurableService.PARENT_CHANNEL_NAME_SPACE);
+ // initShareConfigurableService(namespace);
+ }
+ configureService.refreshConfigurable(namespace);
+ return true;
+ }
+
+
+
+ @Override
+ public List<IConfigurable> queryConfigurable(String type) {
+ List<IConfigurable> result = configureService.queryConfigurable(type);
+ if (result == null) {
+ result = new ArrayList<>();
+ }
+ //if (shareConfigureService != null) {
+ // List<IConfigurable> share = shareConfigureService.queryConfigurable(type);
+ // if (share != null) {
+ // result.addAll(share);
+ // }
+ //}
+ if (parentConfigureService == null) {
+ return result;
+ }
+ List<IConfigurable> parent = parentConfigureService.queryConfigurable(type);
+ if (parent != null) {
+ result.addAll(parent);
+ }
+ return result;
+ }
+
+ @Override
+ public IConfigurable queryConfigurableByIdent(String type, String name) {
+ IConfigurable configurable = configureService.queryConfigurableByIdent(type, name);
+ if (configurable != null) {
+ return configurable;
+ }
+ if (parentConfigureService == null) {
+ return null;
+ }
+ //if (shareConfigureService != null) {
+ // configurable = shareConfigureService.queryConfigurableByIdent(type, name);
+ //}
+ if (configurable != null) {
+ return configurable;
+ }
+ return parentConfigureService.queryConfigurableByIdent(type, name);
+ }
+
+ @Override
+ public IConfigurable queryConfigurableByIdent(String identification) {
+ IConfigurable configurable = configureService.queryConfigurableByIdent(identification);
+ if (configurable != null) {
+ return configurable;
+ }
+ if (parentConfigureService == null) {
+ return null;
+ }
+ //if (shareConfigureService != null) {
+ // configurable = shareConfigureService.queryConfigurableByIdent(identification);
+ //}
+ if (configurable != null) {
+ return configurable;
+ }
+ return parentConfigureService.queryConfigurableByIdent(identification);
+ }
+
+ @Override
+ protected void insertConfigurable(IConfigurable configurable) {
+ if (parentConfigureService != null && configurable.getNameSpace()
+ .equals(IConfigurableService.PARENT_CHANNEL_NAME_SPACE)) {
+ parentConfigureService.insert(configurable);
+ } else {
+ configureService.insert(configurable);
+ }
+ }
+
+ @Override
+ protected void updateConfigurable(IConfigurable configurable) {
+ if (parentConfigureService != null && configurable.getNameSpace()
+ .equals(IConfigurableService.PARENT_CHANNEL_NAME_SPACE)) {
+ parentConfigureService.update(configurable);
+ } else {
+ configureService.update(configurable);
+ }
+ }
+
+ @Override
+ public <T> T queryConfigurable(String configurableType, String name) {
+ return (T) queryConfigurableByIdent(configurableType, name);
+ }
+
+ @Override
+ protected GetConfigureResult loadConfigurable(String namespace) {
+ return null;
+ }
+
+ //protected void initShareConfigurableService(String namespace) {
+ // if (parentConfigureService == null) {
+ // return;
+ // }
+ // shareConfigureService = new AbstractReadOnlyConfigurableService() {
+ //
+ // @Override
+ // public <T extends IConfigurable> List<T> loadConfigurableFromStorage(String type) {
+ // refreshConfigurable(namespace);
+ // return queryConfigurableByType(type);
+ // }
+ //
+ // @Override
+ // protected List<IConfigurable> loadConfigurables(String namespace) {
+ // List<IConfigurable> parent = parentConfigureService.queryConfigurable(ShareConfiguable.TYPE);
+ // List<IConfigurable> shareConfigurables = new ArrayList<>();
+ // if (parent == null) {
+ // return shareConfigurables;
+ // }
+ // for (IConfigurable configurable : parent) {
+ // ShareConfiguable shareConfiguable = (ShareConfiguable) configurable;
+ // if (shareConfiguable.getShareAll() || shareConfiguable.getShareNameSpaces().contains(namespace)) {
+ // String sharedNameSpace = shareConfiguable.getSharedNameSpace();
+ // String sharedType = shareConfiguable.getSharedType();
+ // String sharedName = shareConfiguable.getSharedName();
+ // List<IConfigurable> sharedConfigrables =
+ // createAndQueryConfigurable(sharedNameSpace, sharedType, sharedName);
+ // if (sharedConfigrables != null) {
+ // shareConfigurables.addAll(sharedConfigrables);
+ // }
+ // }
+ // }
+ // return shareConfigurables;
+ // }
+ //
+ //
+ // };
+ // shareConfigureService.refreshConfigurable(namespace);
+ //
+ //}
+
+ protected List<IConfigurable> createAndQueryConfigurable(String sharedNameSpace, String sharedType,
+ String sharedName) {
+ IConfigurableService innerSharedConfigurableService =
+ ConfigurableServiceFactory.createConfigurableService(properties);
+ innerSharedConfigurableService.refreshConfigurable(sharedNameSpace);
+ if (StringUtil.isNotEmpty(sharedName)) {
+ List<IConfigurable> configurables = new ArrayList<>();
+ IConfigurable configurable = innerSharedConfigurableService.queryConfigurableByIdent(sharedType, sharedName);
+ configurables.add(configurable);
+ return configurables;
+ } else {
+ return innerSharedConfigurableService.queryConfigurable(sharedType);
+ }
+
+ }
+
+
+ @Override
+ public Collection<IConfigurable> findAll() {
+ List<IConfigurable> configurables=new ArrayList<>();
+ if (parentConfigureService != null ) {
+ Collection<IConfigurable> tmp=parentConfigureService.findAll();
+ if(tmp!=null||tmp.size()>0){
+ configurables.addAll(tmp);
+ }
+ }
+ Collection<IConfigurable> tmp=configureService.findAll();
+ if(tmp!=null||tmp.size()>0){
+ configurables.addAll(tmp);
+ }
+ return configurables;
+ }
+
+ public IConfigurableService getConfigureService() {
+ return configureService;
+ }
+
+ @Override
+ public <T extends IConfigurable> List<T> loadConfigurableFromStorage(String type) {
+ List<T> configurables=new ArrayList<>();
+ if (parentConfigureService != null ) {
+ Collection<T> tmp=parentConfigureService.loadConfigurableFromStorage(type);
+ if(tmp!=null||tmp.size()>0){
+ configurables.addAll(tmp);
+ }
+ }
+ Collection<T> tmp=configureService.loadConfigurableFromStorage(type);
+ if(tmp!=null||tmp.size()>0){
+ configurables.addAll(tmp);
+ }
+ return configurables;
+ }
+}
diff --git a/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configuable/service/ConfigurableServcieType.java b/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configuable/service/ConfigurableServcieType.java
new file mode 100644
index 0000000..f2b72ab
--- /dev/null
+++ b/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configuable/service/ConfigurableServcieType.java
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.streams.configuable.service;
+
+import org.apache.rocketmq.streams.common.configurable.IConfigurableService;
+
+@Deprecated
+public class ConfigurableServcieType {
+
+ public static final String DEFAULT_SERVICE_NAME = IConfigurableService.DEFAULT_SERVICE_NAME;
+ public static final String MEMORY_SERVICE_NAME = IConfigurableService.MEMORY_SERVICE_NAME;
+ public static final String FILE_SERVICE_NAME = IConfigurableService.FILE_SERVICE_NAME;
+ public static final String HTTP_SERVICE_NAME = IConfigurableService.HTTP_SERVICE_NAME;
+
+
+}
diff --git a/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configuable/service/ConfigurableServiceFactory.java b/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configuable/service/ConfigurableServiceFactory.java
new file mode 100644
index 0000000..307dcae
--- /dev/null
+++ b/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configuable/service/ConfigurableServiceFactory.java
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.streams.configuable.service;
+
+import java.util.Properties;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.rocketmq.streams.common.configure.ConfigureFileKey;
+import org.apache.rocketmq.streams.common.configurable.IConfigurableService;
+import org.apache.rocketmq.streams.common.utils.ReflectUtil;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+import org.apache.rocketmq.streams.serviceloader.ServiceLoaderComponent;
+
+public class ConfigurableServiceFactory {
+ private static final ServiceLoaderComponent<IConfigurableService> configurableServiceLoaderComponent = ServiceLoaderComponent.getInstance(IConfigurableService.class);
+ private static final Log LOG = LogFactory.getLog(ConfigurableServiceFactory.class);
+
+ public static IConfigurableService createConfigurableService(Properties properties) {
+ try {
+ Properties properties1 = new Properties();
+ properties1.putAll(properties);
+ String type = properties1.getProperty(ConfigureFileKey.CONNECT_TYPE);
+ if (StringUtil.isEmpty(type)) {
+ type = IConfigurableService.DEFAULT_SERVICE_NAME;
+ }
+ IConfigurableService configurableService = getConfigurableServiceType(type);
+ if (configurableService instanceof AbstractSupportParentConfigureService) {
+ ((AbstractSupportParentConfigureService)configurableService).initMethod(properties1);
+ }
+ return configurableService;
+ } catch (Exception e) {
+ LOG.error("create ConfigurableService error", e);
+ return null;
+ }
+
+ }
+
+ public static IConfigurableService getConfigurableServiceType(String type) {
+ IConfigurableService configurableService = configurableServiceLoaderComponent.getService().loadService(type);
+ return ReflectUtil.forInstance(configurableService.getClass().getName());
+ }
+}
diff --git a/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configuable/service/impl/FileConfigureService.java b/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configuable/service/impl/FileConfigureService.java
new file mode 100644
index 0000000..d251bda
--- /dev/null
+++ b/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configuable/service/impl/FileConfigureService.java
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.streams.configuable.service.impl;
+
+import org.apache.rocketmq.streams.common.configurable.IConfigurable;
+import org.apache.rocketmq.streams.common.configurable.IConfigurableService;
+import org.apache.rocketmq.streams.configuable.service.AbstractConfigurableService;
+import org.apache.rocketmq.streams.configuable.model.Configure;
+import org.apache.rocketmq.streams.common.model.Entity;
+import org.apache.rocketmq.streams.common.utils.DateUtil;
+import org.apache.rocketmq.streams.common.utils.FileUtil;
+import org.apache.rocketmq.streams.common.utils.MapKeyUtil;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Properties;
+
+public class FileConfigureService extends AbstractConfigurableService {
+
+ public static final String FILE_PATH_NAME = IConfigurableService.FILE_PATH_NAME;
+ // 配置文件的路径
+ private static final Log LOG = LogFactory.getLog(FileConfigureService.class);
+ private static final String DEFAULT_FILE_NAME = "dipper_configure.cs"; // 默认文件名
+ private static final String SIGN = "&&&&"; // 字段分割附号
+ public String fileName;
+
+ public FileConfigureService(Properties properties) {
+ super(properties);
+ initService(properties.getProperty(FILE_PATH_NAME));
+ }
+
+ protected void initService(String fileAndPath) {
+ if (StringUtil.isEmpty(fileAndPath)) {
+ String path = this.getClass().getProtectionDomain().getCodeSource().getLocation().getPath();
+ if (path.endsWith(".jar")) {
+ int index = path.lastIndexOf(File.separator);
+ path = path.substring(0, index);
+ }
+ fileName = FileUtil.concatFilePath(path, DEFAULT_FILE_NAME);
+ } else {
+ fileName = fileAndPath;
+ }
+ LOG.info("load file from path = " + fileName);
+ }
+
+ @Override
+ protected GetConfigureResult loadConfigurable(String namespace) {
+ GetConfigureResult result = new GetConfigureResult();
+ try {
+ List<Configure> configures = selectOpening(namespace);
+ List<IConfigurable> configurables = convert(configures);
+ LOG.info("load configure namespace=" + namespace + " count=" + configures.size());
+ result.setConfigurables(configurables);
+ result.setQuerySuccess(true);// 该字段标示查询是否成功，若不成功则不会更新配置
+ } catch (Exception e) {
+ result.setQuerySuccess(false);
+ e.printStackTrace();
+ LOG.error("load configurable error ", e);
+ }
+ return result;
+ }
+
+ protected List<Configure> selectOpening(String namespace) {
+ List<String> list = loadFileLine(fileName);
+ List<Configure> configures = convert2Configure(list);
+ return filter(configures, namespace);
+ }
+
+ protected List<Configure> filter(List<Configure> configures, String namespace) {
+ if (configures == null) {
+ return new ArrayList<>();
+ }
+ if (StringUtil.isEmpty(namespace)) {
+ throw new RuntimeException("namespace can not empty ");
+ }
+ List<Configure> filterConfigures = new ArrayList<>();
+ for (Configure configure : configures) {
+ if (!namespace.equals(configure.getNameSpace())) {
+ continue;
+ }
+ filterConfigures.add(configure);
+ }
+ return filterConfigures;
+ }
+
+ @Override
+ protected void insertConfigurable(IConfigurable configure) {
+ if (configure == null) {
+ LOG.warn("insert configure is null");
+ return;
+ }
+ String row = configure2String(configure);
+
+ List<String> rows = loadFileLine(fileName);
+ if (rows == null) {
+ rows = new ArrayList<>();
+ }
+ List<Configure> configures = convert2Configure(rows);
+ String newKey =
+ MapKeyUtil.createKey(configure.getNameSpace(), configure.getType(), configure.getConfigureName());
+ boolean isReplace = false;
+ for (int i = 0; i < configures.size(); i++) {
+ Configure c = configures.get(i);
+ String old = MapKeyUtil.createKey(c.getNameSpace(), c.getType(), c.getName());
+ if (old.equals(newKey)) {
+ rows.set(i, configure2String(configure));
+ isReplace = true;
+ break;
+ }
+ }
+ if (!isReplace) {
+ rows.add(configure2String(configure));
+ }
+ writeFile(fileName, rows);
+ }
+
+ @Override
+ protected void updateConfigurable(IConfigurable configure) {
+ if (configure == null) {
+ LOG.warn("insert configure is null");
+ return;
+ }
+
+ List<String> rows = FileUtil.loadFileLine(fileName);
+ if (rows == null) {
+ rows = new ArrayList<>();
+ }
+ for (int i = 0; i < rows.size(); i++) {
+ String row = rows.get(i);
+ Configure oldConfigure = convert(row);
+ if (configure.getNameSpace().equals(oldConfigure.getNameSpace()) && configure.getType()
+ .equals(oldConfigure.getType()) && configure.getConfigureName().equals(oldConfigure.getName())) {
+ rows.set(i, configure2String(configure));
+ }
+ }
+ writeFile(fileName, rows);
+
+ }
+
+ protected Configure convert(String row) {
+ String[] values = row.split(SIGN);
+ String namespace = getColumnValue(values, 0, "namespace");
+ String type = getColumnValue(values, 1, "type");
+ String name = getColumnValue(values, 2, "name");
+ String jsonValue = getColumnValue(values, 3, "json_value");
+ String createDate = getColumnValue(values, 4, "gmt_create");
+ String modifiedDate = getColumnValue(values, 5, "gmt_modified");
+ String id = getColumnValue(values, 6, "id");
+ Configure configure = new Configure();
+ configure.setNameSpace(namespace);
+ configure.setType(type);
+ configure.setName(name);
+ configure.setJsonValue(jsonValue);
+ configure.setGmtCreate(DateUtil.parse(createDate));
+ configure.setGmtCreate(DateUtil.parse(modifiedDate));
+ configure.setId((id == null ? null : Long.valueOf(id)));
+
+ return configure;
+ }
+
+ protected List<Configure> convert2Configure(List<String> rows) {
+ List<Configure> configures = new ArrayList<Configure>();
+ for (String row : rows) {
+ configures.add(convert(row));
+ }
+ return configures;
+ }
+
+ protected String getColumnValue(String[] values, int i, String namespace) {
+ if (values == null || values.length == 0) {
+ return null;
+ }
+ if (values.length <= i) {
+ return null;
+ }
+ if ("null".equals(values[i])) {
+ return null;
+ }
+ return values[i];
+ }
+
+ /**
+ * 解密文件，并加载到内存
+ *
+ * @param fileName
+ * @return
+ */
+ protected List<String> loadFileLine(String fileName) {
+ List<String> rows = FileUtil.loadFileLine(fileName);
+ if (rows == null) {
+ rows = new ArrayList<>();
+ }
+ return doDecRowList(rows);
+ }
+
+ protected void writeFile(String fileName, List<String> rows) {
+ List<String> rowList = doEncryptRowList(rows);
+ FileUtil.write(fileName, rowList);
+ }
+
+ private List<String> doEncryptRowList(List<String> rows) {
+ return rows;
+ }
+
+ private List<String> doDecRowList(List<String> rows) {
+ return rows;
+ }
+
+ protected String configure2String(IConfigurable configure) {
+ Entity entity = null;
+ if (configure instanceof Entity) {
+ entity = (Entity)configure;
+ } else {
+ entity = new Entity();
+ }
+ String row = MapKeyUtil.createKeyBySign(SIGN, configure.getNameSpace(), configure.getType(),
+ configure.getConfigureName(), configure.toJson(), DateUtil.format(entity.getGmtCreate()),
+ DateUtil.format(entity.getGmtModified()), entity.getId() + "");
+ return row;
+ }
+
+ public String getFileName() {
+ return fileName;
+ }
+
+ @Override
+ public <T extends IConfigurable> List<T> loadConfigurableFromStorage(String type) {
+ refreshConfigurable(getNamespace());
+ return queryConfigurableByType(type);
+ }
+}
diff --git a/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configuable/service/impl/FileSupportParentConfigureService.java b/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configuable/service/impl/FileSupportParentConfigureService.java
new file mode 100644
index 0000000..bf8d151
--- /dev/null
+++ b/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configuable/service/impl/FileSupportParentConfigureService.java
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.streams.configuable.service.impl;
+
+import org.apache.rocketmq.streams.configuable.service.AbstractSupportParentConfigureService;
+import org.apache.rocketmq.streams.common.configurable.IConfigurableService;
+import org.apache.rocketmq.streams.configuable.service.ConfigurableServcieType;
+import org.apache.rocketmq.streams.common.model.ServiceName;
+import com.google.auto.service.AutoService;
+
+import java.util.Properties;
+
+@AutoService(IConfigurableService.class)
+@ServiceName(ConfigurableServcieType.FILE_SERVICE_NAME)
+public class FileSupportParentConfigureService extends AbstractSupportParentConfigureService {
+
+ @Override
+ protected void initBeforeInitConfigurable(Properties property) {
+ this.configureService = new FileConfigureService(properties);
+ this.parentConfigureService = new FileConfigureService(properties);
+ }
+
+}
diff --git a/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configuable/service/impl/MemoryConfigureService.java b/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configuable/service/impl/MemoryConfigureService.java
new file mode 100644
index 0000000..e846e44
--- /dev/null
+++ b/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configuable/service/impl/MemoryConfigureService.java
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.streams.configuable.service.impl;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import org.apache.rocketmq.streams.common.configurable.AbstractConfigurable;
+import org.apache.rocketmq.streams.common.configurable.IConfigurable;
+import org.apache.rocketmq.streams.configuable.service.AbstractConfigurableService;
+import org.apache.rocketmq.streams.common.utils.ReflectUtil;
+
+public class MemoryConfigureService extends AbstractConfigurableService {
+
+ private static Map<String, List<IConfigurable>> namespace2Configure = new HashMap<>();
+
+ public MemoryConfigureService(Properties properties) {
+ super(properties);
+ }
+
+ @Override
+ protected GetConfigureResult loadConfigurable(String namespace) {
+ GetConfigureResult result = new GetConfigureResult();
+ result.setQuerySuccess(true);
+ List<IConfigurable> configurableList = new ArrayList<>();
+ List<IConfigurable> configurables = namespace2Configure.get(namespace);
+ if (configurables == null) {
+ configurableList = null;
+ } else {
+ List<IConfigurable> tmps = new ArrayList<>();
+ tmps.addAll(configurables);
+ for (IConfigurable configurable : tmps) {
+ IConfigurable tmp = ReflectUtil.forInstance(configurable.getClass());
+ tmp.toObject(configurable.toJson());
+ tmp.setNameSpace(configurable.getNameSpace());
+ tmp.setConfigureName(configurable.getConfigureName());
+ configurableList.add(tmp);
+ }
+ }
+ result.setConfigurables(configurableList);
+ return result;
+ }
+
+ @Override
+ protected void insertConfigurable(IConfigurable configurable) {
+ if (configurable == null) {
+ return;
+ }
+
+ String namespace = configurable.getNameSpace();
+ List<IConfigurable> list = namespace2Configure.get(namespace);
+ if (list == null) {
+ synchronized (this) {
+ list = namespace2Configure.get(namespace);
+ if (list == null) {
+ list = new ArrayList<>();
+ namespace2Configure.put(namespace, list);
+ }
+ }
+ }
+ int removeIndex = -1;
+ for (int i = 0; i < list.size(); i++) {
+ IConfigurable config = list.get(i);
+ if (config.getType().equals(configurable.getType()) && config.getConfigureName()
+ .equals(configurable.getConfigureName())) {
+ removeIndex = i;
+ }
+ }
+ if (AbstractConfigurable.class.isInstance(configurable)) {
+ ((AbstractConfigurable)configurable).setConfigurableService(this);
+ }
+ if (removeIndex != -1) {
+ list.remove(removeIndex);
+ }
+ list.add(configurable);
+ }
+
+ @Override
+ protected void updateConfigurable(IConfigurable configure) {
+ List<IConfigurable> list = namespace2Configure.get(configure.getNameSpace());
+ if (list == null || list.size() == 0) {
+ throw new RuntimeException(
+ "not have exist configure " + configure.getNameSpace() + "," + configure.getType() + ","
+ + configure.getConfigureName());
+ }
+ for (int i = 0; i < list.size(); i++) {
+ IConfigurable config = list.get(i);
+ if (config.getType().equals(configure.getType()) && config.getConfigureName()
+ .equals(configure.getConfigureName())) {
+ list.set(i, configure);
+ return;
+ }
+ }
+ throw new RuntimeException(
+ "not have exist configure " + configure.getNameSpace() + "," + configure.getType() + ","
+ + configure.getConfigureName());
+ }
+
+ @Override
+ public <T extends IConfigurable> List<T> loadConfigurableFromStorage(String type) {
+ refreshConfigurable(getNamespace());
+ return queryConfigurableByType(type);
+ }
+}
diff --git a/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configuable/service/impl/MemorySupportParentConfigureService.java b/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configuable/service/impl/MemorySupportParentConfigureService.java
new file mode 100644
index 0000000..528d087
--- /dev/null
+++ b/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configuable/service/impl/MemorySupportParentConfigureService.java
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.streams.configuable.service.impl;
+
+import java.util.Properties;
+
+import org.apache.rocketmq.streams.configuable.service.AbstractSupportParentConfigureService;
+import org.apache.rocketmq.streams.common.configurable.IConfigurableService;
+import org.apache.rocketmq.streams.configuable.service.ConfigurableServcieType;
+import org.apache.rocketmq.streams.common.model.ServiceName;
+import com.google.auto.service.AutoService;
+
+
+@AutoService(IConfigurableService.class)
+@ServiceName(ConfigurableServcieType.MEMORY_SERVICE_NAME)
+public class MemorySupportParentConfigureService extends AbstractSupportParentConfigureService {
+
+ @Override
+ protected void initBeforeInitConfigurable(Properties property) {
+ // this.rootConfigureService = new MemoryConfigureService(property);
+ this.parentConfigureService = new MemoryConfigureService(properties);
+ this.configureService = new MemoryConfigureService(properties);
+ }
+}
diff --git a/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configurable/ConfigurableComponent.java b/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configurable/ConfigurableComponent.java
new file mode 100644
index 0000000..91b3171
--- /dev/null
+++ b/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configurable/ConfigurableComponent.java
@@ -0,0 +1,188 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.configurable;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import org.apache.rocketmq.streams.common.component.AbstractComponent;
+import org.apache.rocketmq.streams.common.component.ComponentCreator;
+import org.apache.rocketmq.streams.common.component.ConfigureDescriptor;
+import org.apache.rocketmq.streams.common.configurable.IConfigurable;
+import org.apache.rocketmq.streams.common.configurable.IConfigurableService;
+import org.apache.rocketmq.streams.configurable.service.AbstractConfigurableService;
+import org.apache.rocketmq.streams.configurable.service.ConfigurableServcieType;
+import org.apache.rocketmq.streams.configurable.service.ConfigurableServiceFactory;
+import org.apache.rocketmq.streams.common.utils.ConfigurableUtil;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+/**
+ * 对Configurable对象，做统一的管理，统一查询，插入和更新。 insert/update 把configuabel对象写入存储，支持文件存储（file），内存存储（memory）和db存储（DB）。可以在配置通过这个ConfigureFileKey.CONNECT_TYPE key 配置 query 是基于内存的查询，对象定时load到内存，可以在属性文件通过这个ConfigureFileKey.POLLING_TIME key配置加载周期，单位是秒 新对象加载后生效，已经存在的对象只有updateFlag发生变化才会被替换
+ */
+public class ConfigurableComponent extends AbstractComponent<IConfigurableService>
+ implements IConfigurableService {
+
+ private static final Log LOG = LogFactory.getLog(ConfigurableComponent.class);
+
+ protected volatile IConfigurableService configureService = null;
+
+ protected transient String namespace;
+
+ public ConfigurableComponent() {
+ initConfigurableServiceDescriptor();
+ addConfigureDescriptor(
+ new ConfigureDescriptor(CONNECT_TYPE, false, ConfigurableServcieType.DEFAULT_SERVICE_NAME));
+ }
+
+ public static ConfigurableComponent getInstance(String namespace) {
+ return ComponentCreator.getComponent(namespace, ConfigurableComponent.class);
+ }
+
+ @Override
+ protected boolean initProperties(Properties properties) {
+ try {
+ if (configureService != null) {
+ return true;
+ }
+ this.configureService = ConfigurableServiceFactory.createConfigurableService(properties);
+ return true;
+ } catch (Exception e) {
+ LOG.error("ConfigurableComponent create error,properties= " + properties, e);
+ return false;
+ }
+
+ }
+
+ @Override
+ public boolean startComponent(String namespace) {
+ try {
+ this.namespace = namespace;
+ configureService.initConfigurables(namespace);
+ return true;
+ } catch (Exception e) {
+ LOG.error("ConfigurableComponent init error, namespace is " + namespace, e);
+ return false;
+ }
+
+ }
+
+ /**
+ * 启动测试模式，用内存数据库存储和加载configurable数据
+ */
+ public static void begineTestMode() {
+ System.setProperty(ConfigurableComponent.CONNECT_TYPE, ConfigurableServcieType.MEMORY_SERVICE_NAME);
+ }
+
+ /**
+ * 关闭测试模式，用配置文件中配置的属性加载configuable数据
+ */
+ public static void endTestMode() {
+ System.clearProperty(ConfigurableComponent.CONNECT_TYPE);
+ }
+
+ @Override
+ public boolean stop() {
+ return true;
+ }
+
+ @Override
+ public IConfigurableService getService() {
+ return configureService;
+ }
+
+ @Override
+ public void initConfigurables(String namespace) {
+ configureService.initConfigurables(namespace);
+ }
+
+ @Override
+ public boolean refreshConfigurable(String namespace) {
+ return configureService.refreshConfigurable(namespace);
+ }
+
+ public void mockConfigurable(String namespace) {
+ refreshConfigurable(namespace);
+
+ }
+
+ @Override
+ public List<IConfigurable> queryConfigurable(String type) {
+ return configureService.queryConfigurable(type);
+ }
+
+ @Override
+ public <T extends IConfigurable> List<T> queryConfigurableByType(String type) {
+ return configureService.queryConfigurableByType(type);
+ }
+
+ @Override
+ public IConfigurable queryConfigurableByIdent(String type, String name) {
+ return configureService.queryConfigurableByIdent(type, name);
+ }
+
+ @Override
+ public IConfigurable queryConfigurableByIdent(String identification) {
+ return configureService.queryConfigurableByIdent(identification);
+ }
+
+ @Override
+ public void insert(IConfigurable configurable) {
+ configureService.insert(configurable);
+ ConfigurableUtil.refreshMock(configurable);
+ }
+
+ @Override
+ public void update(IConfigurable configurable) {
+ configureService.update(configurable);
+ }
+
+ @Override
+ public <T> Map<String, T> queryConfigurableMapByType(String type) {
+ return configureService.queryConfigurableMapByType(type);
+ }
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public <T> T queryConfigurable(String configurableType, String name) {
+ return (T)queryConfigurableByIdent(configurableType, name);
+ }
+
+ //protected void insertConfigurable(JSONObject message, IConfigurable configurable) {
+ // ConfigurableUtil.insertConfigurable(message, configurable, this.configureService);
+ //}
+
+ @Override
+ public String getNamespace() {
+ if (AbstractConfigurableService.class.isInstance(configureService)) {
+ return ((AbstractConfigurableService)configureService).getNamespace();
+ }
+ return namespace;
+ }
+
+ @Override
+ public Collection<IConfigurable> findAll() {
+ return configureService.findAll();
+ }
+
+ @Override
+ public <T extends IConfigurable> List<T> loadConfigurableFromStorage(String type) {
+ return configureService.loadConfigurableFromStorage(type);
+ }
+}
diff --git a/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configurable/model/Configure.java b/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configurable/model/Configure.java
new file mode 100644
index 0000000..bb4531f
--- /dev/null
+++ b/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configurable/model/Configure.java
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.configurable.model;
+
+import org.apache.rocketmq.streams.common.model.Entity;
+
+/**
+ * configuable如果存储在db，这个是db表的映射对象
+ */
+public class Configure extends Entity {
+
+ private static final long serialVersionUID = 5668017348345235669L;
+
+ private String nameSpace;
+ private String type;
+ private String name;
+ // private String identification;
+ private String jsonValue;
+ private String modifyTime;
+ private String remark;
+ private int openRange;
+
+ public static String createTableSQL(String tableName) {
+ return "/******************************************/\n"
+ + "/* TableName = dipper_configure */\n"
+ + "/******************************************/\n"
+ + "CREATE TABLE IF NOT EXISTS `" + tableName + "` (\n"
+ + " `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT COMMENT '主键',\n"
+ + " `gmt_create` datetime NOT NULL COMMENT '创建时间',\n"
+ + " `gmt_modified` datetime NOT NULL COMMENT '修改时间',\n"
+ + " `namespace` varchar(32) NOT NULL COMMENT '项目标识',\n"
+ + " `type` varchar(32) NOT NULL COMMENT '配置类型',\n"
+ + " `name` varchar(128) NOT NULL COMMENT '配置名称',\n"
+ + " `json_value` text NOT NULL COMMENT '配置内容',\n"
+ + " `status` tinyint(3) unsigned NOT NULL DEFAULT '0' COMMENT '1:正在使用 0:已失效',\n"
+ + " PRIMARY KEY (`id`),\n"
+ + " UNIQUE KEY `uk_namespace_type_name` (`namespace`,`type`,`name`),\n"
+ + " KEY `idx_namespace` (`namespace`)\n"
+ + ") ENGINE=InnoDB AUTO_INCREMENT=1814834 DEFAULT CHARSET=utf8 COMMENT='统一接入配置项'\n"
+ + ";";
+ }
+
+ public String getNameSpace() {
+ return nameSpace;
+ }
+
+ public void setNameSpace(String nameSpace) {
+ this.nameSpace = nameSpace;
+ }
+
+ public String getType() {
+ return type;
+ }
+
+ public void setType(String type) {
+ this.type = type;
+ }
+
+ public String getName() {
+ return name;
+ }
+
+ public void setName(String name) {
+ this.name = name;
+ }
+
+ // public String getIdentification() {
+ // return identification;
+ // }
+
+ // public void createIdentification() {
+ // this.identification = MapKeyUtil.createKey(nameSpace, type, name);
+ // }
+
+ public String getJsonValue() {
+ return jsonValue;
+ }
+
+ public void setJsonValue(String jsonValue) {
+ this.jsonValue = jsonValue;
+ }
+
+ public String getModifyTime() {
+ return modifyTime;
+ }
+
+ public void setModifyTime(String modifyTime) {
+ this.modifyTime = modifyTime;
+ }
+
+ public String getRemark() {
+ return remark;
+ }
+
+ public void setRemark(String remark) {
+ this.remark = remark;
+ }
+
+ public int getOpenRange() {
+ return openRange;
+ }
+
+ public void setOpenRange(int openRange) {
+ this.openRange = openRange;
+ }
+
+ @Override
+ public String toString() {
+ return "Configure{" + "nameSpace='" + nameSpace + '\'' + ", type='" + type + '\'' + ", name='" + name + '\''
+ + ", jsonValue='" + jsonValue + '\'' + ", modifyTime='" + modifyTime + '\'' + ", remark='" + remark + '\''
+ + ", openRange=" + openRange + '}';
+ }
+}
diff --git a/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configurable/service/AbstractConfigurableService.java b/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configurable/service/AbstractConfigurableService.java
new file mode 100644
index 0000000..85ef626
--- /dev/null
+++ b/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configurable/service/AbstractConfigurableService.java
@@ -0,0 +1,553 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.configurable.service;
+
+import com.alibaba.fastjson.JSONObject;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.rocketmq.streams.common.component.AbstractComponent;
+import org.apache.rocketmq.streams.common.configurable.AbstractConfigurable;
+import org.apache.rocketmq.streams.common.configurable.IAfterConfiguableRefreshListerner;
+import org.apache.rocketmq.streams.common.configurable.IConfigurable;
+import org.apache.rocketmq.streams.common.configurable.IConfigurableService;
+import org.apache.rocketmq.streams.common.datatype.DataType;
+import org.apache.rocketmq.streams.common.model.Entity;
+import org.apache.rocketmq.streams.common.utils.*;
+import org.apache.rocketmq.streams.configurable.model.Configure;
+
+import java.util.*;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+public abstract class AbstractConfigurableService implements IConfigurableService {
+
+ private static final Log LOG = LogFactory.getLog(AbstractConfigurableService.class);
+
+ private static final String CLASS_NAME = IConfigurableService.CLASS_NAME;
+
+ protected Map<String, List<IConfigurable>> type2ConfigurableMap = new HashMap<>();
+
+ protected Map<String, IConfigurable> name2ConfigurableMap = new HashMap<>();
+
+ protected Map<String, IConfigurable> configurableMap = new HashMap<>();
+
+ protected Properties properties;
+
+ protected transient String namespace;
+
+ public AbstractConfigurableService(Properties properties) {
+ this.properties = properties;
+ }
+
+ public AbstractConfigurableService() {
+ }
+
+ @Override
+ public IConfigurable queryConfigurableByIdent(String identification) {
+ return name2ConfigurableMap.get(identification);
+ }
+
+ protected String getConfigureKey(String nameSpace, String type, String name) {
+ return MapKeyUtil.createKey(nameSpace, type, name);
+ }
+
+ protected void updateConfiguresCache(IConfigurable configurable) {
+ if (configurable == null) {
+ return;
+ }
+ configurable.toJson();
+ String key = getConfigureKey(configurable.getNameSpace(), configurable.getType(), configurable.getConfigureName());
+ configurableMap.put(key, configurable);
+ }
+
+ protected void updateConfiguresCache(List<IConfigurable> configureList) {
+ for (IConfigurable iConfigurable : configureList) {
+ updateConfiguresCache(iConfigurable);
+ }
+ }
+
+ protected boolean equals(String key, List<?> newConfigureList) {
+ for (Object o : newConfigureList) {
+ IConfigurable configure = (IConfigurable)o;
+ String tempKey = getConfigureKey(configure.getNameSpace(), configure.getType(), configure.getConfigureName());
+ if (key.equals(tempKey)) {
+ IConfigurable oldConfigure = configurableMap.get(key);
+ if (oldConfigure == null) {
+ continue;
+ }
+ return ConfigurableUtil.compare(oldConfigure, configure);
+ }
+ }
+ return false;
+ }
+
+ @Override
+ public <T extends IConfigurable> List<T> queryConfigurableByType(String type) {
+ List<IConfigurable> list = queryConfigurable(type);
+ if (list == null) {
+ return new ArrayList<T>();
+ }
+ List<T> result = new ArrayList<T>();
+ for (IConfigurable configurable : list) {
+ result.add((T)configurable);
+ }
+ return result;
+ }
+
+ @Override
+ public boolean refreshConfigurable(String namespace) {
+ //每次刷新，重新刷新配置文件
+ //if(ComponentCreator.propertiesPath!=null){
+ // ComponentCreator.setProperties(ComponentCreator.propertiesPath);
+ //}
+ this.namespace = namespace;
+ // Map<String, List<IConfigurable>> namespace2ConfigurableMap = new HashMap<>();
+ Map<String, List<IConfigurable>> tempType2ConfigurableMap = new HashMap<>();
+ Map<String, IConfigurable> tempName2ConfigurableMap = new HashMap<>();
+ GetConfigureResult configures = loadConfigurable(namespace);
+ // updateConfiguresCache(configures.getConfigure());
+ if (configures != null && configures.isQuerySuccess() && configures.getConfigurables() != null) {
+ // List<Configure> configureList = filterConfigure(configures.getConfigure());
+ List<IConfigurable> configurables = configures.getConfigurables();
+ List<IConfigurable> configurableList = checkAndUpdateConfigurables(namespace, configurables,
+ tempType2ConfigurableMap, tempName2ConfigurableMap,
+ configures.getConfigurables());
+ // this.namespace2ConfigurableMap = namespace2ConfigurableMap;
+ for (IConfigurable configurable : configurableList) {
+ if (configurable instanceof IAfterConfiguableRefreshListerner) {
+ ((IAfterConfiguableRefreshListerner)configurable).doProcessAfterRefreshConfigurable(this);
+ }
+ }
+ return true;
+ }
+ return false;
+ }
+
+ @Override
+ public <T> T queryConfigurable(String configurableType, String name) {
+ return (T)queryConfigurableByIdent(configurableType, name);
+ }
+
+ protected List<IConfigurable> checkAndUpdateConfigurables(String namespace, List<IConfigurable> configurables,
+ Map<String, List<IConfigurable>> tempType2ConfigurableMap,
+ Map<String, IConfigurable> tempName2ConfigurableMap,
+ List configures) {
+ List<IConfigurable> configurableList = new ArrayList<>();
+ for (IConfigurable configurable : configurables) {
+ try {
+ boolean isUpdate = update(configurable, tempName2ConfigurableMap, tempType2ConfigurableMap);
+ if (isUpdate) {
+ configurableList.add(configurable);
+ }
+ } catch (Exception e) {
+ LOG.error("组件初始化异常：" + e.getMessage() + ",name=" + configurable.getConfigureName(), e);
+ }
+ }
+ destroyOldConfigurables(tempName2ConfigurableMap);
+ this.name2ConfigurableMap = tempName2ConfigurableMap;
+ this.type2ConfigurableMap = tempType2ConfigurableMap;
+ return configurableList;
+ }
+
+ private void destroyOldConfigurables(Map<String, IConfigurable> tempName2ConfigurableMap) {
+ Iterator<Map.Entry<String, IConfigurable>> it = this.name2ConfigurableMap.entrySet().iterator();
+ while (it.hasNext()) {
+ Map.Entry<String, IConfigurable> entry = it.next();
+ String key = entry.getKey();
+ IConfigurable value = entry.getValue();
+ if (!tempName2ConfigurableMap.containsKey(key)) {
+ destroyOldConfigurable(value);
+ }
+ }
+
+ }
+
+ private void destroyOldConfigurable(IConfigurable oldConfigurable) {
+ if (AbstractConfigurable.class.isInstance(oldConfigurable)) {
+ ((AbstractConfigurable)oldConfigurable).destroy();
+ }
+ String key = getConfigureKey(oldConfigurable.getNameSpace(), oldConfigurable.getType(),
+ oldConfigurable.getConfigureName());
+ configurableMap.remove(key);
+ }
+
+ protected void initConfigurable(IConfigurable configurable) {
+ if (AbstractConfigurable.class.isInstance(configurable)) {
+ AbstractConfigurable abstractConfigurable = (AbstractConfigurable)configurable;
+ abstractConfigurable.setConfigurableService(this);
+ }
+
+ configurable.init();
+
+ }
+
+ /**
+ * 内部使用
+ */
+ private ScheduledExecutorService scheduledExecutorService;
+
+ @Override
+ public void initConfigurables(final String namespace) {
+ refreshConfigurable(namespace);
+ long polingTime = -1;
+ if (this.properties != null) {
+ String pollingTimeStr = this.properties.getProperty(AbstractComponent.POLLING_TIME);
+ if (StringUtil.isNotEmpty(pollingTimeStr)) {
+ polingTime = Long.valueOf(pollingTimeStr);
+ }
+ }
+ if (polingTime > 0) {
+ scheduledExecutorService = new ScheduledThreadPoolExecutor(3);
+ scheduledExecutorService.scheduleWithFixedDelay(new Runnable() {
+
+ @Override
+ public void run() {
+ try {
+ refreshConfigurable(namespace);
+ } catch (Exception e) {
+ LOG.error("Load configurables error:" + e.getMessage(), e);
+ }
+ }
+ }, polingTime, polingTime, TimeUnit.SECONDS);
+ }
+ }
+ // @Override
+ // public List<IConfigurable> queryConfigurable(String nameSpace) {
+ // return namespace2ConfigurableMap.get(nameSpace);
+ // }
+
+ @Override
+ public List<IConfigurable> queryConfigurable(String type) {
+ String key = MapKeyUtil.createKey(type);
+ return type2ConfigurableMap.get(key);
+ }
+
+ @Override
+ public IConfigurable queryConfigurableByIdent(String type, String name) {
+ String key = MapKeyUtil.createKey(type, name);
+ return name2ConfigurableMap.get(key);
+ }
+
+ /**
+ * 根据namespace加载配置信息
+ *
+ * @param namespace
+ * @return
+ * @throws Exception
+ */
+ protected abstract GetConfigureResult loadConfigurable(String namespace);
+
+ @Override
+ public void update(IConfigurable configurable) {
+ // update(configurable,name2ConfigurableMap,type2ConfigurableMap);
+ updateConfigurable(configurable);
+ }
+
+ protected abstract void updateConfigurable(IConfigurable configurable);
+
+ protected abstract void insertConfigurable(IConfigurable configurable);
+
+ protected boolean update(IConfigurable configurable, Map<String, IConfigurable> name2ConfigurableMap,
+ Map<String, List<IConfigurable>> type2ConfigurableMap) {
+ if (configurable == null) {
+ return false;
+ }
+
+ boolean isUpdate = false;
+ List<IConfigurable> configurableList = new ArrayList<>();
+ configurableList.add(configurable);
+
+ String nameKey = MapKeyUtil.createKey(configurable.getType(), configurable.getConfigureName());
+ if (this.name2ConfigurableMap.containsKey(nameKey)) {
+ String configureKey = getConfigureKey(namespace, configurable.getType(), configurable.getConfigureName());
+ IConfigurable oldConfigurable = this.name2ConfigurableMap.get(nameKey);
+ if (equals(configureKey, configurableList)) {
+ configurable = oldConfigurable;
+ // name2ConfigurableMap.put(nameKey, name2ConfigurableMap.get(nameKey));
+ } else {
+ destroyOldConfigurable(oldConfigurable);
+ initConfigurable(configurable);
+ isUpdate = true;
+ }
+ } else {
+ initConfigurable(configurable);
+ isUpdate = true;
+ }
+ updateConfiguresCache(configurable);
+ name2ConfigurableMap.put(nameKey, configurable);
+ String typeKey = MapKeyUtil.createKey(configurable.getType());
+ // put2Map(namespace2ConfigurableMap, namespace, configurable);
+ put2Map(type2ConfigurableMap, typeKey, configurable);
+ return isUpdate;
+ }
+
+ @Override
+ public void insert(IConfigurable configurable) {
+ // update(configurable,name2ConfigurableMap,type2ConfigurableMap);
+ insertConfigurable(configurable);
+ }
+
+ /**
+ * 给一个扣，可以跨命名空间查询数据
+ *
+ * @param namespaces
+ * @return
+ */
+ public List<IConfigurable> queryConfiguableByNamespace(String... namespaces) {
+ List<IConfigurable> configurables = new ArrayList<>();
+ if (namespaces == null || namespaces.length == 0) {
+ return configurables;
+ }
+ for (String namespace : namespaces) {
+ GetConfigureResult result = loadConfigurable(namespace);
+ if (result.querySuccess) {
+ if (result.configurables != null && result.configurables.size() > 0) {
+ configurables.addAll(result.configurables);
+ }
+ } else {
+ throw new RuntimeException("Load configurable error, the namespace is " + namespace);
+ }
+ }
+ return configurables;
+
+ }
+
+ /**
+ * 往一个value是list的map中添加数据，如果list是空创建，否则直接插入
+ *
+ * @param map
+ * @param key
+ * @param configurable
+ */
+ protected void put2Map(Map<String, List<IConfigurable>> map, String key, IConfigurable configurable) {
+ List<IConfigurable> list = map.get(key);
+ if (list == null) {
+ list = new ArrayList<IConfigurable>();
+ map.put(key, list);
+ }
+ list.add(configurable);
+ }
+
+ @Override
+ public Collection<IConfigurable> findAll() {
+ return name2ConfigurableMap.values();
+ }
+
+ /**
+ * 把configurable转换成configure
+ *
+ * @param configurable
+ * @return
+ */
+ protected Configure createConfigure(IConfigurable configurable) {
+ Configure configure = new Configure();
+ configure.setType(configurable.getType());
+ configure.setName(configurable.getConfigureName());
+ configure.setNameSpace(configurable.getNameSpace());
+ String jsonString = configurable.toJson();
+ if (!StringUtil.isEmpty(jsonString)) {
+ JSONObject jsonObject = JSONObject.parseObject(jsonString);
+ jsonObject.put(CLASS_NAME, configurable.getClass().getName());
+ configure.setJsonValue(jsonObject.toJSONString());
+ }
+ // configure.createIdentification();
+ return configure;
+ }
+
+ @Override
+ public <T> Map<String, T> queryConfigurableMapByType(String type) {
+ List<IConfigurable> configurables = queryConfigurable(type);
+ if (configurables == null) {
+ return new HashMap<String, T>();
+ }
+ Map<String, T> result = new HashMap<String, T>();
+ for (IConfigurable configurable : configurables) {
+ result.put(configurable.getConfigureName(), (T)configurable);
+ }
+ return result;
+ }
+
+ /**
+ * 把configure转换成configurable
+ *
+ * @param configures
+ * @return
+ */
+ protected List<IConfigurable> convert(List<Configure> configures) {
+ if (configures == null) {
+ return new ArrayList<IConfigurable>();
+ }
+ List<IConfigurable> configurables = new ArrayList<IConfigurable>();
+ for (Configure configure : configures) {
+ IConfigurable configurable = convert(configure);
+ if (configurable != null) {
+ configurables.add(configurable);
+ }
+
+ }
+ return configurables;
+ }
+
+ protected IConfigurable createConfigurableFromJson(String namespace, String type, String name, String jsonValue) {
+ if (StringUtil.isEmpty(jsonValue)) {
+ return null;
+ }
+ JSONObject jsonObject = JSONObject.parseObject(jsonValue);
+ String className = jsonObject.getString(CLASS_NAME);
+ IConfigurable configurable = createConfigurable(className);
+ if (configurable == null) {
+ return null;
+ }
+ configurable.setConfigureName(name);
+ configurable.setNameSpace(namespace);
+ configurable.setType(type);
+ if (AbstractConfigurable.class.isInstance(configurable)) {
+ AbstractConfigurable abstractConfigurable = (AbstractConfigurable)configurable;
+ abstractConfigurable.setConfigurableService(this);
+ }
+ configurable.toObject(jsonValue);
+ return configurable;
+ }
+
+ /**
+ * 提供一个入口，可以让外部用户改变configure对应的configurable的值
+ *
+ * @param configure
+ * @return
+ */
+ protected IConfigurable convert(Configure configure) {
+
+ return convertConfigurable(configure);
+ }
+
+ protected IConfigurable convertConfigurable(Configure configure) {
+ String className = null;
+ try {
+ String jsonString = configure.getJsonValue();
+ IConfigurable configurable =
+ createConfigurableFromJson(configure.getNameSpace(), configure.getType(), configure.getName(),
+ jsonString);
+ if (configurable instanceof Entity) {
+ // add by wangtl 20171110 Configurable接口第三方包也在用，故不能Configurable里加接口，只能加到抽象类里，这里强转下
+ Entity abs = (Entity)configurable;
+ abs.setId(configure.getId());
+ abs.setGmtCreate(configure.getGmtCreate());
+ abs.setGmtModified(configure.getGmtModified());
+ /*
+ * abs.setTempKey((configurable.getNameSpace() + configurable.getType() +
+ * configurable.getConfigureName() + jsonString).hashCode());
+ */
+ }
+ convertPost(configurable);
+ return configurable;
+ } catch (Exception e) {
+ LOG.error("转换异常：" + configure.toString(), e);
+ return null;
+ }
+ }
+
+ /**
+ * 如果需要改变configurable的属性，可以再这里设置
+ *
+ * @param configurable
+ */
+ @SuppressWarnings("rawtypes")
+ protected void convertPost(IConfigurable configurable) {
+ if (this.properties == null) {
+ return;
+ }
+ String identification =
+ MapKeyUtil.createKey(configurable.getNameSpace(), configurable.getType(), configurable.getConfigureName());
+ String propertyValue = this.properties.getProperty(identification);
+ if (StringUtil.isEmpty(propertyValue)) {
+ return;
+ }
+ String[] fieldName2Values = propertyValue.split(",");
+ if (fieldName2Values == null || fieldName2Values.length == 0) {
+ return;
+ }
+ for (String fieldName2Value : fieldName2Values) {
+ try {
+ String[] fieldName2ValueArray = fieldName2Value.split(":");
+ if (fieldName2ValueArray == null || fieldName2ValueArray.length != 2) {
+ continue;
+ }
+ String fieldName = fieldName2ValueArray[0];
+ String value = fieldName2ValueArray[1];
+ Class clazz = ReflectUtil.getBeanFieldType(configurable.getClass(), fieldName);
+ DataType dataType = DataTypeUtil.createDataType(clazz, null);
+ if (dataType == null) {
+ continue;
+ }
+ Object fieldValue = dataType.getData(value);
+ ReflectUtil.setBeanFieldValue(configurable, fieldName, fieldValue);
+
+ } catch (Exception e) {
+ LOG.error("convert post error " + fieldName2Value, e);
+ continue;
+ }
+
+ }
+ }
+
+ /**
+ * 创建configurable对象
+ *
+ * @param className class name
+ * @return
+ */
+ @SuppressWarnings("rawtypes")
+ protected IConfigurable createConfigurable(String className) {
+ return ReflectUtil.forInstance(className);
+ }
+
+ public class GetConfigureResult {
+
+ private boolean querySuccess;
+ private List<IConfigurable> configurables;
+
+ public boolean isQuerySuccess() {
+ return querySuccess;
+ }
+
+ public void setQuerySuccess(boolean querySuccess) {
+ this.querySuccess = querySuccess;
+ }
+
+ public List<IConfigurable> getConfigurables() {
+ return configurables;
+ }
+
+ public void setConfigurables(List<IConfigurable> configurables) {
+ this.configurables = configurables;
+ }
+ }
+
+ @Override
+ public String getNamespace() {
+ return namespace;
+ }
+
+ public Properties getProperties() {
+ return properties;
+ }
+
+ public void setProperties(Properties properties) {
+ this.properties = properties;
+ }
+}
diff --git a/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configurable/service/AbstractSupportParentConfigureService.java b/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configurable/service/AbstractSupportParentConfigureService.java
new file mode 100644
index 0000000..d2d26cd
--- /dev/null
+++ b/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configurable/service/AbstractSupportParentConfigureService.java
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.configurable.service;
+
+import org.apache.rocketmq.streams.common.configurable.IConfigurable;
+import org.apache.rocketmq.streams.common.configurable.IConfigurableService;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.Properties;
+
+/**
+ * namespace 分层，支持顶级命名空间，顶级命名空间的对象，所有命名空间都可见。顶级命名空间是固定值IConfigurableService.PARENT_CHANNEL_NAME_SPACE
+ */
+public abstract class AbstractSupportParentConfigureService extends AbstractConfigurableService
+ implements IConfigurableService {
+
+ private static final Log LOG = LogFactory.getLog(AbstractSupportParentConfigureService.class);
+ protected IConfigurableService configureService = null;
+ protected IConfigurableService parentConfigureService = null;
+ //protected IConfigurableService shareConfigureService = null;
+ protected Properties properties;
+
+ public AbstractSupportParentConfigureService() {
+ super(null);
+ }
+
+ public void initMethod(Properties property) {
+ this.properties = property;
+ initBeforeInitConfigurable(property);
+ }
+
+ protected abstract void initBeforeInitConfigurable(Properties property);
+
+ @Override
+ public void initConfigurables(String namespace) {
+
+ if (!IConfigurableService.PARENT_CHANNEL_NAME_SPACE.equals(namespace)) {
+ parentConfigureService.initConfigurables(IConfigurableService.PARENT_CHANNEL_NAME_SPACE);
+ } else {
+ parentConfigureService = null;
+ }
+ configureService.initConfigurables(namespace);
+ }
+
+ @Override
+ public boolean refreshConfigurable(String namespace) {
+
+ if (!IConfigurableService.PARENT_CHANNEL_NAME_SPACE.equals(namespace)) {
+ parentConfigureService.refreshConfigurable(IConfigurableService.PARENT_CHANNEL_NAME_SPACE);
+ // initShareConfigurableService(namespace);
+ }
+ configureService.refreshConfigurable(namespace);
+ return true;
+ }
+
+ @Override
+ public List<IConfigurable> queryConfigurable(String type) {
+ List<IConfigurable> result = configureService.queryConfigurable(type);
+ if (result == null) {
+ result = new ArrayList<>();
+ }
+ //if (shareConfigureService != null) {
+ // List<IConfigurable> share = shareConfigureService.queryConfigurable(type);
+ // if (share != null) {
+ // result.addAll(share);
+ // }
+ //}
+ if (parentConfigureService == null) {
+ return result;
+ }
+ List<IConfigurable> parent = parentConfigureService.queryConfigurable(type);
+ if (parent != null) {
+ result.addAll(parent);
+ }
+ return result;
+ }
+
+ @Override
+ public IConfigurable queryConfigurableByIdent(String type, String name) {
+ IConfigurable configurable = configureService.queryConfigurableByIdent(type, name);
+ if (configurable != null) {
+ return configurable;
+ }
+ if (parentConfigureService == null) {
+ return null;
+ }
+ //if (shareConfigureService != null) {
+ // configurable = shareConfigureService.queryConfigurableByIdent(type, name);
+ //}
+ if (configurable != null) {
+ return configurable;
+ }
+ return parentConfigureService.queryConfigurableByIdent(type, name);
+ }
+
+ @Override
+ public IConfigurable queryConfigurableByIdent(String identification) {
+ IConfigurable configurable = configureService.queryConfigurableByIdent(identification);
+ if (configurable != null) {
+ return configurable;
+ }
+ if (parentConfigureService == null) {
+ return null;
+ }
+ //if (shareConfigureService != null) {
+ // configurable = shareConfigureService.queryConfigurableByIdent(identification);
+ //}
+ if (configurable != null) {
+ return configurable;
+ }
+ return parentConfigureService.queryConfigurableByIdent(identification);
+ }
+
+ @Override
+ protected void insertConfigurable(IConfigurable configurable) {
+ if (parentConfigureService != null && configurable.getNameSpace()
+ .equals(IConfigurableService.PARENT_CHANNEL_NAME_SPACE)) {
+ parentConfigureService.insert(configurable);
+ } else {
+ configureService.insert(configurable);
+ }
+ }
+
+ @Override
+ protected void updateConfigurable(IConfigurable configurable) {
+ if (parentConfigureService != null && configurable.getNameSpace()
+ .equals(IConfigurableService.PARENT_CHANNEL_NAME_SPACE)) {
+ parentConfigureService.update(configurable);
+ } else {
+ configureService.update(configurable);
+ }
+ }
+
+ @Override
+ public <T> T queryConfigurable(String configurableType, String name) {
+ return (T)queryConfigurableByIdent(configurableType, name);
+ }
+
+ @Override
+ protected GetConfigureResult loadConfigurable(String namespace) {
+ return null;
+ }
+
+ //protected void initShareConfigurableService(String namespace) {
+ // if (parentConfigureService == null) {
+ // return;
+ // }
+ // shareConfigureService = new AbstractReadOnlyConfigurableService() {
+ //
+ // @Override
+ // public <T extends IConfigurable> List<T> loadConfigurableFromStorage(String type) {
+ // refreshConfigurable(namespace);
+ // return queryConfigurableByType(type);
+ // }
+ //
+ // @Override
+ // protected List<IConfigurable> loadConfigurables(String namespace) {
+ // List<IConfigurable> parent = parentConfigureService.queryConfigurable(ShareConfiguable.TYPE);
+ // List<IConfigurable> shareConfigurables = new ArrayList<>();
+ // if (parent == null) {
+ // return shareConfigurables;
+ // }
+ // for (IConfigurable configurable : parent) {
+ // ShareConfiguable shareConfiguable = (ShareConfiguable) configurable;
+ // if (shareConfiguable.getShareAll() || shareConfiguable.getShareNameSpaces().contains(namespace)) {
+ // String sharedNameSpace = shareConfiguable.getSharedNameSpace();
+ // String sharedType = shareConfiguable.getSharedType();
+ // String sharedName = shareConfiguable.getSharedName();
+ // List<IConfigurable> sharedConfigrables =
+ // createAndQueryConfigurable(sharedNameSpace, sharedType, sharedName);
+ // if (sharedConfigrables != null) {
+ // shareConfigurables.addAll(sharedConfigrables);
+ // }
+ // }
+ // }
+ // return shareConfigurables;
+ // }
+ //
+ //
+ // };
+ // shareConfigureService.refreshConfigurable(namespace);
+ //
+ //}
+
+ protected List<IConfigurable> createAndQueryConfigurable(String sharedNameSpace, String sharedType,
+ String sharedName) {
+ IConfigurableService innerSharedConfigurableService =
+ ConfigurableServiceFactory.createConfigurableService(properties);
+ innerSharedConfigurableService.refreshConfigurable(sharedNameSpace);
+ if (StringUtil.isNotEmpty(sharedName)) {
+ List<IConfigurable> configurables = new ArrayList<>();
+ IConfigurable configurable = innerSharedConfigurableService.queryConfigurableByIdent(sharedType, sharedName);
+ configurables.add(configurable);
+ return configurables;
+ } else {
+ return innerSharedConfigurableService.queryConfigurable(sharedType);
+ }
+
+ }
+
+ @Override
+ public Collection<IConfigurable> findAll() {
+ List<IConfigurable> configurables = new ArrayList<>();
+ if (parentConfigureService != null) {
+ Collection<IConfigurable> tmp = parentConfigureService.findAll();
+ if (tmp != null || tmp.size() > 0) {
+ configurables.addAll(tmp);
+ }
+ }
+ Collection<IConfigurable> tmp = configureService.findAll();
+ if (tmp != null || tmp.size() > 0) {
+ configurables.addAll(tmp);
+ }
+ return configurables;
+ }
+
+ public IConfigurableService getConfigureService() {
+ return configureService;
+ }
+
+ @Override
+ public <T extends IConfigurable> List<T> loadConfigurableFromStorage(String type) {
+ List<T> configurables = new ArrayList<>();
+ if (parentConfigureService != null) {
+ Collection<T> tmp = parentConfigureService.loadConfigurableFromStorage(type);
+ if (tmp != null || tmp.size() > 0) {
+ configurables.addAll(tmp);
+ }
+ }
+ Collection<T> tmp = configureService.loadConfigurableFromStorage(type);
+ if (tmp != null || tmp.size() > 0) {
+ configurables.addAll(tmp);
+ }
+ return configurables;
+ }
+}
diff --git a/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configurable/service/ConfigurableServcieType.java b/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configurable/service/ConfigurableServcieType.java
new file mode 100644
index 0000000..a4998ff
--- /dev/null
+++ b/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configurable/service/ConfigurableServcieType.java
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.configurable.service;
+
+import org.apache.rocketmq.streams.common.configurable.IConfigurableService;
+
+@Deprecated
+public class ConfigurableServcieType {
+
+ public static final String DEFAULT_SERVICE_NAME = IConfigurableService.DEFAULT_SERVICE_NAME;
+ public static final String MEMORY_SERVICE_NAME = IConfigurableService.MEMORY_SERVICE_NAME;
+ public static final String FILE_SERVICE_NAME = IConfigurableService.FILE_SERVICE_NAME;
+ public static final String HTTP_SERVICE_NAME = IConfigurableService.HTTP_SERVICE_NAME;
+
+}
diff --git a/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configurable/service/ConfigurableServiceFactory.java b/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configurable/service/ConfigurableServiceFactory.java
new file mode 100644
index 0000000..449f04a
--- /dev/null
+++ b/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configurable/service/ConfigurableServiceFactory.java
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.configurable.service;
+
+import org.apache.rocketmq.streams.common.configurable.IConfigurableService;
+import org.apache.rocketmq.streams.common.utils.ReflectUtil;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.rocketmq.streams.serviceloader.ServiceLoaderComponent;
+
+import java.util.Properties;
+
+public class ConfigurableServiceFactory {
+ private static ServiceLoaderComponent<IConfigurableService> configurableServiceLoaderComponent =
+ ServiceLoaderComponent.getInstance(IConfigurableService.class);
+ public static final String CONFIGURABLE_SERVICE_TYPE = "dipper.configurable.service.type";
+ private static final Log LOG = LogFactory.getLog(ConfigurableServiceFactory.class);
+
+ public static IConfigurableService createConfigurableService(Properties properties) {
+ try {
+ Properties properties1 = new Properties();
+ properties1.putAll(properties);
+ String type = properties1.getProperty(CONFIGURABLE_SERVICE_TYPE);
+ if (StringUtil.isEmpty(type)) {
+ type = IConfigurableService.DEFAULT_SERVICE_NAME;
+ ;
+ }
+ IConfigurableService configurableService = getConfigurableServcieType(type);
+ if (AbstractSupportParentConfigureService.class.isInstance(configurableService)) {
+ ((AbstractSupportParentConfigureService)configurableService).initMethod(properties1);
+ }
+ return configurableService;
+ } catch (Exception e) {
+ LOG.error("create ConfigurableService error", e);
+ return null;
+ }
+
+ }
+
+ public static IConfigurableService getConfigurableServcieType(String type) {
+ IConfigurableService configurableService = (IConfigurableService)configurableServiceLoaderComponent.getService().loadService(type);
+ return ReflectUtil.forInstance(configurableService.getClass().getName());
+ }
+}
diff --git a/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configurable/service/impl/FileConfigureService.java b/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configurable/service/impl/FileConfigureService.java
new file mode 100644
index 0000000..1a5fe9b
--- /dev/null
+++ b/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configurable/service/impl/FileConfigureService.java
@@ -0,0 +1,249 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.configurable.service.impl;
+
+import org.apache.rocketmq.streams.common.configurable.IConfigurable;
+import org.apache.rocketmq.streams.common.configurable.IConfigurableService;
+import org.apache.rocketmq.streams.configurable.service.AbstractConfigurableService;
+import org.apache.rocketmq.streams.configurable.model.Configure;
+import org.apache.rocketmq.streams.common.model.Entity;
+import org.apache.rocketmq.streams.common.utils.DateUtil;
+import org.apache.rocketmq.streams.common.utils.FileUtil;
+import org.apache.rocketmq.streams.common.utils.MapKeyUtil;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Properties;
+
+public class FileConfigureService extends AbstractConfigurableService {
+
+ public static final String FILE_PATH_NAME = IConfigurableService.FILE_PATH_NAME;
+ // 配置文件的路径
+ private static final Log LOG = LogFactory.getLog(FileConfigureService.class);
+ private static final String DEFAULT_FILE_NAME = "dipper_configure.cs"; // 默认文件名
+ private static final String SIGN = "&&&&"; // 字段分割附号
+ public String fileName;
+
+ public FileConfigureService(Properties properties) {
+ super(properties);
+ initService(properties.getProperty(FILE_PATH_NAME));
+ }
+
+ protected void initService(String fileAndPath) {
+ if (StringUtil.isEmpty(fileAndPath)) {
+ String path = this.getClass().getProtectionDomain().getCodeSource().getLocation().getPath();
+ if (path.endsWith(".jar")) {
+ int index = path.lastIndexOf(File.separator);
+ path = path.substring(0, index);
+ }
+ fileName = FileUtil.concatFilePath(path, DEFAULT_FILE_NAME);
+ } else {
+ fileName = fileAndPath;
+ }
+ LOG.info("load file from path = " + fileName);
+ }
+
+ @Override
+ protected GetConfigureResult loadConfigurable(String namespace) {
+ GetConfigureResult result = new GetConfigureResult();
+ try {
+ List<Configure> configures = selectOpening(namespace);
+ List<IConfigurable> configurables = convert(configures);
+ LOG.info("load configure namespace=" + namespace + " count=" + configures.size());
+ result.setConfigurables(configurables);
+ result.setQuerySuccess(true);// 该字段标示查询是否成功，若不成功则不会更新配置
+ } catch (Exception e) {
+ result.setQuerySuccess(false);
+ e.printStackTrace();
+ LOG.error("load configurable error ", e);
+ }
+ return result;
+ }
+
+ protected List<Configure> selectOpening(String namespace) {
+ List<String> list = loadFileLine(fileName);
+ List<Configure> configures = convert2Configure(list);
+ return filter(configures, namespace);
+ }
+
+ protected List<Configure> filter(List<Configure> configures, String namespace) {
+ if (configures == null) {
+ return new ArrayList<>();
+ }
+ if (StringUtil.isEmpty(namespace)) {
+ throw new RuntimeException("namespace can not empty ");
+ }
+ List<Configure> filterConfigures = new ArrayList<>();
+ for (Configure configure : configures) {
+ if (!namespace.equals(configure.getNameSpace())) {
+ continue;
+ }
+ filterConfigures.add(configure);
+ }
+ return filterConfigures;
+ }
+
+ @Override
+ protected void insertConfigurable(IConfigurable configure) {
+ if (configure == null) {
+ LOG.warn("insert configure is null");
+ return;
+ }
+ String row = configure2String(configure);
+
+ List<String> rows = loadFileLine(fileName);
+ if (rows == null) {
+ rows = new ArrayList<>();
+ }
+ List<Configure> configures = convert2Configure(rows);
+ String newKey =
+ MapKeyUtil.createKey(configure.getNameSpace(), configure.getType(), configure.getConfigureName());
+ boolean isReplace = false;
+ for (int i = 0; i < configures.size(); i++) {
+ Configure c = configures.get(i);
+ String old = MapKeyUtil.createKey(c.getNameSpace(), c.getType(), c.getName());
+ if (old.equals(newKey)) {
+ rows.set(i, configure2String(configure));
+ isReplace = true;
+ break;
+ }
+ }
+ if (!isReplace) {
+ rows.add(configure2String(configure));
+ }
+ writeFile(fileName, rows);
+ }
+
+ @Override
+ protected void updateConfigurable(IConfigurable configure) {
+ if (configure == null) {
+ LOG.warn("insert configure is null");
+ return;
+ }
+
+ List<String> rows = FileUtil.loadFileLine(fileName);
+ if (rows == null) {
+ rows = new ArrayList<>();
+ }
+ for (int i = 0; i < rows.size(); i++) {
+ String row = rows.get(i);
+ Configure oldConfigure = convert(row);
+ if (configure.getNameSpace().equals(oldConfigure.getNameSpace()) && configure.getType()
+ .equals(oldConfigure.getType()) && configure.getConfigureName().equals(oldConfigure.getName())) {
+ rows.set(i, configure2String(configure));
+ }
+ }
+ writeFile(fileName, rows);
+
+ }
+
+ protected Configure convert(String row) {
+ String[] values = row.split(SIGN);
+ String namespace = getColumnValue(values, 0, "namespace");
+ String type = getColumnValue(values, 1, "type");
+ String name = getColumnValue(values, 2, "name");
+ String jsonValue = getColumnValue(values, 3, "json_value");
+ String createDate = getColumnValue(values, 4, "gmt_create");
+ String modifiedDate = getColumnValue(values, 5, "gmt_modified");
+ String id = getColumnValue(values, 6, "id");
+ Configure configure = new Configure();
+ configure.setNameSpace(namespace);
+ configure.setType(type);
+ configure.setName(name);
+ configure.setJsonValue(jsonValue);
+ configure.setGmtCreate(DateUtil.parse(createDate));
+ configure.setGmtCreate(DateUtil.parse(modifiedDate));
+ configure.setId((id == null ? null : Long.valueOf(id)));
+
+ return configure;
+ }
+
+ protected List<Configure> convert2Configure(List<String> rows) {
+ List<Configure> configures = new ArrayList<Configure>();
+ for (String row : rows) {
+ configures.add(convert(row));
+ }
+ return configures;
+ }
+
+ protected String getColumnValue(String[] values, int i, String namespace) {
+ if (values == null || values.length == 0) {
+ return null;
+ }
+ if (values.length <= i) {
+ return null;
+ }
+ if ("null".equals(values[i])) {
+ return null;
+ }
+ return values[i];
+ }
+
+ /**
+ * 解密文件，并加载到内存
+ *
+ * @param fileName
+ * @return
+ */
+ protected List<String> loadFileLine(String fileName) {
+ List<String> rows = FileUtil.loadFileLine(fileName);
+ if (rows == null) {
+ rows = new ArrayList<>();
+ }
+ return doDecRowList(rows);
+ }
+
+ protected void writeFile(String fileName, List<String> rows) {
+ List<String> rowList = doEncryptRowList(rows);
+ FileUtil.write(fileName, rowList);
+ }
+
+ private List<String> doEncryptRowList(List<String> rows) {
+ return rows;
+ }
+
+ private List<String> doDecRowList(List<String> rows) {
+ return rows;
+ }
+
+ protected String configure2String(IConfigurable configure) {
+ Entity entity = null;
+ if (configure instanceof Entity) {
+ entity = (Entity)configure;
+ } else {
+ entity = new Entity();
+ }
+ String row = MapKeyUtil.createKeyBySign(SIGN, configure.getNameSpace(), configure.getType(),
+ configure.getConfigureName(), configure.toJson(), DateUtil.format(entity.getGmtCreate()),
+ DateUtil.format(entity.getGmtModified()), entity.getId() + "");
+ return row;
+ }
+
+ public String getFileName() {
+ return fileName;
+ }
+
+ @Override
+ public <T extends IConfigurable> List<T> loadConfigurableFromStorage(String type) {
+ refreshConfigurable(getNamespace());
+ return queryConfigurableByType(type);
+ }
+}
diff --git a/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configurable/service/impl/FileSupportParentConfigureService.java b/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configurable/service/impl/FileSupportParentConfigureService.java
new file mode 100644
index 0000000..aafa805
--- /dev/null
+++ b/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configurable/service/impl/FileSupportParentConfigureService.java
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.configurable.service.impl;
+
+import org.apache.rocketmq.streams.configurable.service.AbstractSupportParentConfigureService;
+import org.apache.rocketmq.streams.common.configurable.IConfigurableService;
+import org.apache.rocketmq.streams.configurable.service.ConfigurableServcieType;
+import org.apache.rocketmq.streams.common.model.ServiceName;
+import com.google.auto.service.AutoService;
+
+import java.util.Properties;
+
+@AutoService(IConfigurableService.class)
+@ServiceName(ConfigurableServcieType.FILE_SERVICE_NAME)
+public class FileSupportParentConfigureService extends AbstractSupportParentConfigureService {
+
+ @Override
+ protected void initBeforeInitConfigurable(Properties property) {
+ this.configureService = new FileConfigureService(properties);
+ this.parentConfigureService = new FileConfigureService(properties);
+ }
+
+}
diff --git a/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configurable/service/impl/MemoryConfigureService.java b/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configurable/service/impl/MemoryConfigureService.java
new file mode 100644
index 0000000..b6d1fd9
--- /dev/null
+++ b/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configurable/service/impl/MemoryConfigureService.java
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.configurable.service.impl;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import org.apache.rocketmq.streams.common.configurable.AbstractConfigurable;
+import org.apache.rocketmq.streams.common.configurable.IConfigurable;
+import org.apache.rocketmq.streams.configurable.service.AbstractConfigurableService;
+import org.apache.rocketmq.streams.common.utils.ReflectUtil;
+
+public class MemoryConfigureService extends AbstractConfigurableService {
+
+ private static Map<String, List<IConfigurable>> namespace2Configure = new HashMap<>();
+
+ public MemoryConfigureService(Properties properties) {
+ super(properties);
+ }
+
+ @Override
+ protected GetConfigureResult loadConfigurable(String namespace) {
+ GetConfigureResult result = new GetConfigureResult();
+ result.setQuerySuccess(true);
+ List<IConfigurable> configurableList = new ArrayList<>();
+ List<IConfigurable> configurables = namespace2Configure.get(namespace);
+ if (configurables == null) {
+ configurableList = null;
+ } else {
+ List<IConfigurable> tmps = new ArrayList<>();
+ tmps.addAll(configurables);
+ for (IConfigurable configurable : tmps) {
+ IConfigurable tmp = ReflectUtil.forInstance(configurable.getClass());
+ tmp.toObject(configurable.toJson());
+ tmp.setNameSpace(configurable.getNameSpace());
+ tmp.setConfigureName(configurable.getConfigureName());
+ configurableList.add(tmp);
+ }
+ }
+ result.setConfigurables(configurableList);
+ return result;
+ }
+
+ @Override
+ protected void insertConfigurable(IConfigurable configurable) {
+ if (configurable == null) {
+ return;
+ }
+
+ String namespace = configurable.getNameSpace();
+ List<IConfigurable> list = namespace2Configure.get(namespace);
+ if (list == null) {
+ synchronized (this) {
+ list = namespace2Configure.get(namespace);
+ if (list == null) {
+ list = new ArrayList<>();
+ namespace2Configure.put(namespace, list);
+ }
+ }
+ }
+ int removeIndex = -1;
+ for (int i = 0; i < list.size(); i++) {
+ IConfigurable config = list.get(i);
+ if (config.getType().equals(configurable.getType()) && config.getConfigureName()
+ .equals(configurable.getConfigureName())) {
+ removeIndex = i;
+ }
+ }
+ if (AbstractConfigurable.class.isInstance(configurable)) {
+ ((AbstractConfigurable)configurable).setConfigurableService(this);
+ }
+ if (removeIndex != -1) {
+ list.remove(removeIndex);
+ }
+ list.add(configurable);
+ }
+
+ @Override
+ protected void updateConfigurable(IConfigurable configure) {
+ List<IConfigurable> list = namespace2Configure.get(configure.getNameSpace());
+ if (list == null || list.size() == 0) {
+ throw new RuntimeException(
+ "not have exist configure " + configure.getNameSpace() + "," + configure.getType() + ","
+ + configure.getConfigureName());
+ }
+ for (int i = 0; i < list.size(); i++) {
+ IConfigurable config = list.get(i);
+ if (config.getType().equals(configure.getType()) && config.getConfigureName()
+ .equals(configure.getConfigureName())) {
+ list.set(i, configure);
+ return;
+ }
+ }
+ throw new RuntimeException(
+ "not have exist configure " + configure.getNameSpace() + "," + configure.getType() + ","
+ + configure.getConfigureName());
+ }
+
+ @Override
+ public <T extends IConfigurable> List<T> loadConfigurableFromStorage(String type) {
+ refreshConfigurable(getNamespace());
+ return queryConfigurableByType(type);
+ }
+}
diff --git a/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configurable/service/impl/MemorySupportParentConfigureService.java b/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configurable/service/impl/MemorySupportParentConfigureService.java
new file mode 100644
index 0000000..92d2095
--- /dev/null
+++ b/rocketmq-streams-configurable/src/main/java/org/apache/rocketmq/streams/configurable/service/impl/MemorySupportParentConfigureService.java
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.configurable.service.impl;
+
+import java.util.Properties;
+
+import org.apache.rocketmq.streams.configurable.service.AbstractSupportParentConfigureService;
+import org.apache.rocketmq.streams.common.configurable.IConfigurableService;
+import org.apache.rocketmq.streams.configurable.service.ConfigurableServcieType;
+import org.apache.rocketmq.streams.common.model.ServiceName;
+import com.google.auto.service.AutoService;
+
+@AutoService(IConfigurableService.class)
+@ServiceName(ConfigurableServcieType.MEMORY_SERVICE_NAME)
+public class MemorySupportParentConfigureService extends AbstractSupportParentConfigureService {
+
+ @Override
+ protected void initBeforeInitConfigurable(Properties property) {
+ // this.rootConfigureService = new MemoryConfigureService(property);
+ this.parentConfigureService = new MemoryConfigureService(properties);
+ this.configureService = new MemoryConfigureService(properties);
+ }
+}
diff --git a/rocketmq-streams-configurable/src/main/resources/log4j.xml b/rocketmq-streams-configurable/src/main/resources/log4j.xml
new file mode 100755
index 0000000..7812fe7
--- /dev/null
+++ b/rocketmq-streams-configurable/src/main/resources/log4j.xml
@@ -0,0 +1,20 @@
+<?xml version="1.0" encoding="UTF-8" ?>
+<!DOCTYPE log4j:configuration SYSTEM "http://toolkit.alibaba-inc.com/dtd/log4j/log4j.dtd">
+<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
+
+ <appender name="Console" class="org.apache.log4j.ConsoleAppender">
+ <layout class="org.apache.log4j.PatternLayout">
+ <param name="ConversionPattern" value="%d{ISO8601} %l [%t] %-5p - %m%n%n"/>
+ </layout>
+ <filter class="org.apache.log4j.varia.LevelRangeFilter">
+ <param name="LevelMin" value="INFO"/>
+ <param name="LevelMax" value="ERROR"/>
+ </filter>
+ </appender>
+
+ <root>
+ <priority value="INFO"/>
+ <appender-ref ref="Console"/>
+ </root>
+
+</log4j:configuration>
\ No newline at end of file
diff --git a/rocketmq-streams-configurable/src/test/java/org/apache/rocketmq/streams/configuable/ConfiguableComponentTest.java b/rocketmq-streams-configurable/src/test/java/org/apache/rocketmq/streams/configuable/ConfiguableComponentTest.java
new file mode 100644
index 0000000..50cc83b
--- /dev/null
+++ b/rocketmq-streams-configurable/src/test/java/org/apache/rocketmq/streams/configuable/ConfiguableComponentTest.java
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.streams.configuable;
+
+import java.util.List;
+
+import org.apache.rocketmq.streams.common.component.AbstractComponent;
+import org.apache.rocketmq.streams.common.component.ComponentCreator;
+import org.apache.rocketmq.streams.common.configurable.IConfigurableService;
+import org.apache.rocketmq.streams.configuable.model.Person;
+import org.junit.Test;
+
+import static junit.framework.TestCase.assertTrue;
+
+public class ConfiguableComponentTest {
+
+ @Test
+ public void testInsertConfiguable(){
+ String namespace="org.apache.configuable.test";
+ ConfigurableComponent configurableComponent= ConfigurableComponent.getInstance(namespace);
+ Person person=createPerson(namespace);
+ configurableComponent.insert(person);//完成数据存储，在配置文件配置存储类型，支持内存，db和文件，默认是内存
+ //查询只操作内存，存储的数据定时加载到内存，刚插入的数据，还未加载，查询不到
+ assertTrue(configurableComponent.queryConfigurable("person","personName")==null);
+ configurableComponent.refreshConfigurable(namespace);//强制加载数据到内存，可以查询数据
+ assertTrue(configurableComponent.queryConfigurable("person","peronName")!=null);
+ }
+
+
+ @Test
+ public void testConfiguableENVDependence(){
+ String namespace="org.apache.configuable.test";
+ ConfigurableComponent configurableComponent= ConfigurableComponent.getInstance(namespace);
+ Person person=createPerson(namespace);
+ person.setName("persion.name");//对于有ENVDependence的字段，可以不存储真值，存储一个key，把真值配置在配置文件中
+ configurableComponent.insert(person);//完成数据存储，在配置文件配置存储类型，支持内存，db和文件，默认是内存
+ ComponentCreator.getProperties().put("persion.name","realName");//这个代表真实的配置文件，启动时会把配置文件的内容加载到ComponentCreator.getProperties()中
+ configurableComponent.refreshConfigurable(namespace);//刷新存储
+ person=configurableComponent.queryConfigurable("person","peronName");
+ assertTrue(person.getName().equals("realName"));
+ }
+
+
+ @Test
+ public void testSupportParentNameSpace(){
+ String namespace="org.apache.configuable.test";
+ ConfigurableComponent configurableComponent= ConfigurableComponent.getInstance(namespace);
+ Person person=createPerson(namespace);
+ Person otherPerson=createPerson("org.apache.configuable.test1");
+ configurableComponent.insert(person);
+ configurableComponent.insert(otherPerson);
+ configurableComponent.refreshConfigurable(namespace);
+ //只加载自己命名空间的对象
+ List<Person> personList=configurableComponent.queryConfigurableByType("person");
+ assertTrue(personList.size()==1);
+
+ /**
+ * 顶级命名空间的对象，所有namespace都可见
+ */
+ Person thirdPerson=createPerson(IConfigurableService.PARENT_CHANNEL_NAME_SPACE);
+ configurableComponent.insert(thirdPerson);
+ configurableComponent.refreshConfigurable(namespace);//只加载自己命名空间的对象
+ personList=configurableComponent.queryConfigurableByType("person");
+ assertTrue(personList.size()==2);
+ }
+
+
+ //测试定时加载逻辑，当对象的updateFlag值变化后，才会被替换旧对象
+ @Test
+ public void testAutoLoader() throws InterruptedException {
+ ComponentCreator.getProperties().put(AbstractComponent.POLLING_TIME,"1");//1秒后动态加载对象
+ String namespace="org.apache.configuable.test";
+ ConfigurableComponent configurableComponent= ConfigurableComponent.getInstance(namespace);
+ Person person=createPerson(namespace);
+ configurableComponent.insert(person);
+ Thread.sleep(2000);//1秒后，新插入的对象会被加载
+ person=configurableComponent.queryConfigurable("person","peronName");
+ assertTrue(person!=null);
+
+
+ }
+
+ /**
+ * 创建configuable对象
+ * @param namespace
+ * @return
+ */
+ protected Person createPerson(String namespace){
+ Person person=new Person();
+ person.setName("chris");
+ person.setAge(18);
+ person.setNameSpace(namespace);
+ person.setConfigureName("peronName");
+ person.setType("person");
+ return person;
+ }
+}
diff --git a/rocketmq-streams-configurable/src/test/java/org/apache/rocketmq/streams/configuable/model/Person.java b/rocketmq-streams-configurable/src/test/java/org/apache/rocketmq/streams/configuable/model/Person.java
new file mode 100644
index 0000000..8c4f076
--- /dev/null
+++ b/rocketmq-streams-configurable/src/test/java/org/apache/rocketmq/streams/configuable/model/Person.java
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.streams.configuable.model;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.rocketmq.streams.common.configurable.BasedConfigurable;
+import org.apache.rocketmq.streams.common.configurable.annotation.ENVDependence;
+public class Person extends BasedConfigurable{
+ @ENVDependence
+ private String name;
+ private int age;
+ private Boolean isMale;
+ private List<String> addresses;
+ private Map<String, Integer> childName2Age;
+
+ public static Person createPerson(String namespace) {
+ Person person = new Person();
+ person.setNameSpace(namespace);
+ person.setType("person");
+ person.setConfigureName("Chris");
+ person.setName("Chris");
+ List<String> addresses = new ArrayList<>();
+ addresses.add("huilongguan");
+ addresses.add("shangdi");
+ person.setAddresses(addresses);
+ Map<String, Integer> childName2Age = new HashMap<>();
+ childName2Age.put("yuanyahan", 8);
+ childName2Age.put("yuanruxi", 4);
+ person.setChildName2Age(childName2Age);
+ person.setMale(true);
+ person.setAge(18);
+ return person;
+ }
+
+ @Override
+ public String toString() {
+ return "Person{" + "name='" + name + '\'' + ", age=" + age + ", isMale=" + isMale + ", addresses=" + addresses
+ + ", childName2Age=" + childName2Age + '}';
+ }
+
+ public String getName() {
+ return name;
+ }
+
+ public void setName(String name) {
+ this.name = name;
+ }
+
+ public int getAge() {
+ return age;
+ }
+
+ public void setAge(int age) {
+ this.age = age;
+ }
+
+ public Boolean getMale() {
+ return isMale;
+ }
+
+ public void setMale(Boolean male) {
+ isMale = male;
+ }
+
+ public List<String> getAddresses() {
+ return addresses;
+ }
+
+ public void setAddresses(List<String> addresses) {
+ this.addresses = addresses;
+ }
+
+ public Map<String, Integer> getChildName2Age() {
+ return childName2Age;
+ }
+
+ public void setChildName2Age(Map<String, Integer> childName2Age) {
+ this.childName2Age = childName2Age;
+ }
+
+ @Override
+ public Object clone() {
+ Person person = null;
+ try {
+ person = (Person)super.clone();
+ } catch (CloneNotSupportedException e) {
+ System.out.println("clone error " + e);
+ }
+ return person;
+ }
+}
diff --git a/rocketmq-streams-configurable/src/test/java/org/apache/rocketmq/streams/configurable/ConfigurableComponentTest.java b/rocketmq-streams-configurable/src/test/java/org/apache/rocketmq/streams/configurable/ConfigurableComponentTest.java
new file mode 100644
index 0000000..6e632b1
--- /dev/null
+++ b/rocketmq-streams-configurable/src/test/java/org/apache/rocketmq/streams/configurable/ConfigurableComponentTest.java
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.configurable;
+
+import java.util.List;
+
+import org.apache.rocketmq.streams.common.component.AbstractComponent;
+import org.apache.rocketmq.streams.common.component.ComponentCreator;
+import org.apache.rocketmq.streams.common.configurable.IConfigurableService;
+import org.apache.rocketmq.streams.configurable.model.Person;
+import org.junit.Test;
+
+import static junit.framework.TestCase.assertTrue;
+
+public class ConfigurableComponentTest {
+
+ @Test
+ public void testInsertConfigurable() {
+ String namespace = "org.apache.configurable.test";
+ ConfigurableComponent configurableComponent = ConfigurableComponent.getInstance(namespace);
+ Person person = createPerson(namespace);
+ configurableComponent.insert(person);//完成数据存储，在配置文件配置存储类型，支持内存，db和文件，默认是内存
+ //查询只操作内存，存储的数据定时加载到内存，刚插入的数据，还未加载，查询不到
+ assertTrue(configurableComponent.queryConfigurable("person", "personName") == null);
+ configurableComponent.refreshConfigurable(namespace);//强制加载数据到内存，可以查询数据
+ assertTrue(configurableComponent.queryConfigurable("person", "peronName") != null);
+ }
+
+ @Test
+ public void testConfigurableENVDependence() {
+ String namespace = "org.apache.configurable.test";
+ ConfigurableComponent configurableComponent = ConfigurableComponent.getInstance(namespace);
+ Person person = createPerson(namespace);
+ person.setName("persion.name");//对于有ENVDependence的字段，可以不存储真值，存储一个key，把真值配置在配置文件中
+ configurableComponent.insert(person);//完成数据存储，在配置文件配置存储类型，支持内存，db和文件，默认是内存
+ ComponentCreator.getProperties().put("persion.name", "realName");//这个代表真实的配置文件，启动时会把配置文件的内容加载到ComponentCreator.getProperties()中
+ configurableComponent.refreshConfigurable(namespace);//刷新存储
+ person = configurableComponent.queryConfigurable("person", "peronName");
+ assertTrue(person.getName().equals("realName"));
+ }
+
+ @Test
+ public void testSupportParentNameSpace() {
+ String namespace = "org.apache.configurable.test";
+ ConfigurableComponent configurableComponent = ConfigurableComponent.getInstance(namespace);
+ Person person = createPerson(namespace);
+ Person otherPerson = createPerson("org.apache.configuable.test1");
+ configurableComponent.insert(person);
+ configurableComponent.insert(otherPerson);
+ configurableComponent.refreshConfigurable(namespace);
+ //只加载自己命名空间的对象
+ List<Person> personList = configurableComponent.queryConfigurableByType("person");
+ assertTrue(personList.size() == 1);
+
+ /**
+ * 顶级命名空间的对象，所有namespace都可见
+ */
+ Person thirdPerson = createPerson(IConfigurableService.PARENT_CHANNEL_NAME_SPACE);
+ configurableComponent.insert(thirdPerson);
+ configurableComponent.refreshConfigurable(namespace);//只加载自己命名空间的对象
+ personList = configurableComponent.queryConfigurableByType("person");
+ assertTrue(personList.size() == 2);
+ }
+
+ //测试定时加载逻辑，当对象的updateFlag值变化后，才会被替换旧对象
+ @Test
+ public void testAutoLoader() throws InterruptedException {
+ ComponentCreator.getProperties().put(AbstractComponent.POLLING_TIME, "1");//1秒后动态加载对象
+ String namespace = "org.apache.configurable.test";
+ ConfigurableComponent configurableComponent = ConfigurableComponent.getInstance(namespace);
+ Person person = createPerson(namespace);
+ configurableComponent.insert(person);
+ Thread.sleep(2000);//1秒后，新插入的对象会被加载
+ person = configurableComponent.queryConfigurable("person", "peronName");
+ assertTrue(person != null);
+
+ }
+
+ /**
+ * 创建configurable对象
+ *
+ * @param namespace
+ * @return
+ */
+ protected Person createPerson(String namespace) {
+ Person person = new Person();
+ person.setName("chris");
+ person.setAge(18);
+ person.setNameSpace(namespace);
+ person.setConfigureName("peronName");
+ person.setType("person");
+ return person;
+ }
+}
diff --git a/rocketmq-streams-configurable/src/test/java/org/apache/rocketmq/streams/configurable/model/Person.java b/rocketmq-streams-configurable/src/test/java/org/apache/rocketmq/streams/configurable/model/Person.java
new file mode 100644
index 0000000..06fd678
--- /dev/null
+++ b/rocketmq-streams-configurable/src/test/java/org/apache/rocketmq/streams/configurable/model/Person.java
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.configurable.model;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.rocketmq.streams.common.configurable.BasedConfigurable;
+import org.apache.rocketmq.streams.common.configurable.annotation.ENVDependence;
+
+public class Person extends BasedConfigurable {
+ @ENVDependence
+ private String name;
+ private int age;
+ private Boolean isMale;
+ private List<String> addresses;
+ private Map<String, Integer> childName2Age;
+
+ public static Person createPerson(String namespace) {
+ Person person = new Person();
+ person.setNameSpace(namespace);
+ person.setType("person");
+ person.setConfigureName("Chris");
+ person.setName("Chris");
+ List<String> addresses = new ArrayList<>();
+ addresses.add("huilongguan");
+ addresses.add("shangdi");
+ person.setAddresses(addresses);
+ Map<String, Integer> childName2Age = new HashMap<>();
+ childName2Age.put("yuanyahan", 8);
+ childName2Age.put("yuanruxi", 4);
+ person.setChildName2Age(childName2Age);
+ person.setMale(true);
+ person.setAge(18);
+ return person;
+ }
+
+ @Override
+ public String toString() {
+ return "Person{" + "name='" + name + '\'' + ", age=" + age + ", isMale=" + isMale + ", addresses=" + addresses
+ + ", childName2Age=" + childName2Age + '}';
+ }
+
+ public String getName() {
+ return name;
+ }
+
+ public void setName(String name) {
+ this.name = name;
+ }
+
+ public int getAge() {
+ return age;
+ }
+
+ public void setAge(int age) {
+ this.age = age;
+ }
+
+ public Boolean getMale() {
+ return isMale;
+ }
+
+ public void setMale(Boolean male) {
+ isMale = male;
+ }
+
+ public List<String> getAddresses() {
+ return addresses;
+ }
+
+ public void setAddresses(List<String> addresses) {
+ this.addresses = addresses;
+ }
+
+ public Map<String, Integer> getChildName2Age() {
+ return childName2Age;
+ }
+
+ public void setChildName2Age(Map<String, Integer> childName2Age) {
+ this.childName2Age = childName2Age;
+ }
+
+ @Override
+ public Object clone() {
+ Person person = null;
+ try {
+ person = (Person)super.clone();
+ } catch (CloneNotSupportedException e) {
+ System.out.println("clone error " + e);
+ }
+ return person;
+ }
+}
diff --git a/rocketmq-streams-configurable/src/test/resources/log4j.xml b/rocketmq-streams-configurable/src/test/resources/log4j.xml
new file mode 100755
index 0000000..7812fe7
--- /dev/null
+++ b/rocketmq-streams-configurable/src/test/resources/log4j.xml
@@ -0,0 +1,20 @@
+<?xml version="1.0" encoding="UTF-8" ?>
+<!DOCTYPE log4j:configuration SYSTEM "http://toolkit.alibaba-inc.com/dtd/log4j/log4j.dtd">
+<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
+
+ <appender name="Console" class="org.apache.log4j.ConsoleAppender">
+ <layout class="org.apache.log4j.PatternLayout">
+ <param name="ConversionPattern" value="%d{ISO8601} %l [%t] %-5p - %m%n%n"/>
+ </layout>
+ <filter class="org.apache.log4j.varia.LevelRangeFilter">
+ <param name="LevelMin" value="INFO"/>
+ <param name="LevelMax" value="ERROR"/>
+ </filter>
+ </appender>
+
+ <root>
+ <priority value="INFO"/>
+ <appender-ref ref="Console"/>
+ </root>
+
+</log4j:configuration>
\ No newline at end of file
[rocketmq-streams] 28/35: Merge pull request #3 from xstorm1/dev
Posted by du...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
duhengforever pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/rocketmq-streams.git
commit 1b71925435f3a90c3d23713099412ffeb2faac79
Merge: 177cc75 513deaf
Author: rongtong <ji...@163.com>
AuthorDate: Mon Aug 2 19:18:58 2021 +0800
Merge pull request #3 from xstorm1/dev
merge to main branch
rocketmq-streams-channel-rocketmq/pom.xml | 87 +
.../rocketmq/streams/RocketMQChannelBuilder.java | 90 +
.../apache/rocketmq/streams/RocketMQOffset.java | 89 +
.../streams/queue/RocketMQMessageQueue.java | 107 +
.../apache/rocketmq/streams/sink/RocketMQSink.java | 318 +++
.../rocketmq/streams/source/RocketMQSource.java | 292 ++
.../rocketmq/streams/AbstractChannelTest.java | 58 +
.../rocketmq/streams/RocketMQChannelTest.java | 176 ++
rocketmq-streams-serviceloader/pom.xml | 24 +
.../serviceloader/IServiceLoaderService.java | 31 +
.../serviceloader/ServiceLoaderComponent.java | 132 +
.../namefinder/IServiceNameGetter.java | 30 +
.../impl/AnnotationServiceNameGetter.java | 39 +
.../streams/serviceloader/utils/FileUtil.java | 73 +
.../component/ServiceLoaderComponent.properties | 1 +
.../serviceloader/ServiceLoaderComponentTest.java | 17 +
.../src/test/resources/log4j.xml | 20 +
rocketmq-streams-window/pom.xml | 36 +
.../streams/window/builder/WindowBuilder.java | 95 +
.../streams/window/model/FunctionExecutor.java | 51 +
.../rocketmq/streams/window/model/WindowCache.java | 149 +
.../streams/window/model/WindowInstance.java | 506 ++++
.../window/offset/IWindowMaxValueManager.java | 87 +
.../streams/window/offset/WindowMaxValue.java | 89 +
.../window/offset/WindowMaxValueManager.java | 247 ++
.../window/operator/AbstractShuffleWindow.java | 83 +
.../streams/window/operator/AbstractWindow.java | 781 ++++++
.../streams/window/operator/impl/OverWindow.java | 154 ++
.../window/operator/impl/SessionWindow.java | 248 ++
.../window/operator/impl/WindowOperator.java | 451 +++
.../streams/window/operator/join/DBOperator.java | 427 +++
.../streams/window/operator/join/JoinWindow.java | 553 ++++
.../streams/window/operator/join/Operator.java | 76 +
.../window/shuffle/AbstractSystemChannel.java | 321 +++
.../streams/window/shuffle/ShuffleChannel.java | 542 ++++
.../streams/window/source/WindowRireSource.java | 336 +++
.../streams/window/state/WindowBaseValue.java | 188 ++
.../streams/window/state/impl/JoinLeftState.java | 20 +
.../streams/window/state/impl/JoinRightState.java | 20 +
.../streams/window/state/impl/JoinState.java | 94 +
.../streams/window/state/impl/WindowValue.java | 548 ++++
.../window/storage/AbstractWindowStorage.java | 120 +
.../streams/window/storage/ICommonStorage.java | 50 +
.../streams/window/storage/IKeyGenerator.java | 29 +
.../window/storage/IShufflePartitionManager.java | 52 +
.../rocketmq/streams/window/storage/IStorage.java | 79 +
.../streams/window/storage/IWindowStorage.java | 68 +
.../window/storage/ShufflePartitionManager.java | 86 +
.../streams/window/storage/StorageManager.java | 307 +++
.../streams/window/storage/WindowStorage.java | 183 ++
.../streams/window/storage/db/DBStorage.java | 186 ++
.../streams/window/storage/file/FileStorage.java | 196 ++
.../window/storage/rocksdb/RocksdbStorage.java | 293 ++
.../src/main/resources/dipper.cs | 2892 ++++++++++++++++++++
.../src/main/resources/dipper.properties | 21 +
.../streams/configruation/model/Person.java | 84 +
.../streams/configurable/model/Person.java | 97 +
.../streams/window/WindowInstanceTest.java | 122 +
.../streams/window/WindowMaxValueTest.java | 35 +
.../rocketmq/streams/window/WindowValueTest.java | 36 +
.../src/test/resources/log4j.xml | 20 +
61 files changed, 12642 insertions(+)
[rocketmq-streams] 09/35: add filter channel-http schedule modules
Posted by du...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
duhengforever pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/rocketmq-streams.git
commit b3c42426876dfc7f4deaced0e64a43b00252d9a1
Author: arthur.lyj <wr...@vip.qq.com>
AuthorDate: Mon Aug 2 12:05:10 2021 +0800
add filter channel-http schedule modules
---
rocketmq-streams-channel-http/pom.xml | 30 ++
.../rocketmq-streams-channel-http.iml | 62 +++
.../rocketmq/streams/http/source/HttpSource.java | 157 ++++++
.../streams/http/source/server/HttpServer.java | 273 ++++++++++
.../http/source/server/HttpServerManager.java | 60 +++
.../streams/http/source/util/HttpUtil.java | 251 +++++++++
.../streams/http/source/http/HttpChannelTest.java | 45 ++
rocketmq-streams-filter/pom.xml | 31 ++
.../rocketmq-streams-filter.iml | 72 +++
.../rocketmq/streams/filter/FilterComponent.java | 115 ++++
.../streams/filter/builder/ExpressionBuilder.java | 409 ++++++++++++++
.../streams/filter/builder/RuleBuilder.java | 592 +++++++++++++++++++++
.../streams/filter/builder/RuleElementBuilder.java | 137 +++++
.../streams/filter/contants/RuleElementType.java | 43 ++
.../streams/filter/contants/RuleStatus.java | 98 ++++
.../streams/filter/context/ContextConfigure.java | 272 ++++++++++
.../streams/filter/context/RuleContext.java | 399 ++++++++++++++
.../streams/filter/context/RuleMessage.java | 48 ++
.../streams/filter/engine/IRuleEngine.java | 51 ++
.../rocketmq/streams/filter/engine/SplitFlow.java | 159 ++++++
.../filter/engine/impl/DefaultRuleEngine.java | 280 ++++++++++
.../filter/exception/RegexTimeoutException.java | 48 ++
.../filter/function/etl/RenameFunction.java | 37 ++
.../expression/AbstractExpressionFunction.java | 67 +++
.../function/expression/CompareFunction.java | 74 +++
.../ContainsCaseInsensitiveFunction.java | 74 +++
.../function/expression/ContainsFunction.java | 62 +++
.../streams/filter/function/expression/Equals.java | 166 ++++++
.../function/expression/ExpressionFunction.java | 33 ++
.../filter/function/expression/GreaterEquals.java | 163 ++++++
.../filter/function/expression/GreaterThan.java | 132 +++++
.../filter/function/expression/InFunction.java | 85 +++
.../filter/function/expression/InMetaData.java | 65 +++
.../function/expression/IpContainsFunction.java | 180 +++++++
.../filter/function/expression/IsNotNull.java | 50 ++
.../streams/filter/function/expression/IsNull.java | 49 ++
.../filter/function/expression/LessEquals.java | 118 ++++
.../filter/function/expression/LessThan.java | 119 +++++
.../filter/function/expression/LikeFunction.java | 167 ++++++
.../NotContainsCaseInsensitiveFunction.java | 76 +++
.../function/expression/NotContainsFunction.java | 63 +++
.../filter/function/expression/NotEquals.java | 169 ++++++
.../filter/function/expression/NotInFunction.java | 35 ++
.../function/expression/NotLikeFunction.java | 36 ++
.../NotRegexCaseInsensitiveFunction.java | 36 ++
.../function/expression/NotRegexFunction.java | 36 ++
.../expression/RegexCaseInsensitiveFunction.java | 41 ++
.../filter/function/expression/RegexFunction.java | 123 +++++
.../filter/function/expression/ScriptFunction.java | 145 +++++
.../filter/function/script/CaseFunction.java | 86 +++
.../rocketmq/streams/filter/monitor/Monitor.java | 160 ++++++
.../filter/monitor/contants/MonitorType.java | 40 ++
.../streams/filter/monitor/rule/ActionMonitor.java | 27 +
.../filter/monitor/rule/ExpressionMonitor.java | 27 +
.../filter/monitor/rule/MessageMonitor.java | 119 +++++
.../streams/filter/monitor/rule/NullMonitor.java | 55 ++
.../streams/filter/monitor/rule/RuleMonitor.java | 62 +++
.../streams/filter/monitor/rule/VarMonitor.java | 27 +
.../streams/filter/operator/FilterOperator.java | 31 ++
.../rocketmq/streams/filter/operator/Rule.java | 538 +++++++++++++++++++
.../streams/filter/operator/action/Action.java | 28 +
.../operator/action/IConfigurableAction.java | 41 ++
.../filter/operator/action/impl/ChannelAction.java | 69 +++
.../operator/action/impl/MetaDataAction.java | 185 +++++++
.../filter/operator/expression/Expression.java | 405 ++++++++++++++
.../operator/expression/ExpressionPerformance.java | 99 ++++
.../expression/ExpressionRelationParser.java | 106 ++++
.../expression/ExpressionRelationPaser.java | 107 ++++
.../operator/expression/GroupExpression.java | 146 +++++
.../expression/GroupExpressionManager.java | 86 +++
.../operator/expression/OptimizationRule.java | 28 +
.../operator/expression/RelationExpression.java | 347 ++++++++++++
.../operator/expression/SimpleExpression.java | 113 ++++
.../streams/filter/operator/var/ConstantVar.java | 118 ++++
.../streams/filter/operator/var/ContextVar.java | 85 +++
.../streams/filter/operator/var/InnerVar.java | 86 +++
.../rocketmq/streams/filter/operator/var/Var.java | 79 +++
.../optimization/EqualsExpressionOptimization.java | 39 ++
.../optimization/ExpressionOptimization.java | 144 +++++
.../optimization/IExpressionOptimization.java | 38 ++
.../optimization/LikeExpressionOptimization.java | 39 ++
.../optimization/OptimizationExpression.java | 71 +++
.../PiplineLogFingerprintAnalysis.java | 433 +++++++++++++++
.../optimization/RegexExpressionOptimization.java | 39 ++
.../streams/filter/service/IRuleEngineService.java | 50 ++
.../filter/service/impl/RuleEngineServiceImpl.java | 104 ++++
.../rocketmq/streams/filter/utils/IPUtil.java | 209 ++++++++
.../rocketmq/streams/filter/utils/RegexUtil.java | 211 ++++++++
.../streams/filter/FilterComponentTest.java | 36 ++
.../streams/filter/operator/ExpressionTest.java | 49 ++
.../rocketmq/streams/filter/operator/RuleTest.java | 40 ++
.../test/resources/credible/ChannelComponent.xml | 70 +++
.../resources/credible/credible-channel.properties | 12 +
.../test/resources/credible/credible.properties | 8 +
.../src/test/resources/log4j.xml | 20 +
rocketmq-streams-schedule/pom.xml | 33 ++
.../rocketmq-streams-schedule.iml | 65 +++
.../streams/schedule/ScheduleComponent.java | 65 +++
.../schedule/job/ConfigurableExecutorJob.java | 55 ++
.../streams/schedule/service/IScheduleService.java | 90 ++++
.../schedule/service/impl/ScheduleServiceImpl.java | 264 +++++++++
.../stream/schedule/ScheduleComponentTest.java | 62 +++
.../src/test/resources/log4j.xml | 20 +
103 files changed, 11599 insertions(+)
diff --git a/rocketmq-streams-channel-http/pom.xml b/rocketmq-streams-channel-http/pom.xml
new file mode 100644
index 0000000..05596b7
--- /dev/null
+++ b/rocketmq-streams-channel-http/pom.xml
@@ -0,0 +1,30 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xmlns="http://maven.apache.org/POM/4.0.0"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams</artifactId>
+ <version>2.0.0-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>rocketmq-streams-channel-http</artifactId>
+ <name>ROCKETMQ STREAMS :: channel-http</name>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams-commons</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>commons-io</groupId>
+ <artifactId>commons-io</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.httpcomponents</groupId>
+ <artifactId>httpclient</artifactId>
+ </dependency>
+ </dependencies>
+
+</project>
\ No newline at end of file
diff --git a/rocketmq-streams-channel-http/rocketmq-streams-channel-http.iml b/rocketmq-streams-channel-http/rocketmq-streams-channel-http.iml
new file mode 100644
index 0000000..51a5150
--- /dev/null
+++ b/rocketmq-streams-channel-http/rocketmq-streams-channel-http.iml
@@ -0,0 +1,62 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<module org.jetbrains.idea.maven.project.MavenProjectsManager.isMavenModule="true" type="JAVA_MODULE" version="4">
+ <component name="NewModuleRootManager" LANGUAGE_LEVEL="JDK_1_8">
+ <output url="file://$MODULE_DIR$/target/classes" />
+ <output-test url="file://$MODULE_DIR$/target/test-classes" />
+ <content url="file://$MODULE_DIR$">
+ <sourceFolder url="file://$MODULE_DIR$/src/main/java" isTestSource="false" />
+ <sourceFolder url="file://$MODULE_DIR$/src/test/java" isTestSource="true" />
+ <excludeFolder url="file://$MODULE_DIR$/target" />
+ </content>
+ <orderEntry type="inheritedJdk" />
+ <orderEntry type="sourceFolder" forTests="false" />
+ <orderEntry type="module" module-name="rocketmq-streams-commons" />
+ <orderEntry type="library" name="Maven: org.apache.commons:commons-lang3:3.11" level="project" />
+ <orderEntry type="library" name="Maven: junit:junit:4.12" level="project" />
+ <orderEntry type="library" name="Maven: org.hamcrest:hamcrest-core:1.3" level="project" />
+ <orderEntry type="library" name="Maven: com.alibaba:fastjson:1.2.27" level="project" />
+ <orderEntry type="library" name="Maven: commons-logging:commons-logging:1.1" level="project" />
+ <orderEntry type="library" name="Maven: logkit:logkit:1.0.1" level="project" />
+ <orderEntry type="library" name="Maven: avalon-framework:avalon-framework:4.1.3" level="project" />
+ <orderEntry type="library" name="Maven: javax.servlet:servlet-api:2.3" level="project" />
+ <orderEntry type="library" name="Maven: log4j:log4j:1.2.17" level="project" />
+ <orderEntry type="library" name="Maven: com.google.code.gson:gson:2.8.5" level="project" />
+ <orderEntry type="library" name="Maven: com.google.auto.service:auto-service:1.0-rc5" level="project" />
+ <orderEntry type="library" name="Maven: com.google.auto.service:auto-service-annotations:1.0-rc5" level="project" />
+ <orderEntry type="library" name="Maven: com.google.auto:auto-common:0.10" level="project" />
+ <orderEntry type="library" name="Maven: com.google.guava:guava:25.1-jre" level="project" />
+ <orderEntry type="library" name="Maven: com.google.code.findbugs:jsr305:3.0.2" level="project" />
+ <orderEntry type="library" name="Maven: org.checkerframework:checker-qual:2.0.0" level="project" />
+ <orderEntry type="library" name="Maven: com.google.errorprone:error_prone_annotations:2.1.3" level="project" />
+ <orderEntry type="library" name="Maven: com.google.j2objc:j2objc-annotations:1.1" level="project" />
+ <orderEntry type="library" name="Maven: org.codehaus.mojo:animal-sniffer-annotations:1.14" level="project" />
+ <orderEntry type="library" name="Maven: com.lmax:disruptor:3.2.0" level="project" />
+ <orderEntry type="library" name="Maven: com.gliwka.hyperscan:hyperscan:5.4.0-2.0.0" level="project" />
+ <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:5.4.0-1.0.0" level="project" />
+ <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:linux-x86_64:5.4.0-1.0.0" level="project" />
+ <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:windows-x86_64:5.4.0-1.0.0" level="project" />
+ <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:macosx-x86_64:5.4.0-1.0.0" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp-platform:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-arm:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-arm64:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-x86:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-x86_64:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:ios-arm64:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:ios-x86_64:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-armhf:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-arm64:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-ppc64le:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-x86:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-x86_64:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:macosx-x86_64:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:windows-x86:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:windows-x86_64:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: net.java.dev.jna:platform:3.5.2" level="project" />
+ <orderEntry type="library" name="Maven: net.java.dev.jna:jna:3.5.2" level="project" />
+ <orderEntry type="library" name="Maven: commons-io:commons-io:2.5" level="project" />
+ <orderEntry type="library" name="Maven: org.apache.httpcomponents:httpclient:4.5.2" level="project" />
+ <orderEntry type="library" name="Maven: org.apache.httpcomponents:httpcore:4.4.4" level="project" />
+ <orderEntry type="library" name="Maven: commons-codec:commons-codec:1.9" level="project" />
+ </component>
+</module>
\ No newline at end of file
diff --git a/rocketmq-streams-channel-http/src/main/java/org/apache/rocketmq/streams/http/source/HttpSource.java b/rocketmq-streams-channel-http/src/main/java/org/apache/rocketmq/streams/http/source/HttpSource.java
new file mode 100644
index 0000000..6b7ea5e
--- /dev/null
+++ b/rocketmq-streams-channel-http/src/main/java/org/apache/rocketmq/streams/http/source/HttpSource.java
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.http.source;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+import org.apache.rocketmq.streams.http.source.server.HttpServerManager;
+import org.apache.rocketmq.streams.common.channel.source.AbstractUnreliableSource;
+import org.apache.rocketmq.streams.common.utils.IPUtil;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+/**
+ * receive http(s) post data
+ **/
+public class HttpSource extends AbstractUnreliableSource {
+ private static final Log LOG = LogFactory.getLog(HttpSource.class);
+ protected boolean isHttps = false;//发送的请求是否是https的
+
+ /**
+ * 下面三个可以都配置，或者只配置一个，只要有匹配上的请求，都会被这个channel处理
+ */
+
+ //消息发送源的ip或域名列表，支持ip范围-192.168.1.1-192.168.1.10，ip端-192.168.0.0/22，和正则表达式
+ protected List<String> ipList = new ArrayList<>();
+ protected List<String> rootPath = new ArrayList<>();//请求的跟path，支持正则和列表
+ protected String routeLabel;//可以在请求的参数中加入一个固定的参数，这个值是参数名称
+ protected String routeValue;
+
+ public boolean match(String hostOrIp, String query, String uri) {
+ String routePara = routeLabel + "=" + routeValue;
+ if (StringUtil.isNotEmpty(query) && query.contains(routePara)) {
+ return true;
+ }
+ if (StringUtil.isNotEmpty(uri)) {
+ for (String path : rootPath) {
+ if (uri.startsWith(path)) {
+ return true;
+ }
+ if (StringUtil.matchRegex(uri, path)) {
+ return true;
+ }
+ }
+ }
+ if (StringUtil.isNotEmpty(hostOrIp)) {
+ for (String ip : ipList) {
+ if (StringUtil.isEmpty(ip)) {
+ continue;
+ }
+ if (ip.equals(hostOrIp)) {
+ return true;
+ } else if (ip.contains("-")) {
+ boolean match = IPUtil.ipInSection(hostOrIp, ip);
+ if (match) {
+ return true;
+ }
+ } else if (ip.contains("/")) {
+ boolean match = IPUtil.isInRange(hostOrIp, ip);
+ if (match) {
+ return true;
+ }
+ } else {
+ boolean match = StringUtil.matchRegex(hostOrIp, ip);
+ if (match) {
+ return true;
+ }
+ }
+ }
+ }
+
+ return false;
+ }
+
+ @Override
+ protected boolean initConfigurable() {
+ boolean success = super.initConfigurable();
+ HttpServerManager.register(this, isHttps);
+ return success;
+ }
+
+ @Override
+ protected boolean startSource() {
+ HttpServerManager.startServer(isHttps);
+ return true;
+ }
+
+ public boolean isHttps() {
+ return isHttps;
+ }
+
+ public void setHttps(boolean https) {
+ isHttps = https;
+ }
+
+ public List<String> getIpList() {
+ return ipList;
+ }
+
+ public void setIpList(List<String> ipList) {
+ this.ipList = ipList;
+ }
+
+ public List<String> getRootPath() {
+ return rootPath;
+ }
+
+ public void setRootPath(List<String> rootPath) {
+ this.rootPath = rootPath;
+ }
+
+ public String getRouteLabel() {
+ return routeLabel;
+ }
+
+ public void setRouteLabel(String routeLabel) {
+ this.routeLabel = routeLabel;
+ }
+
+ public String getRouteValue() {
+ return routeValue;
+ }
+
+ public void setRouteValue(String routeValue) {
+ this.routeValue = routeValue;
+ }
+
+ public void addIps(String... ips) {
+ if (ips == null) {
+ return;
+ }
+ Collections.addAll(this.ipList, ips);
+ }
+
+ public void addPath(String... paths) {
+ if (paths == null) {
+ return;
+ }
+ Collections.addAll(this.rootPath, paths);
+ }
+}
\ No newline at end of file
diff --git a/rocketmq-streams-channel-http/src/main/java/org/apache/rocketmq/streams/http/source/server/HttpServer.java b/rocketmq-streams-channel-http/src/main/java/org/apache/rocketmq/streams/http/source/server/HttpServer.java
new file mode 100644
index 0000000..62a0ae8
--- /dev/null
+++ b/rocketmq-streams-channel-http/src/main/java/org/apache/rocketmq/streams/http/source/server/HttpServer.java
@@ -0,0 +1,273 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.http.source.server;
+
+import com.alibaba.fastjson.JSONObject;
+
+import org.apache.rocketmq.streams.http.source.HttpSource;
+import org.apache.rocketmq.streams.common.channel.source.AbstractUnreliableSource;
+import org.apache.rocketmq.streams.common.configurable.annotation.ENVDependence;
+import org.apache.rocketmq.streams.common.interfaces.IStreamOperator;
+import org.apache.rocketmq.streams.common.context.AbstractContext;
+import org.apache.rocketmq.streams.common.context.IMessage;
+import com.sun.net.httpserver.HttpExchange;
+import com.sun.net.httpserver.HttpHandler;
+import com.sun.net.httpserver.HttpsConfigurator;
+import com.sun.net.httpserver.HttpsParameters;
+import com.sun.net.httpserver.HttpsServer;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.SSLEngine;
+import javax.net.ssl.SSLParameters;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.net.InetSocketAddress;
+import java.security.KeyStore;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+public class HttpServer extends AbstractUnreliableSource {
+ private static final Log LOG = LogFactory.getLog(HttpServer.class);
+ private transient com.sun.net.httpserver.HttpServer server;
+ @ENVDependence
+ private int port = 8000;
+ @ENVDependence
+ private String serverIp = "localhost";
+ @ENVDependence
+ private int backlog = 10;
+ @ENVDependence
+ private int stopDelaySecond;
+ private boolean useHttps = false;
+
+ protected transient List<HttpSource> channels = new ArrayList<>();
+
+ public HttpServer() {
+ setJsonData(false);
+ setMsgIsJsonArray(false);
+ }
+
+ public void register(HttpSource channel) {
+ this.channels.add(channel);
+ }
+
+ @Override
+ protected boolean initConfigurable() {
+ try {
+
+ if (useHttps) {
+ HttpsServer httpsServer = HttpsServer.create(new InetSocketAddress(serverIp, port), backlog);
+ SSLContext sslContext = SSLContext.getInstance("TLS");
+ // initialise the keystore
+ char[] password = "password".toCharArray();
+ KeyStore ks = KeyStore.getInstance("JKS");
+ InputStream fis = this.getClass().getResourceAsStream("/testkey.jks");
+ ks.load(fis, password);
+ // setup the key manager factory
+ KeyManagerFactory kmf = KeyManagerFactory.getInstance("SunX509");
+ kmf.init(ks, password);
+ // setup the trust manager factory
+ TrustManagerFactory tmf = TrustManagerFactory.getInstance("SunX509");
+ tmf.init(ks);
+ // setup the HTTPS context and parameters
+ sslContext.init(kmf.getKeyManagers(), tmf.getTrustManagers(), null);
+ httpsServer.setHttpsConfigurator(new HttpsConfigurator(sslContext) {
+ @Override
+ public void configure(HttpsParameters params) {
+ try {
+ // initialise the SSL context
+ SSLContext context = getSSLContext();
+ SSLEngine engine = context.createSSLEngine();
+ params.setNeedClientAuth(false);
+ params.setCipherSuites(engine.getEnabledCipherSuites());
+ params.setProtocols(engine.getEnabledProtocols());
+
+ // Set the SSL parameters
+ SSLParameters sslParameters = context.getSupportedSSLParameters();
+ params.setSSLParameters(sslParameters);
+
+ } catch (Exception ex) {
+ LOG.error("Failed to create HTTPS port", ex);
+ }
+ }
+ });
+ server = httpsServer;
+ } else {
+ server = com.sun.net.httpserver.HttpServer.create(new InetSocketAddress(serverIp, port), backlog);
+ }
+
+ } catch (IOException e) {
+ // System.out.println(e);
+ LOG.error("http channel init get io exception", e);
+ return false;
+ } catch (NoSuchAlgorithmException e) {
+ // System.out.println(e);
+ LOG.error("http channel init https ssl context exception", e);
+ return false;
+ } catch (Exception e) {
+ // System.out.println(e);
+ LOG.error("http channel init http cert exception", e);
+ return false;
+ }
+ return true;
+ }
+
+ @Override
+ protected boolean startSource() {
+ ExecutorService cachedThreadPool = new ThreadPoolExecutor(maxThread, maxThread,
+ 0L, TimeUnit.MILLISECONDS,
+ new LinkedBlockingQueue<Runnable>(1000));
+ server.setExecutor(cachedThreadPool);
+ server.createContext("/", new DataHandler());
+ setReceiver((message, context) -> {
+ String clientIp = message.getMessageBody().getString("clientIp");
+ String query = message.getMessageBody().getString("query");
+ String uri = message.getMessageBody().getString("uri");
+ List<HttpSource> destroyChannels = new ArrayList<>();
+ for (HttpSource channel : channels) {
+ if (channel.match(clientIp, query, uri)) {
+ if (channel.isDestroy()) {
+ destroyChannels.add(channel);
+ continue;
+ }
+ channel.doReceiveMessage(message.getMessageBody());
+ }
+ }
+ for (HttpSource httpChannel : destroyChannels) {
+ channels.remove(httpChannel);
+ }
+ return message;
+ });
+ server.start();
+ if (isUseHttps()) {
+ LOG.info("https server is start");
+ } else {
+ LOG.info("http server is start");
+ }
+
+ return true;
+ }
+
+ public boolean isUseHttps() {
+ return useHttps;
+ }
+
+ public void setUseHttps(boolean useHttps) {
+ this.useHttps = useHttps;
+ }
+
+ public String getServerIp() {
+ return serverIp;
+ }
+
+ public void setServerIp(String serverIp) {
+ this.serverIp = serverIp;
+ }
+
+ public int getPort() {
+ return port;
+ }
+
+ public void setPort(int port) {
+ this.port = port;
+ }
+
+ public int getBacklog() {
+ return backlog;
+ }
+
+ public void setBacklog(int backlog) {
+ this.backlog = backlog;
+ }
+
+ public int getStopDelaySecond() {
+ return stopDelaySecond;
+ }
+
+ public void setStopDelaySecond(int delaySecond) {
+ this.stopDelaySecond = delaySecond;
+ }
+
+ private class DataHandler implements HttpHandler {
+ @Override
+ public void handle(HttpExchange exchange) throws IOException {
+ OutputStream os = exchange.getResponseBody();
+ InputStream in = exchange.getRequestBody();
+ try {
+ JSONObject jsonObject = createHttpHeader(exchange);
+ jsonObject.put("query", exchange.getRequestURI().getQuery());
+ String data = getContent(in);
+ jsonObject.put(IMessage.DATA_KEY, data);
+
+ doReceiveMessage(jsonObject);
+ String response = "{\"code\": \"200\", \"data\" :\"received\", \"message\" :\"\"}";
+ exchange.sendResponseHeaders(200, 0);
+ os.write(response.getBytes());
+ os.flush();
+ } catch (Exception e) {
+ LOG.error("");
+ } finally {
+ if (os != null) {
+ os.close();
+ }
+ if (in != null) {
+ in.close();
+ }
+ }
+
+ }
+ }
+
+ protected JSONObject createHttpHeader(HttpExchange exchange) {
+ JSONObject header = new JSONObject();
+ header.put("clientIp", exchange.getRemoteAddress().getAddress().getHostAddress());
+ header.put("uri", exchange.getRequestURI().getPath());
+ header.put("User-agent", exchange.getRequestHeaders().getFirst("User-agent"));
+ header.put("Content-type", exchange.getRequestHeaders().getFirst("Content-type"));
+ header.put("Host", exchange.getRequestHeaders().getFirst("Host"));
+ header.put("method", exchange.getRequestMethod());
+ return header;
+ }
+
+ /**
+ * 从请求里获取内容
+ *
+ * @param in
+ * @return
+ * @throws IOException
+ */
+ protected String getContent(InputStream in) throws IOException {
+ BufferedReader reader = new BufferedReader(new InputStreamReader(in));
+ String line;
+ StringBuilder stringBuilder = new StringBuilder();
+ while ((line = reader.readLine()) != null) {
+ stringBuilder.append(line);
+ }
+ return stringBuilder.toString();
+ }
+}
\ No newline at end of file
diff --git a/rocketmq-streams-channel-http/src/main/java/org/apache/rocketmq/streams/http/source/server/HttpServerManager.java b/rocketmq-streams-channel-http/src/main/java/org/apache/rocketmq/streams/http/source/server/HttpServerManager.java
new file mode 100644
index 0000000..ca878e3
--- /dev/null
+++ b/rocketmq-streams-channel-http/src/main/java/org/apache/rocketmq/streams/http/source/server/HttpServerManager.java
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.http.source.server;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.rocketmq.streams.http.source.HttpSource;
+
+import java.util.concurrent.atomic.AtomicBoolean;
+
+/**
+ * receive http(s) post data
+ **/
+public class HttpServerManager {
+ private static final Log LOG = LogFactory.getLog(HttpServerManager.class);
+ protected transient static HttpServer HTTP_SERVER = new HttpServer();//http server，全局只有一个
+ protected transient static HttpServer HTTPS_SERVER = new HttpServer();//https server 全局只有一个
+ protected transient static AtomicBoolean isHttpStart = new AtomicBoolean(false);//只启动一次
+ protected transient static AtomicBoolean isHttpsStart = new AtomicBoolean(false);//只启动一次
+
+ public static void register(HttpSource channel, boolean isHttps) {
+ if (isHttps) {
+ HTTPS_SERVER.register(channel);
+ } else {
+ HTTP_SERVER.register(channel);
+ }
+ }
+
+ public static void startServer(boolean isHttps) {
+ if (isHttps) {
+ if (isHttpsStart.compareAndSet(false, true)) {
+ HTTPS_SERVER.setUseHttps(true);
+ HTTPS_SERVER.setPort(443);
+ HTTPS_SERVER.init();
+ HTTPS_SERVER.startSource();
+ }
+ } else {
+ if (isHttpStart.compareAndSet(false, true)) {
+ HTTP_SERVER.setPort(8000);
+ HTTP_SERVER.setUseHttps(false);
+ HTTP_SERVER.init();
+ HTTP_SERVER.startSource();
+ }
+ }
+ }
+}
\ No newline at end of file
diff --git a/rocketmq-streams-channel-http/src/main/java/org/apache/rocketmq/streams/http/source/util/HttpUtil.java b/rocketmq-streams-channel-http/src/main/java/org/apache/rocketmq/streams/http/source/util/HttpUtil.java
new file mode 100644
index 0000000..92dddf8
--- /dev/null
+++ b/rocketmq-streams-channel-http/src/main/java/org/apache/rocketmq/streams/http/source/util/HttpUtil.java
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.http.source.util;
+
+import java.security.cert.CertificateException;
+import java.security.cert.X509Certificate;
+
+import javax.net.ssl.HostnameVerifier;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.SSLSession;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.http.Consts;
+import org.apache.http.Header;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpDelete;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.client.methods.HttpPatch;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.config.ConnectionConfig;
+import org.apache.http.config.Registry;
+import org.apache.http.config.RegistryBuilder;
+import org.apache.http.conn.socket.ConnectionSocketFactory;
+import org.apache.http.conn.socket.PlainConnectionSocketFactory;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.conn.ssl.TrustStrategy;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClientBuilder;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.impl.conn.PoolingHttpClientConnectionManager;
+import org.apache.http.ssl.SSLContexts;
+import org.apache.http.util.EntityUtils;
+
+public class HttpUtil {
+
+ public static final int NORMAL_STATUES = 200;
+ public static final String CHARSET = "UTF-8";
+ public static final int TIMOUT = 10000;
+ public static final int CONNECT_TIMOUT = 10000;
+
+ private static CloseableHttpClient httpclient;
+
+ static {
+ RequestConfig.Builder configBuilder = RequestConfig.custom();
+ configBuilder.setConnectionRequestTimeout(CONNECT_TIMOUT);
+ configBuilder.setConnectTimeout(CONNECT_TIMOUT);
+ configBuilder.setSocketTimeout(TIMOUT);
+ SSLConnectionSocketFactory sslsf = null;
+ try {
+ SSLContext sslcontext = SSLContexts.custom().loadTrustMaterial(new TrustStrategy() {
+ @Override
+ public boolean isTrusted(X509Certificate[] chain, String authType) throws CertificateException {
+ return true;
+ }
+ }).build();
+ sslsf = new SSLConnectionSocketFactory(sslcontext, new HostnameVerifier() {
+ @Override
+ public boolean verify(String s, SSLSession sslSession) {
+ return true;
+ }
+ });
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ ConnectionConfig connectionConfig = ConnectionConfig.custom().setCharset(Consts.UTF_8).build();
+ Registry<ConnectionSocketFactory> socketFactoryRegistry = RegistryBuilder.<ConnectionSocketFactory>create().register("https",
+ sslsf).register("http",
+ new PlainConnectionSocketFactory()).build();
+ PoolingHttpClientConnectionManager connManager = new PoolingHttpClientConnectionManager(socketFactoryRegistry);
+ connManager.setDefaultConnectionConfig(connectionConfig);
+ connManager.setMaxTotal(500);
+ connManager.setDefaultMaxPerRoute(50);
+ HttpClientBuilder clientBuilder = HttpClients.custom();
+
+ clientBuilder.setDefaultRequestConfig(configBuilder.build());
+ clientBuilder.setSSLSocketFactory(sslsf);
+ clientBuilder.setConnectionManager(connManager);
+
+ httpclient = clientBuilder.build();
+ }
+
+ public static String getContent(String url) {
+ return getContent(url, null);
+ }
+
+ public static String getContent(String url, Header... headers) {
+ try {
+ String content = EntityUtils.toString(get(url, headers).getEntity(), CHARSET);
+ return content;
+ } catch (Exception e) {
+ e.printStackTrace();
+ }
+ return null;
+ }
+
+ public static String deleteContent(String url) {
+ return deleteContent(url, null);
+ }
+
+ public static String deleteContent(String url, Header... headers) {
+ try {
+ String content = EntityUtils.toString(delete(url, headers).getEntity(), CHARSET);
+ return content;
+ } catch (Exception e) {
+ e.printStackTrace();
+ }
+ return null;
+ }
+
+ public static String postContent(String url, String body) {
+ return postContent(url, body, null);
+ }
+
+ public static String patchContent(String url, String body) {
+ return patchContent(url, body, null);
+ }
+
+ public static String postContent(String url, String body, Header... headers) {
+ try {
+ String content = EntityUtils.toString(post(url, body, headers).getEntity(), CHARSET);
+ return content;
+ } catch (Exception e) {
+ e.printStackTrace();
+ }
+ return null;
+ }
+
+ public static String patchContent(String url, String body, Header... headers) {
+ try {
+ String content = EntityUtils.toString(patch(url, body, headers).getEntity(), CHARSET);
+ return content;
+ } catch (Exception e) {
+ e.printStackTrace();
+ }
+ return null;
+ }
+
+ public static CloseableHttpResponse get(String url) {
+ return get(url, null);
+ }
+
+ public static CloseableHttpResponse delete(String url) {
+ return delete(url, null);
+ }
+
+ public static CloseableHttpResponse get(String url, Header... headers) {
+ try {
+ HttpGet httpGet = new HttpGet(url);
+ if (headers != null && headers.length > 0) {
+ for (Header header : headers) {
+ httpGet.addHeader(header);
+ }
+ }
+ CloseableHttpResponse response = httpclient.execute(httpGet);
+ return response;
+ } catch (Exception e) {
+ e.printStackTrace();
+ }
+ return null;
+ }
+
+ public static CloseableHttpResponse delete(String url, Header... headers) {
+ try {
+ HttpDelete httpDelete = new HttpDelete(url);
+ if (headers != null && headers.length > 0) {
+ for (Header header : headers) {
+ httpDelete.addHeader(header);
+ }
+ }
+ CloseableHttpResponse response = httpclient.execute(httpDelete);
+ return response;
+ } catch (Exception e) {
+ e.printStackTrace();
+ }
+ return null;
+ }
+
+ public static CloseableHttpResponse post(String url, String body) {
+ return post(url, body, null);
+ }
+
+ public static CloseableHttpResponse patch(String url, String body) {
+ return patch(url, body, null);
+ }
+
+ public static CloseableHttpResponse post(String url, String body, Header... headers) {
+ try {
+ HttpPost httpPost = new HttpPost(url);
+ StringEntity stringEntity = new StringEntity(body, CHARSET);
+ stringEntity.setContentEncoding(CHARSET);
+ stringEntity.setContentType("application/json");
+ httpPost.setEntity(stringEntity);
+ if (headers != null && headers.length > 0) {
+ for (Header header : headers) {
+ httpPost.addHeader(header);
+ }
+ }
+ CloseableHttpResponse response = httpclient.execute(httpPost);
+ return response;
+
+ } catch (Exception e) {
+ e.printStackTrace();
+ }
+ return null;
+ }
+
+ public static CloseableHttpResponse patch(String url, String body, Header... headers) {
+ try {
+ HttpPatch httpPatch = new HttpPatch(url);
+ StringEntity stringEntity = new StringEntity(body, CHARSET);
+ stringEntity.setContentEncoding(CHARSET);
+ stringEntity.setContentType("application/json");
+ httpPatch.setEntity(stringEntity);
+ if (headers != null && headers.length > 0) {
+ for (Header header : headers) {
+ httpPatch.addHeader(header);
+ }
+ }
+ CloseableHttpResponse response = httpclient.execute(httpPatch);
+ return response;
+
+ } catch (Exception e) {
+ e.printStackTrace();
+ }
+ return null;
+ }
+
+ public static String replaceSpace(String str) {
+ if (StringUtils.isNotBlank(str)) {
+ return str.replace(" ", "").replace("\r", "").replace("\n", "").replace("\r\n", "");
+ }
+ return str;
+ }
+}
+
diff --git a/rocketmq-streams-channel-http/src/test/java/org/apache/rocketmq/streams/http/source/http/HttpChannelTest.java b/rocketmq-streams-channel-http/src/test/java/org/apache/rocketmq/streams/http/source/http/HttpChannelTest.java
new file mode 100644
index 0000000..1e6341f
--- /dev/null
+++ b/rocketmq-streams-channel-http/src/test/java/org/apache/rocketmq/streams/http/source/http/HttpChannelTest.java
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.http.source.http;
+
+import org.apache.rocketmq.streams.common.channel.source.ISource;
+import org.apache.rocketmq.streams.common.utils.ThreadUtil;
+import org.apache.rocketmq.streams.http.source.HttpSource;
+import org.junit.Test;
+
+public class HttpChannelTest {
+
+ @Test
+ public void startHttpChannel() {
+ ISource channel = create();
+ channel.start((message, context) -> {
+ System.out.println("receive message " + message.getMessageBody());
+ return message;
+ });
+ while (true) {
+ ThreadUtil.sleep(1000);
+ }
+ }
+
+ private ISource create() {
+ HttpSource httpChannel = new HttpSource();
+ httpChannel.addPath("/chris/yuanxiaodong");
+ httpChannel.setHttps(false);
+ httpChannel.init();
+ return httpChannel;
+ }
+}
\ No newline at end of file
diff --git a/rocketmq-streams-filter/pom.xml b/rocketmq-streams-filter/pom.xml
new file mode 100755
index 0000000..6e7d41d
--- /dev/null
+++ b/rocketmq-streams-filter/pom.xml
@@ -0,0 +1,31 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams</artifactId>
+ <version>2.0.0-SNAPSHOT</version>
+ </parent>
+ <artifactId>rocketmq-streams-filter</artifactId>
+ <name>ROCKETMQ STREAMS :: filter</name>
+ <packaging>jar</packaging>
+ <properties>
+ <file_encoding>UTF-8</file_encoding>
+ <project.build.sourceEncoding>${file_encoding}</project.build.sourceEncoding>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams-script</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams-db-operator</artifactId>
+ </dependency>
+ </dependencies>
+
+
+</project>
diff --git a/rocketmq-streams-filter/rocketmq-streams-filter.iml b/rocketmq-streams-filter/rocketmq-streams-filter.iml
new file mode 100644
index 0000000..70f1781
--- /dev/null
+++ b/rocketmq-streams-filter/rocketmq-streams-filter.iml
@@ -0,0 +1,72 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<module org.jetbrains.idea.maven.project.MavenProjectsManager.isMavenModule="true" type="JAVA_MODULE" version="4">
+ <component name="NewModuleRootManager" LANGUAGE_LEVEL="JDK_1_8">
+ <output url="file://$MODULE_DIR$/target/classes" />
+ <output-test url="file://$MODULE_DIR$/target/test-classes" />
+ <content url="file://$MODULE_DIR$">
+ <sourceFolder url="file://$MODULE_DIR$/src/main/java" isTestSource="false" />
+ <sourceFolder url="file://$MODULE_DIR$/src/test/java" isTestSource="true" />
+ <sourceFolder url="file://$MODULE_DIR$/src/test/resources" type="java-test-resource" />
+ <excludeFolder url="file://$MODULE_DIR$/target" />
+ </content>
+ <orderEntry type="inheritedJdk" />
+ <orderEntry type="sourceFolder" forTests="false" />
+ <orderEntry type="module" module-name="rocketmq-streams-script" />
+ <orderEntry type="module" module-name="rocketmq-streams-commons" />
+ <orderEntry type="library" name="Maven: org.apache.commons:commons-lang3:3.11" level="project" />
+ <orderEntry type="library" name="Maven: junit:junit:4.12" level="project" />
+ <orderEntry type="library" name="Maven: org.hamcrest:hamcrest-core:1.3" level="project" />
+ <orderEntry type="library" name="Maven: com.alibaba:fastjson:1.2.27" level="project" />
+ <orderEntry type="library" name="Maven: commons-logging:commons-logging:1.1" level="project" />
+ <orderEntry type="library" name="Maven: logkit:logkit:1.0.1" level="project" />
+ <orderEntry type="library" name="Maven: avalon-framework:avalon-framework:4.1.3" level="project" />
+ <orderEntry type="library" name="Maven: javax.servlet:servlet-api:2.3" level="project" />
+ <orderEntry type="library" name="Maven: commons-io:commons-io:2.5" level="project" />
+ <orderEntry type="library" name="Maven: log4j:log4j:1.2.17" level="project" />
+ <orderEntry type="library" name="Maven: com.google.code.gson:gson:2.8.5" level="project" />
+ <orderEntry type="library" name="Maven: com.google.auto.service:auto-service:1.0-rc5" level="project" />
+ <orderEntry type="library" name="Maven: com.google.auto.service:auto-service-annotations:1.0-rc5" level="project" />
+ <orderEntry type="library" name="Maven: com.google.auto:auto-common:0.10" level="project" />
+ <orderEntry type="library" name="Maven: com.google.guava:guava:25.1-jre" level="project" />
+ <orderEntry type="library" name="Maven: com.google.code.findbugs:jsr305:3.0.2" level="project" />
+ <orderEntry type="library" name="Maven: org.checkerframework:checker-qual:2.0.0" level="project" />
+ <orderEntry type="library" name="Maven: com.google.errorprone:error_prone_annotations:2.1.3" level="project" />
+ <orderEntry type="library" name="Maven: com.google.j2objc:j2objc-annotations:1.1" level="project" />
+ <orderEntry type="library" name="Maven: org.codehaus.mojo:animal-sniffer-annotations:1.14" level="project" />
+ <orderEntry type="library" name="Maven: com.lmax:disruptor:3.2.0" level="project" />
+ <orderEntry type="library" name="Maven: com.gliwka.hyperscan:hyperscan:5.4.0-2.0.0" level="project" />
+ <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:5.4.0-1.0.0" level="project" />
+ <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:linux-x86_64:5.4.0-1.0.0" level="project" />
+ <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:windows-x86_64:5.4.0-1.0.0" level="project" />
+ <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:macosx-x86_64:5.4.0-1.0.0" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp-platform:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-arm:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-arm64:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-x86:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-x86_64:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:ios-arm64:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:ios-x86_64:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-armhf:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-arm64:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-ppc64le:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-x86:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-x86_64:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:macosx-x86_64:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:windows-x86:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: org.bytedeco:javacpp:windows-x86_64:1.5.4" level="project" />
+ <orderEntry type="library" name="Maven: net.java.dev.jna:platform:3.5.2" level="project" />
+ <orderEntry type="library" name="Maven: net.java.dev.jna:jna:3.5.2" level="project" />
+ <orderEntry type="library" name="Maven: io.krakens:java-grok:0.1.9" level="project" />
+ <orderEntry type="library" name="Maven: org.codehaus.groovy:groovy-all:2.1.8" level="project" />
+ <orderEntry type="library" name="Maven: org.python:jython-standalone:2.7.0" level="project" />
+ <orderEntry type="module" module-name="rocketmq-streams-db-operator" />
+ <orderEntry type="module" module-name="rocketmq-streams-configurable" />
+ <orderEntry type="module" module-name="rocketmq-streams-serviceloader" />
+ <orderEntry type="library" name="Maven: org.springframework:spring-jdbc:3.2.13.RELEASE" level="project" />
+ <orderEntry type="library" name="Maven: org.springframework:spring-beans:3.2.13.RELEASE" level="project" />
+ <orderEntry type="library" name="Maven: org.springframework:spring-core:3.2.13.RELEASE" level="project" />
+ <orderEntry type="library" name="Maven: org.springframework:spring-tx:3.2.13.RELEASE" level="project" />
+ <orderEntry type="library" name="Maven: mysql:mysql-connector-java:5.1.40" level="project" />
+ </component>
+</module>
\ No newline at end of file
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/FilterComponent.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/FilterComponent.java
new file mode 100644
index 0000000..37384ef
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/FilterComponent.java
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter;
+
+import java.util.List;
+import java.util.Properties;
+
+import com.alibaba.fastjson.JSONObject;
+
+import org.apache.rocketmq.streams.filter.context.ContextConfigure;
+import org.apache.rocketmq.streams.filter.builder.RuleBuilder;
+import org.apache.rocketmq.streams.filter.context.RuleMessage;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+import org.apache.rocketmq.streams.filter.service.IRuleEngineService;
+import org.apache.rocketmq.streams.filter.service.impl.RuleEngineServiceImpl;
+import org.apache.rocketmq.streams.common.component.AbstractComponent;
+import org.apache.rocketmq.streams.common.component.ComponentCreator;
+import org.apache.rocketmq.streams.common.component.IgnoreNameSpace;
+import org.apache.rocketmq.streams.common.context.AbstractContext;
+import org.apache.rocketmq.streams.common.context.IMessage;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+/**
+ * 通过表达式，做过滤 表达式的表达形式(varName,functionName,value)&varName,functionName,datatypeName,value)
+ */
+public class FilterComponent extends AbstractComponent<IRuleEngineService> implements IRuleEngineService, IgnoreNameSpace {
+
+ private static final Log LOG = LogFactory.getLog(FilterComponent.class);
+
+ private static final FilterComponent filterComponent = ComponentCreator.getComponent(null, FilterComponent.class);
+
+ protected RuleEngineServiceImpl ruleEngineService;
+
+ public FilterComponent() {
+ }
+
+ public static FilterComponent getInstance() {
+ return filterComponent;
+ }
+
+ @Override
+ public boolean stop() {
+ return true;
+ }
+
+ @Override
+ public IRuleEngineService getService() {
+ return this.ruleEngineService;
+ }
+
+ @Override
+ protected boolean startComponent(String namespace) {
+ return true;
+ }
+
+ public static RuleEngineServiceImpl createRuleEngineService(Properties properties) {
+ RuleEngineServiceImpl ruleEngineService = new RuleEngineServiceImpl();
+ ContextConfigure config = new ContextConfigure(properties);
+ ruleEngineService.initRuleContext(config);
+ return ruleEngineService;
+ }
+
+ @Override
+ protected boolean initProperties(Properties properties) {
+ if (ruleEngineService != null) {
+ return true;
+ }
+ RuleEngineServiceImpl ruleEngineService = createRuleEngineService(properties);
+
+ this.ruleEngineService = ruleEngineService;
+ return true;
+ }
+
+ @Override
+ public Rule createRule(String namespace, String ruleName, String expressionStr, String... msgMetaInfo) {
+ RuleBuilder ruleCreator = new RuleBuilder(namespace, ruleName, expressionStr, msgMetaInfo);
+ Rule rule = ruleCreator.generateRule(null);
+ return rule;
+ }
+
+ @Override
+ public List<Rule> excuteRule(JSONObject message, Rule... rules) {
+ return ruleEngineService.excuteRule(message, rules);
+ }
+
+ @Override
+ public List<Rule> executeRule(IMessage message, AbstractContext context, Rule... rules) {
+ return ruleEngineService.executeRule(message, context, rules);
+ }
+
+ @Override
+ public List<Rule> excuteRule(RuleMessage message, Rule... rules) {
+ return ruleEngineService.excuteRule(message, rules);
+ }
+
+ @Override
+ public List<Rule> excuteRule(JSONObject message, List<Rule> rules) {
+ return ruleEngineService.excuteRule(message, rules);
+ }
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/builder/ExpressionBuilder.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/builder/ExpressionBuilder.java
new file mode 100644
index 0000000..c3356e4
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/builder/ExpressionBuilder.java
@@ -0,0 +1,409 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.builder;
+
+import com.alibaba.fastjson.JSONObject;
+
+import org.apache.rocketmq.streams.filter.optimization.ExpressionOptimization;
+import org.apache.rocketmq.streams.common.model.NameCreator;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+import org.apache.rocketmq.streams.filter.operator.expression.Expression;
+import org.apache.rocketmq.streams.filter.operator.expression.ExpressionRelationParser;
+import org.apache.rocketmq.streams.filter.operator.expression.RelationExpression;
+import org.apache.rocketmq.streams.filter.operator.expression.SimpleExpression;
+import org.apache.rocketmq.streams.filter.operator.var.Var;
+import org.apache.rocketmq.streams.common.cache.softreference.ICache;
+import org.apache.rocketmq.streams.common.cache.softreference.impl.SoftReferenceCache;
+import org.apache.rocketmq.streams.common.configurable.IConfigurable;
+import org.apache.rocketmq.streams.common.datatype.ListDataType;
+import org.apache.rocketmq.streams.common.utils.ContantsUtil;
+import org.apache.rocketmq.streams.common.utils.DataTypeUtil;
+import org.apache.rocketmq.streams.common.utils.MapKeyUtil;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * 必须先初始化组件后才可以使用
+ */
+public class ExpressionBuilder {
+ private static ListDataType listDataType = new ListDataType(DataTypeUtil.getDataTypeFromClass(String.class));
+ private static ICache<String, Rule> cache = new SoftReferenceCache<>();
+
+ /**
+ * 根据表达式，创建一个规则，规则本身可执行
+ *
+ * @param namespace
+ * @param expressionStr
+ * @return
+ */
+ public static Rule createRule(String namespace, String ruleName, String expressionStr) {
+ Rule rule = null;
+ String key = MapKeyUtil.createKey(namespace, expressionStr);
+ //rule = sinkcache.get(key);
+ if (rule != null) {
+ return rule;
+
+ }
+ List<Expression> expressions = new ArrayList<>();
+ List<RelationExpression> relationExpressions = new ArrayList<>();
+ Expression expression = createExpression(namespace, ruleName, expressionStr, expressions, relationExpressions);
+ ExpressionOptimization expressionOptimization = new ExpressionOptimization(expression, expressions, relationExpressions);
+ List<Expression> expressionList = expressionOptimization.optimizate();
+ rule = createRule(namespace, ruleName, expression, expressionList);
+ cache.put(key, rule);
+ return rule;
+ }
+
+ /**
+ * 可以简化表达式格式（varName,function,value)&((varName,function,datatype,value)|(varName,function,datatype,value))
+ *
+ * @param expressionStr
+ * @param msg
+ * @return
+ */
+ public static boolean executeExecute(String namespace, String expressionStr, JSONObject msg) {
+ Rule rule = null;
+ String key = expressionStr;
+ rule = cache.get(key);
+ if (rule == null) {
+ rule = createRule(namespace, Rule.class.getSimpleName(), expressionStr);
+ cache.put(key, rule);
+ }
+ return rule.execute(msg);
+
+ }
+
+ public static boolean executeExecute(Expression expression, JSONObject msg) {
+ Rule rule = null;
+ if (expression.getConfigureName() == null) {
+ expression.setConfigureName(expression.getVarName());
+ }
+ String key = MapKeyUtil.createKey(expression.getNameSpace(), expression.toString());
+ rule = cache.get(key);
+ if (rule == null) {
+ rule = createRule(expression.getNameSpace(), Rule.class.getSimpleName(), expression);
+ cache.put(key, rule);
+ }
+ return rule.execute(msg);
+ }
+
+ /**
+ * 创建一个表达式，并返回最顶层的表达式，中间的子表达式，全部relation表达式
+ *
+ * @param expressionStr
+ * @return
+ */
+ public static Expression createExpression(String namespace, String ruleName, String expressionStr, List<Expression> expressions,
+ List<RelationExpression> relationExpressions) {
+ String relationStr = parseExpression(namespace, ruleName, expressionStr, expressions);
+ if (expressions != null && expressions.size() == 1) {
+ if (expressions.get(0).getConfigureName().equals(relationStr)) {
+ return expressions.get(0);
+ }
+ }
+ Expression relationExpression =
+ ExpressionRelationParser.createRelations(namespace, ruleName, relationStr, relationExpressions);
+ if (relationExpressions != null) {
+ for (RelationExpression relation : relationExpressions) {
+ relation.setDataType(listDataType);
+ }
+ }
+ return relationExpression;
+ }
+
+ /**
+ * 创建表达式，并做初步的优化，把同关系的多层relation，拉平。如((a,==,b)&(c,==,d))&(e,==,f)=>(a,==,b)&(c,==,d)&(e,==,f)
+ *
+ * @param expressionStr
+ * @param expressions
+ * @param relationExpressions
+ * @return
+ */
+ public static Expression createOptimizationExpression(String namespace, String ruleName, String expressionStr, List<Expression> expressions,
+ List<RelationExpression> relationExpressions) {
+ Expression expression = createExpression(namespace, ruleName, expressionStr, expressions, relationExpressions);
+ ExpressionOptimization expressionOptimization = new ExpressionOptimization(expression, expressions, relationExpressions);
+ List<Expression> list = expressionOptimization.optimizate();
+ if (list == null || list.size() == 0) {
+ return expression;
+ }
+ expressions.clear();
+ relationExpressions.clear();
+ for (Expression express : list) {
+ if (RelationExpression.class.isInstance(express)) {
+ relationExpressions.add((RelationExpression)express);
+ } else {
+ expressions.add(express);
+ }
+ }
+ return expression;
+ }
+
+ public static Rule createRule(String namespace, String ruleName, Expression expression,
+ List<? extends Expression>... expressionLists) {
+ RuleBuilder ruleCreator = new RuleBuilder(expression.getNameSpace(), ruleName);
+ //RuleContext ruleContext=new RuleContext(msg,ruleCreator.createRule());
+ Rule rule = ruleCreator.getRule();
+ if (expressionLists != null) {
+ for (List<? extends Expression> expressions : expressionLists) {
+ if (expressions == null || expressions.size() == 0) {
+ continue;
+ }
+ for (Expression expression1 : expressions) {
+ createExpression(expression1, ruleCreator);
+ }
+ addConfigurable2Map(rule.getExpressionMap(), ruleCreator.getExpressionList());
+ }
+ }
+ createExpression(expression, ruleCreator);
+
+ addConfigurable2Map(rule.getVarMap(), ruleCreator.getVarList());
+ addConfigurable2Map(rule.getExpressionMap(), ruleCreator.getExpressionList());
+ addConfigurable2Map(rule.getMetaDataMap(), ruleCreator.getMetaDataList());
+ addConfigurable2Map(rule.getDataSourceMap(), ruleCreator.getDataSourceList());
+ addConfigurable2Map(rule.getActionMap(), ruleCreator.getActionList());
+ ruleCreator.setRootExpression(expression);
+ rule = ruleCreator.createRule();
+ ruleCreator.getMetaData().toObject(ruleCreator.getMetaData().toJson());//metadata的一个bug，如果不做这步，map为空。后续版本修复后可以去掉
+ return rule;
+ }
+
+ /**
+ * 一个表达式，需要的metafield，变量，value
+ *
+ * @param expression
+ * @param ruleCreator
+ */
+ private static void createExpression(Expression expression, RuleBuilder ruleCreator) {
+ if (RelationExpression.class.isInstance(expression)) {
+ createRelationExpression(expression, ruleCreator);
+ } else {
+ createSingleExpression(expression, ruleCreator);
+ }
+ }
+
+ /**
+ * @param expression
+ * @param ruleCreator
+ */
+ private static void createSingleExpression(Expression expression, RuleBuilder ruleCreator) {
+ String varName = expression.getVarName();
+ ruleCreator.addVarAndMetaDataField(varName);
+ ruleCreator.addExpression(expression);
+ }
+
+ private static void createRelationExpression(Expression expression, RuleBuilder ruleCreator) {
+ if (!RelationExpression.class.isInstance(expression)) {
+ return;
+ }
+ ruleCreator.addExpression(expression);
+ }
+
+ private static <T extends IConfigurable> void addConfigurable2Map(Map<String, T> map, List<T> list) {
+ if (map == null || list == null) {
+ return;
+ }
+ for (T t : list) {
+ if (Var.class.isInstance(t)) {
+ map.put(((Var)t).getVarName(), t);
+ } else {
+ map.put(t.getConfigureName(), t);
+ }
+
+ }
+ }
+
+ /**
+ * 把一个表达式组合，进行解析，返回一个关系字符串和一组表达式子列表expressions
+ *
+ * @param namespace
+ * @param expressionStr
+ * @param expressions
+ * @return
+ */
+ private static String parseExpression(String namespace, String ruleName, String expressionStr, List<Expression> expressions) {
+ return parseExpression(namespace, ruleName, expressionStr, expressions, ruleExpressionCreator);
+ }
+
+ public static void main(String[] args) {
+
+ }
+
+ /**
+ * 把一个表达式组合，进行解析，返回一个关系字符串和一组表达式子列表expressions
+ *
+ * @param namespace
+ * @param expressionStr
+ * @param expressions
+ * @return
+ */
+ public static String parseExpression(String namespace, String ruleName, String expressionStr, List<Expression> expressions,
+ IRuleExpressionCreator creator) {
+ if (StringUtil.isEmpty(expressionStr)) {
+ return null;
+ }
+
+ expressionStr = expressionStr.replaceAll("\r\n", "");
+ expressionStr = expressionStr.replaceAll("\n", "");
+
+ expressionStr = ContantsUtil.replaceSpeciaSign(expressionStr);
+
+ Map<String, String> flag2ExpressionStr = new HashMap<>();
+ boolean containsContant = ContantsUtil.containContant(expressionStr);
+ if (containsContant) {
+ expressionStr = ContantsUtil.doConstantReplace(expressionStr, flag2ExpressionStr, 1);
+ }
+ String relationStr = expressionStr;
+ NameCreator nameCreator = NameCreator.createOrGet(ruleName);
+ if (expressionStr.indexOf("(") == -1) {
+ relationStr = creator.createExpression(namespace, ruleName, expressionStr, expressions, containsContant, flag2ExpressionStr, nameCreator, relationStr);
+ return relationStr;
+ }
+
+ Map<String, String> expressionFlags = new HashMap<>();
+ relationStr = parseExpression(namespace, ruleName, expressionStr, expressions, expressionFlags, creator, containsContant, nameCreator, flag2ExpressionStr);
+ boolean needReplace = true;
+ while (needReplace) {
+ String tmp = ContantsUtil.restore(relationStr, expressionFlags);
+ if (tmp.equals(relationStr)) {
+ relationStr = tmp;
+ break;
+ } else {
+ relationStr = tmp;
+ }
+ }
+ return relationStr;
+ // for (int i = 0; i < expressionStr.length(); i++) {
+ // String word = expressionStr.substring(i, i + 1);
+ //
+ // if ("(".equals(word) ) {
+ // startExpression = true;
+ // continue;
+ // }
+ // if (")".equals(word)) {
+ // if (startExpression) {
+ // String expresionStr = expressionSb.toString();
+ // relationStr=creator.createExpression(namespace,ruleName,expresionStr,expressions,containsContant,flag2ExpressionStr,nameCreator,relationStr);
+ // expressionSb = new StringBuilder();
+ // }
+ // startExpression = false;
+ // continue;
+ // }
+ // if (startExpression) {
+ // expressionSb.append(word);
+ // }
+ // }
+ // return relationStr;
+ }
+
+ /**
+ * 把表达式中（a,b,c）部分解析出来，只留下关系部分
+ *
+ * @return 返回的是关系串
+ */
+ protected static String parseExpression(String namespace, String ruleName, String relationStr, List<Expression> expressions, Map<String, String> flag2ExpressionStr, IRuleExpressionCreator creator, boolean containsContant, NameCreator nameCreator, Map<String, String> contantsFlags) {
+ int endIndex = relationStr.indexOf(")");
+ if (endIndex == -1) {
+ return relationStr;
+ }
+ if (relationStr.indexOf(",") == -1) {
+ return relationStr;
+ }
+ for (int i = endIndex; i > 0; i--) {
+ String word = relationStr.substring(i - 1, i);
+ if (word.equals("(")) {
+ String expressionStr = relationStr.substring(i, endIndex);
+ String oriStr = "(" + expressionStr + ")";
+ if (expressionStr.indexOf(",") != -1) {
+ relationStr = creator.createExpression(namespace, ruleName, expressionStr, expressions, containsContant, contantsFlags, nameCreator, relationStr);
+ relationStr = parseExpression(namespace, ruleName, relationStr, expressions, flag2ExpressionStr, creator, containsContant, nameCreator, contantsFlags);
+ return relationStr;
+ } else {
+
+ String relationFlag = NameCreator.createNewName("relation");
+ flag2ExpressionStr.put(relationFlag, oriStr);
+ relationStr = relationStr.replace(oriStr, relationFlag);
+ return parseExpression(namespace, ruleName, relationStr, expressions, flag2ExpressionStr, creator, containsContant, nameCreator, contantsFlags);
+ }
+ }
+ }
+ return relationStr;
+ }
+
+ interface IRuleExpressionCreator {
+
+ String createExpression(String namespace, String ruleName, String expresionStr, List<Expression> expressions,
+ boolean containsContant,
+ Map<String, String> flag2ExpressionStr, NameCreator nameCreator, String relationStr);
+ }
+
+ protected static IRuleExpressionCreator ruleExpressionCreator = new IRuleExpressionCreator() {
+ @Override
+ public String createExpression(String namespace, String ruleName, String expresionStr,
+ List<Expression> expressions, boolean containsContant, Map<String, String> flag2ExpressionStr,
+ NameCreator nameCreator, String relationStr) {
+ String[] values = ExpressionBuilder.createElement(expresionStr, containsContant, flag2ExpressionStr);
+ //expresionStr= ContantsUtil.restore(expresionStr,flag2ExpressionStr);
+ String expressionName = nameCreator.createName(ruleName);
+
+ SimpleExpression expression = null;
+ if (values.length == 3) {
+ String value = values[2];
+ expression = new SimpleExpression(values[0], values[1], value);
+
+ }
+ if (values.length == 4) {
+ String value = values[3];
+ expression =
+ new SimpleExpression(values[0], values[1], DataTypeUtil.getDataType(values[2]), value);
+ }
+ if (expression != null) {
+ expression.setNameSpace(namespace);
+ expression.setConfigureName(expressionName);
+ if (relationStr.indexOf("(") == -1) {
+ relationStr = relationStr.replace(expresionStr, expressionName);
+ } else {
+ relationStr = relationStr.replace("(" + expresionStr + ")", expressionName);
+ }
+
+ expressions.add(expression);
+ }
+ return relationStr;
+ }
+ };
+
+ public static String[] createElement(String expresionStr, boolean containsContant, Map<String, String> flag2ExpressionStr) {
+ String[] values = expresionStr.split(",");
+ int i = 0;
+ for (String value : values) {
+ if (containsContant) {
+ value = ContantsUtil.restore(value, flag2ExpressionStr);
+ }
+ if (value != null && ContantsUtil.isContant(value)) {
+ value = value.substring(1, value.length() - 1);
+ values[i] = value;
+ }
+ values[i] = ContantsUtil.restoreSpecialSign(values[i]);
+ i++;
+ }
+ return values;
+ }
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/builder/RuleBuilder.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/builder/RuleBuilder.java
new file mode 100644
index 0000000..17ef312
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/builder/RuleBuilder.java
@@ -0,0 +1,592 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.builder;
+
+import org.apache.rocketmq.streams.common.channel.sink.ISink;
+import org.apache.rocketmq.streams.common.model.NameCreator;
+import org.apache.rocketmq.streams.db.driver.JDBCDriver;
+import org.apache.rocketmq.streams.filter.operator.action.Action;
+import org.apache.rocketmq.streams.filter.operator.action.impl.MetaDataAction;
+import org.apache.rocketmq.streams.filter.contants.RuleElementType;
+import org.apache.rocketmq.streams.filter.operator.expression.ExpressionRelationParser;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+import org.apache.rocketmq.streams.filter.operator.expression.Expression;
+import org.apache.rocketmq.streams.filter.operator.expression.RelationExpression;
+import org.apache.rocketmq.streams.filter.operator.var.ConstantVar;
+import org.apache.rocketmq.streams.filter.operator.var.ContextVar;
+import org.apache.rocketmq.streams.filter.operator.var.InnerVar;
+import org.apache.rocketmq.streams.filter.operator.var.Var;
+import org.apache.rocketmq.streams.common.configurable.IConfigurable;
+import org.apache.rocketmq.streams.common.configurable.IConfigurableService;
+import org.apache.rocketmq.streams.common.datatype.DataType;
+import org.apache.rocketmq.streams.common.metadata.MetaData;
+import org.apache.rocketmq.streams.common.metadata.MetaDataField;
+import org.apache.rocketmq.streams.common.utils.DataTypeUtil;
+import org.apache.rocketmq.streams.common.utils.MapKeyUtil;
+
+import java.util.*;
+
+/**
+ * 通过这个工具可以快速创建一条规则。这个工具默认消息流的字段名＝metadata的字段名
+ */
+public class RuleBuilder {
+ private static final DataType STRING = DataTypeUtil.getDataTypeFromClass(String.class);
+ private Rule rule = new Rule();
+ private List<Var> varList = new ArrayList<>();//变量列表，包含这个规则用到的所有变量
+ private List<MetaData> metaDataList = new ArrayList<>();//包含所有的metadata
+ private List<Expression> expressionList = new ArrayList<>();//包含所有的表达式
+ private List<Action> actionList = new ArrayList<>();//包含所有的action
+ private List<JDBCDriver> dataSourceList = new ArrayList<>();//包含所有的datasource
+ private MetaData metaData;//输入消息的metadata
+ private String namespace;//规则的命名空间
+ private String ruleName;//规则的名字
+ private Expression rootExpression;//最外层的表达式子
+ private String ruleCode;
+ private String ruleTitle;
+ private String ruleDescription;
+ private transient NameCreator actionNameCreator = new NameCreator();
+ private transient NameCreator dataSourceNameCreator = new NameCreator();
+ private transient NameCreator metaDataNameCreator = new NameCreator();
+
+ /**
+ * 创建个一个规则creator，在这个阶段会创建消息的meta，变量，以及表达式，支持多个表达式的关系操作，适合表达式比较少的时候
+ *
+ * @param namespace 规则的命名空间
+ * @param ruleName 规则名字
+ * @param expressionStr 格式如下（varname,functionName,datatype,value)&((varname,functionName,value)|(varname, functionName,value))
+ * @param msgMetaInfo 主要式表述消息的格式。格式如下：msgFieldName;int;true。如果最后以为是false，或第二位是string，可以省略最后两位。 格式如下：msgFieldnName。第二位可以根据datatype.getDataTypeName获取
+ */
+ public RuleBuilder(String namespace, String ruleName, String expressionStr, String... msgMetaInfo) {
+ init(namespace, ruleName);
+ List<Expression> expressions = new ArrayList<>();
+ List<RelationExpression> relationExpressions = new ArrayList<>();
+ rule.setExpressionStr(expressionStr);
+ Expression expression =
+ ExpressionBuilder.createExpression(namespace, ruleName, expressionStr, expressions, relationExpressions);
+ this.rootExpression = expression;
+ expressionList.addAll(expressions);
+ expressionList.addAll(relationExpressions);
+ if (msgMetaInfo == null || msgMetaInfo.length == 0) {
+ for (Expression express : expressionList) {
+ Var var = RuleElementBuilder.createContextVar(namespace, ruleName, express.getVarName(),
+ null, express.getVarName());
+ var.setConfigureName(express.getVarName());
+ varList.add(var);
+ }
+ return;
+ }
+ MetaData metaData =
+ RuleElementBuilder.createMetaData(namespace, metaDataNameCreator.createName(), msgMetaInfo);
+ List<MetaDataField> metaDataFields = metaData.getMetaDataFields();
+ Set<String> existVarNames = new HashSet<>();//
+ for (MetaDataField metaDataField : metaDataFields) {
+ Var var = RuleElementBuilder.createContextVar(namespace, ruleName, metaDataField.getFieldName(),
+ metaData.getConfigureName(), metaDataField.getFieldName());
+ var.setConfigureName(metaDataField.getFieldName());
+ varList.add(var);
+ existVarNames.add(var.getConfigureName());
+ }
+ for (Expression express : expressionList) {
+ if (existVarNames.contains(express.getVarName())) {
+ continue;
+ }
+ if (RelationExpression.class.isInstance(express)) {
+ continue;
+ }
+ Var var = RuleElementBuilder.createContextVar(namespace, ruleName, express.getVarName(),
+ null, express.getVarName());
+ var.setConfigureName(express.getVarName());
+ varList.add(var);
+ }
+ metaData.toObject(metaData.toJson());
+ this.metaData = metaData;
+ if (metaData != null) {
+ this.rule.setMsgMetaDataName(metaData.getConfigureName());
+ }
+ metaDataList.add(this.metaData);
+ }
+
+ /**
+ * 创建规则
+ *
+ * @param namespace 规则的命名空间
+ * @param ruleName 规则名称
+ */
+ public RuleBuilder(String namespace, String ruleName) {
+ init(namespace, ruleName);
+ createChannelMetaData();
+ }
+
+ public Rule getRule() {
+ return rule;
+ }
+
+ public RuleBuilder(IConfigurableService ruleEngineConfigurableService, String namespace, String ruleName) {
+ Rule rule =
+ (Rule)ruleEngineConfigurableService.queryConfigurableByIdent(RuleElementType.RULE.getType(), ruleName);
+ if (rule == null) {
+ init(namespace, ruleName);
+ createChannelMetaData();
+ return;
+ }
+ this.metaData =
+ (MetaData)ruleEngineConfigurableService.queryConfigurableByIdent(RuleElementType.METADATA.getType(),
+ rule.getMsgMetaDataName());
+ this.rootExpression =
+ (Expression)ruleEngineConfigurableService.queryConfigurableByIdent(RuleElementType.EXPRESSION.getType(),
+ rule.getExpressionName());
+ this.rule = rule;
+ this.ruleName = rule.getConfigureName();
+ this.namespace = rule.getNameSpace();
+ this.ruleTitle = rule.getRuleTitle();
+ this.ruleCode = rule.getRuleCode();
+ this.ruleDescription = rule.getRuleDesc();
+ this.varList = ruleEngineConfigurableService.queryConfigurableByType(RuleElementType.VAR.getType());
+ this.expressionList =
+ ruleEngineConfigurableService.queryConfigurableByType(RuleElementType.EXPRESSION.getType());
+ this.metaDataList = ruleEngineConfigurableService.queryConfigurableByType(RuleElementType.METADATA.getType());
+ this.actionList = ruleEngineConfigurableService.queryConfigurableByType(RuleElementType.ACTION.getType());
+ this.dataSourceList =
+ ruleEngineConfigurableService.queryConfigurableByType(RuleElementType.DATASOURCE.getType());
+ }
+
+ /**
+ * 创建规则必须的信息。
+ *
+ * @param namespace
+ * @param ruleName
+ * @param rulecode_title_description 最多3个，依次是rulecode，ruletitle，ruledescription
+ */
+ private void init(String namespace, String ruleName, String... rulecode_title_description) {
+ rule.setNameSpace(namespace);
+ rule.setConfigureName(ruleName);
+ rule.setRuleStatus(3);
+ this.namespace = namespace;
+ this.ruleName = ruleName;
+
+ if (rulecode_title_description != null && rulecode_title_description.length > 0) {
+ rule.setRuleCode(rulecode_title_description[0]);
+ this.ruleCode = rule.getRuleCode();
+ if (rulecode_title_description.length > 1) {
+ rule.setRuleTitle(rulecode_title_description[1]);
+ this.ruleTitle = rule.getRuleTitle();
+ }
+ if (rulecode_title_description.length > 2) {
+ rule.setRuleDesc(rulecode_title_description[2]);
+ ruleDescription = rule.getRuleDesc();
+ }
+ }
+ }
+
+ /**
+ * 快速创建消息流的meta。每一个表述一个metafield
+ *
+ * @param fieldNameTypeIsRequireds 格式如下：msgkeyName:type;isRequired。 如果最后以为是false，或第二位是string，可以省略最后两位。 格式如下：msgFieldnName。第二位可以根据datatype.getDataTypeName获取
+ */
+ public RuleBuilder addChannelMetaDataField(String... fieldNameTypeIsRequireds) {
+ if (metaData == null) {
+ throw new RuntimeException("need create metaData first");
+ }
+ if (metaData.getMetaDataFields() == null) {
+ metaData.setMetaDataFields(new ArrayList<MetaDataField>());
+ }
+ this.metaData.getMetaDataFields().addAll(createMetaDataField(fieldNameTypeIsRequireds));
+ return this;
+ }
+
+ /**
+ * 快速创建消息流的meta。每一个表述一个metafield
+ *
+ * @param fieldNameTypeIsRequireds 格式如下：msgkeyName:type;isRequired。 如果最后以为是false，或第二位是string，可以省略最后两位。 格式如下：msgFieldnName。第二位可以根据datatype.getDataTypeName获取
+ */
+ private List<MetaDataField> createMetaDataField(String... fieldNameTypeIsRequireds) {
+ MetaData metaData = RuleElementBuilder.createMetaData("tmp", "tmp", fieldNameTypeIsRequireds);
+ return metaData.getMetaDataFields();
+ }
+
+ /**
+ * 可以同时创建变量和metadatafield。类型是字符，可空
+ *
+ * @param fieldNameTypeIsRequireds
+ */
+ public RuleBuilder addVarAndMetaDataField(String... fieldNameTypeIsRequireds) {
+ List<MetaDataField> metaDataFields = createMetaDataField(fieldNameTypeIsRequireds);
+ this.metaData.getMetaDataFields().addAll(metaDataFields);
+ for (MetaDataField metaDataField : metaDataFields) {
+ ContextVar var = RuleElementBuilder.createContextVar(namespace, ruleName, metaDataField.getFieldName(),
+ metaData.getConfigureName(), metaDataField.getFieldName());
+ varList.add(var);
+ }
+ return this;
+ }
+
+ public RuleBuilder addInnerVar(String varName) {
+ InnerVar innerVar = new InnerVar();
+ innerVar.setNameSpace(ruleName);
+ innerVar.setVarName(varName);
+ innerVar.setType(RuleElementType.VAR.getType());
+ varList.add(innerVar);
+ return this;
+ }
+
+ /**
+ * 支持增加常量，常量类型是字符
+ *
+ * @param varName
+ * @param value
+ * @return
+ */
+ public RuleBuilder addConstantAndMetaField(String varName, String value) {
+ return addConstantAndMetaField(varName, STRING, value);
+ }
+
+ /**
+ * 支持增加常量，常量类型可以指定
+ *
+ * @param varName
+ * @param value
+ * @return
+ */
+ public RuleBuilder addConstantAndMetaField(String varName, DataType dataType, String value) {
+ ConstantVar var = RuleElementBuilder.createConstantVar(namespace, ruleName, varName, dataType, value);
+ varList.add(var);
+ MetaDataField metaDataField = new MetaDataField();
+ metaDataField.setFieldName(varName);
+ metaDataField.setDataType(dataType);
+ metaDataField.setIsRequired(false);
+ this.metaData.getMetaDataFields().add(metaDataField);
+ return this;
+ }
+
+ public RuleBuilder setExpression(String expressionStr) {
+ List<Expression> expressions = new ArrayList<>();
+ List<RelationExpression> relationExpressions = new ArrayList<>();
+ Expression expression =
+ ExpressionBuilder.createExpression(namespace, ruleName, expressionStr, expressions, relationExpressions);
+ this.rootExpression = expression;
+ this.expressionList.addAll(expressions);
+ this.expressionList.addAll(relationExpressions);
+ return this;
+ }
+
+ /**
+ * 创建表达式，包括表达式名字，变量名，函数和值。值是字符类型
+ *
+ * @param expressionName
+ * @param varName
+ * @param functionName
+ * @param value
+ * @return
+ */
+ public RuleBuilder addExpression(String expressionName, String varName, String functionName, String value) {
+ addExpression(expressionName, varName, functionName, STRING, value);
+ return this;
+ }
+
+ public RuleBuilder addExpression(Expression expression) {
+ expressionList.add(expression);
+ return this;
+ }
+
+ /**
+ * 创建表达式，包括表达式名字，变量名，函数和值。值可以指定
+ *
+ * @param expressionName
+ * @param varName
+ * @param functionName
+ * @param value
+ * @return
+ */
+ public RuleBuilder addExpression(String expressionName, String varName, String functionName, DataType dataType,
+ String value) {
+ Expression expression =
+ RuleElementBuilder.createExpression(ruleName, expressionName, varName, functionName, dataType, value);
+ expressionList.add(expression);
+ return this;
+ }
+
+ /**
+ * 通过 表达式的名字，做关联关系。如表达式的名字为1，2，3。关系可以写为1&(2｜3)
+ *
+ * @param relationStr
+ * @return
+ */
+ public RuleBuilder addRelationExpression(String relationStr) {
+ List<RelationExpression> relationExpressions = new ArrayList<>();
+ RelationExpression relationExpression =
+ ExpressionRelationParser.createRelations(ruleName, ruleName, relationStr, relationExpressions);
+ this.expressionList.addAll(relationExpressions);
+ this.rootExpression = relationExpression;
+ return this;
+ }
+ //
+ ///**
+ // * @param url
+ // * @param userName
+ // * @param password
+ // * @param kvs: varName:fieldName,varName:fieldName
+ // * @return
+ // */
+ //public RuleBuilder addDBAction(String url, String userName, String password, String table, String... kvs) {
+ // return addDBAction(url, userName, password, table, createVarName2FieldName(kvs));
+ //}
+ //
+ //public RuleBuilder addChannelAction(IConvertDataSource convertDataSource) {
+ // Map<String, String> varName2FieldName = new HashMap<>();
+ // List<MetaDataField> metaDataFields = metaData.getMetaDataFields();
+ // for (MetaDataField field : metaDataFields) {
+ // varName2FieldName.put(field.getFieldName(), field.getFieldName());
+ // }
+ // IDataOperator dataSource = convertDataSource.convert();
+ // dataSourceList.add(dataSource);
+ // String metaDataName = metaDataNameCreator.createName();
+ // String actionName = actionNameCreator.createName();
+ // MetaData metaData =
+ // RuleElementBuilder.createMetaData(namespace, metaDataName, createMetaDataFiledStr(varName2FieldName));
+ // metaData.setDataSourceName(dataSource.getConfigureName());
+ // metaDataList.add(metaData);
+ // MetaDataAction action =
+ // RuleElementBuilder.createAction(namespace, actionName, metaDataName, varName2FieldName);
+ // actionList.add(action);
+ // return this;
+ //}
+
+ public RuleBuilder addMetaDataAction(String namespace, String metaDataName, Map<String, String> varName2FieldName) {
+ initVarName2FieldName(varName2FieldName);
+ String actionName = actionNameCreator.createName();
+ MetaDataAction action =
+ RuleElementBuilder.createAction(namespace, actionName, metaDataName, varName2FieldName);
+ actionList.add(action);
+ return this;
+ }
+
+ /**
+ * 创建输出，可以把触发规则的数据写入到指定的为止，此方法主要是db
+ *
+ * @param varName2FieldName 消息中的字段名和数据表的名字映射
+ * @param url
+ * @param userName
+ * @param password
+ * @return
+ */
+ public RuleBuilder addDBAction(String url, String userName, String password, String tableName,
+ Map<String, String> varName2FieldName) {
+ initVarName2FieldName(varName2FieldName);
+
+ JDBCDriver dataSource = new JDBCDriver();
+ dataSource.setUrl(url);
+ dataSource.setNameSpace(namespace);
+ dataSource.setUserName(userName);
+ dataSource.setPassword(password);
+ dataSource.setJdbcDriver("com.mysql.jdbc.Driver");
+ dataSource.setType(ISink.TYPE);
+ dataSource.setConfigureName(dataSourceNameCreator.createName());
+ dataSourceList.add(dataSource);
+
+ String metaDataName = metaDataNameCreator.createName();
+ String actionName = actionNameCreator.createName();
+ MetaData metaData =
+ RuleElementBuilder.createMetaData(namespace, metaDataName, createMetaDataFiledStr(varName2FieldName));
+ metaData.setTableName(tableName);
+ metaData.setDataSourceName(dataSource.getConfigureName());
+ metaDataList.add(metaData);
+ MetaDataAction action =
+ RuleElementBuilder.createAction(namespace, actionName, metaDataName, varName2FieldName);
+ actionList.add(action);
+ return this;
+ }
+
+ /**
+ * 保存规则相关的对象
+ *
+ * @param ruleEngineConfigurableService
+ * @return
+ */
+ public Rule generateRule(IConfigurableService ruleEngineConfigurableService) {
+ Rule rule = createRule();
+ insertOrUpdate(ruleEngineConfigurableService);
+ return rule;
+ }
+
+ private void insertOrUpdate(IConfigurableService ruleEngineConfigurableService) {
+ insertOrUpdate(ruleEngineConfigurableService, metaDataList, MetaData.TYPE);
+ insertOrUpdate(ruleEngineConfigurableService, varList, Var.TYPE);
+ insertOrUpdate(ruleEngineConfigurableService, expressionList, Expression.TYPE);
+ insertOrUpdate(ruleEngineConfigurableService, actionList, Action.TYPE);
+ insertOrUpdate(ruleEngineConfigurableService, dataSourceList, ISink.TYPE);
+ if (ruleEngineConfigurableService != null) {
+ ruleEngineConfigurableService.insert(rule);
+ }
+ }
+
+ /**
+ * 完成规则创建,但还不能使用。
+ *
+ * @return
+ */
+ protected Rule createRule() {
+ if (rootExpression == null && expressionList != null && expressionList.size() > 0) {
+ rootExpression = expressionList.get(0);
+ }
+ rule.setExpressionName(rootExpression.getConfigureName());
+ rule.setActionNames(convert(actionList));
+ rule.setVarNames(convert(varList));
+ rule.setRuleCode(ruleCode);
+ rule.setRuleTitle(ruleTitle);
+ rule.setRuleDesc(ruleDescription);
+ if (this.metaData != null) {
+ rule.setMsgMetaDataName(metaData.getConfigureName());
+ }
+ return rule;
+ }
+
+ private Map<String, String> createVarName2FieldName(String... kvs) {
+ Map<String, String> varName2FieldNames = null;
+ if (kvs != null) {
+ varName2FieldNames = new HashMap<>();
+ for (String varName2FieldName : kvs) {
+ String[] values = varName2FieldName.split(":");
+ varName2FieldNames.put(values[0], values[1]);
+ }
+ return varName2FieldNames;
+ }
+ initVarName2FieldName(varName2FieldNames);
+ return varName2FieldNames;
+ }
+
+ private void initVarName2FieldName(Map<String, String> varName2FieldNames) {
+ if (varName2FieldNames == null) {
+ List<MetaDataField> metaDataFields = metaData.getMetaDataFields();
+ for (MetaDataField field : metaDataFields) {
+ varName2FieldNames.put(field.getFieldName(), field.getFieldName());
+ }
+ }
+ }
+
+ private String[] createMetaDataFiledStr(Map<String, String> varName2FieldName) {
+ if (varName2FieldName == null) {
+ return new String[0];
+ }
+ String[] filedStr = new String[varName2FieldName.size()];
+ Iterator<Map.Entry<String, String>> it = varName2FieldName.entrySet().iterator();
+ metaData.toObject(metaData.toJson());
+ int i = 0;
+ while (it.hasNext()) {
+ Map.Entry<String, String> entry = it.next();
+ String varName = entry.getKey();
+ String fieldName = entry.getValue();
+ DataType dataType = metaData.getMetaDataField(varName).getDataType();
+ boolean isRequired = metaData.getMetaDataField(varName).getIsRequired();
+ filedStr[i] = MapKeyUtil.createKey(fieldName, dataType.getDataTypeName(), String.valueOf(isRequired));
+ i++;
+ }
+ return filedStr;
+ }
+
+ private <T extends IConfigurable> void insertOrUpdate(IConfigurableService ruleEngineConfigurableService,
+ List<T> configurables, String type) {
+ if (configurables == null) {
+ return;
+ }
+ for (IConfigurable configurable : configurables) {
+ if (ruleEngineConfigurableService != null) {
+ ruleEngineConfigurableService.insert(configurable);
+ }
+ rule.putConfigurableMap(configurable, type);
+ }
+ }
+
+ private <T extends IConfigurable> List<String> convert(List<T> configurables) {
+ List<String> configurableNames = new ArrayList<>();
+ for (T t : configurables) {
+ configurableNames.add(t.getConfigureName());
+ }
+ return configurableNames;
+ }
+
+ /**
+ * 创建规则默认创建一个消息流对应的metadata
+ *
+ * @return
+ */
+ private MetaData createChannelMetaData() {
+ MetaData metaData = RuleElementBuilder.createMetaData(namespace, metaDataNameCreator.createName());
+ metaDataList.add(metaData);
+ this.metaData = metaData;
+ return metaData;
+ }
+
+ public MetaData getMetaData() {
+ return metaData;
+ }
+
+ public List<Var> getVarList() {
+ return varList;
+ }
+
+ public List<MetaData> getMetaDataList() {
+ return metaDataList;
+ }
+
+ public List<Expression> getExpressionList() {
+ return expressionList;
+ }
+
+ public List<Action> getActionList() {
+ return actionList;
+ }
+
+ public List<JDBCDriver> getDataSourceList() {
+ return dataSourceList;
+ }
+
+ public String getNamespace() {
+ return namespace;
+ }
+
+ public String getRuleName() {
+ return ruleName;
+ }
+
+ public Expression getRootExpression() {
+ return rootExpression;
+ }
+
+ public void setRootExpression(Expression rootExpression) {
+ this.rootExpression = rootExpression;
+ }
+
+ public void setRuleName(String ruleName) {
+ this.ruleName = ruleName;
+ }
+
+ public String getRuleCode() {
+ return ruleCode;
+ }
+
+ public void setRuleCode(String ruleCode) {
+ this.ruleCode = ruleCode;
+ }
+
+ public String getRuleTitle() {
+ return ruleTitle;
+ }
+
+ public void setRuleTitle(String ruleTitle) {
+ this.ruleTitle = ruleTitle;
+ }
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/builder/RuleElementBuilder.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/builder/RuleElementBuilder.java
new file mode 100644
index 0000000..1d7edba
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/builder/RuleElementBuilder.java
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.builder;
+
+import org.apache.rocketmq.streams.filter.operator.action.impl.MetaDataAction;
+import org.apache.rocketmq.streams.filter.contants.RuleElementType;
+import org.apache.rocketmq.streams.filter.operator.expression.Expression;
+import org.apache.rocketmq.streams.filter.operator.expression.RelationExpression;
+import org.apache.rocketmq.streams.filter.operator.var.ConstantVar;
+import org.apache.rocketmq.streams.filter.operator.var.ContextVar;
+import org.apache.rocketmq.streams.common.datatype.DataType;
+import org.apache.rocketmq.streams.common.metadata.MetaData;
+import org.apache.rocketmq.streams.common.metadata.MetaDataField;
+import org.apache.rocketmq.streams.common.utils.DataTypeUtil;
+import org.apache.rocketmq.streams.common.utils.MapKeyUtil;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+public class RuleElementBuilder {
+ /**
+ * @param namespace
+ * @param name
+ * @param fields 格式：fieldname;datatypename;isRequired。如isRequired==false, 最后部分可以省略，如果datatypename＝＝string，且isRequired==false，可以只写fieldname
+ * @return
+ */
+ public static MetaData createMetaData(String namespace, String name, String... fields) {
+ MetaData metaData = new MetaData();
+ metaData.setNameSpace(namespace);
+ metaData.setConfigureName(name);
+ if (fields == null || fields.length == 0) {
+ metaData.toObject(metaData.toJson());
+ return metaData;
+ }
+ List<MetaDataField> metaDataFieldList = new ArrayList<>();
+ for (String field : fields) {
+ MetaDataField metaDataField = new MetaDataField();
+ String sign = ":";
+ if (field.indexOf(sign) == -1) {
+ sign = ";";
+ }
+ String[] values = field.split(sign);
+ String fieldName = values[0];
+ metaDataField.setFieldName(fieldName);
+ DataType dataType = DataTypeUtil.getDataTypeFromClass(String.class);
+ boolean isRequired = false;
+ if (values.length > 1) {
+ String dataTypeName = values[1];
+ dataType = DataTypeUtil.getDataType(dataTypeName);
+
+ }
+ metaDataField.setDataType(dataType);
+ if (values.length > 2) {
+ isRequired = Boolean.valueOf(values[2]);
+
+ }
+ metaDataField.setIsRequired(isRequired);
+ metaDataFieldList.add(metaDataField);
+ }
+ metaData.getMetaDataFields().addAll(metaDataFieldList);
+ metaData.toObject(metaData.toJson());
+ return metaData;
+ }
+
+ public static ContextVar createContextVar(String namespace, String ruleName, String varName, String metaDataName, String fieldName) {
+ ContextVar contextVar = new ContextVar();
+ contextVar.setNameSpace(namespace);
+ contextVar.setType(RuleElementType.VAR.getType());
+ contextVar.setVarName(varName);
+ contextVar.setConfigureName(MapKeyUtil.createKeyBySign("_", ruleName, varName));
+ contextVar.setMetaDataName(metaDataName);
+ contextVar.setFieldName(fieldName);
+ return contextVar;
+ }
+
+ public static ConstantVar createConstantVar(String namespace, String ruleName, String varName, DataType dataType, String value) {
+ ConstantVar constantVar = new ConstantVar();
+ constantVar.setNameSpace(ruleName);
+ constantVar.setDataType(dataType);
+ constantVar.setVarName(varName);
+ constantVar.setConfigureName(MapKeyUtil.createKeyBySign("_", ruleName, varName));
+ constantVar.setValue(dataType.getData(value));
+ constantVar.setType(RuleElementType.VAR.getType());
+ return constantVar;
+ }
+
+ public static Expression createExpression(String ruleName, String expressionName, String varName,
+ String functionName, DataType dataType, String value) {
+ Expression expression = new Expression();
+ expression.setNameSpace(ruleName);
+ expression.setType(RuleElementType.EXPRESSION.getType());
+ expression.setConfigureName(expressionName);
+ expression.setVarName(varName);
+ expression.setValue(dataType.getData(value));
+ expression.setDataType(dataType);
+ expression.setFunctionName(functionName);
+ return expression;
+ }
+
+ public static RelationExpression createRelationExpression(String ruleName, String expressionName, String relation,
+ List<String> expressionNames) {
+ RelationExpression relationExpression = new RelationExpression();
+ relationExpression.setNameSpace(ruleName);
+ relationExpression.setType(RuleElementType.EXPRESSION.getType());
+ relationExpression.setConfigureName(expressionName);
+ relationExpression.setRelation(relation);
+ relationExpression.setValue(expressionNames);
+ return relationExpression;
+ }
+
+ public static MetaDataAction createAction(String namespace, String name, String metaDataName,
+ Map<String, String> varName2FieldName) {
+ MetaDataAction metaDataAction = new MetaDataAction();
+ metaDataAction.setNameSpace(namespace);
+ metaDataAction.setType(RuleElementType.ACTION.getType());
+ metaDataAction.setConfigureName(name);
+ metaDataAction.setMetaDataName(metaDataName);
+ metaDataAction.setVarName2FieldName(varName2FieldName);
+ return metaDataAction;
+ }
+
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/contants/RuleElementType.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/contants/RuleElementType.java
new file mode 100644
index 0000000..308e836
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/contants/RuleElementType.java
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.contants;
+
+import org.apache.rocketmq.streams.common.channel.sink.ISink;
+import org.apache.rocketmq.streams.filter.operator.action.Action;
+
+import org.apache.rocketmq.streams.filter.operator.Rule;
+import org.apache.rocketmq.streams.filter.operator.expression.Expression;
+import org.apache.rocketmq.streams.filter.operator.var.Var;
+import org.apache.rocketmq.streams.common.metadata.MetaData;
+
+public enum RuleElementType {
+ METADATA(MetaData.TYPE),
+ VAR(Var.TYPE),
+ RULE(Rule.TYPE),
+ EXPRESSION(Expression.TYPE),
+ ACTION(Action.TYPE),
+ DATASOURCE(ISink.TYPE);
+ private String type;
+
+ private RuleElementType(String type) {
+ this.type = type;
+ }
+
+ public String getType() {
+ return this.type;
+ }
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/contants/RuleStatus.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/contants/RuleStatus.java
new file mode 100644
index 0000000..9407116
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/contants/RuleStatus.java
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.contants;
+
+/**
+ * 规则运行状态
+ */
+public enum RuleStatus {
+ RULE_STATUS_TESTING(0, "测试中", "yundun-sasruleengine-CSZ-key-001"),
+ RULE_STATUS_OBSERVING(1, "观察中", "yundun-sasruleengine-GCZ-key-002"),
+ RULE_STATUS_AUDITING(2, "审核中", "yundun-sasruleengine-SHZ-key-003"),
+ RULE_STATUS_ONLINE(3, "已上线", "yundun-sasruleengine-YSX-key-004"),
+ RULE_STATUS_SELF(4, "自定义", "do-self"),
+ KILLCHAIN_DESC_RULES(5, "killchain说明规则", "killchain-desc");
+
+ private Integer ruleStatus;
+ private String desc;
+ private String key;
+
+ private RuleStatus(int ruleStatus, String desc, String key) {
+ this.ruleStatus = ruleStatus;
+ this.desc = desc;
+ this.key = key;
+ }
+
+ public Integer getRuleStatus() {
+ return ruleStatus;
+ }
+
+ public void setRuleStatus(Integer ruleStatus) {
+ this.ruleStatus = ruleStatus;
+ }
+
+ public String getDesc() {
+ return desc;
+ }
+
+ public void setDesc(String desc) {
+ this.desc = desc;
+ }
+
+ public static String getDescByRuleStatus(Integer ruleStatus) {
+ if (RuleStatus.RULE_STATUS_OBSERVING.getRuleStatus().equals(ruleStatus)) {
+ return RuleStatus.RULE_STATUS_OBSERVING.getDesc();
+ } else if (RuleStatus.RULE_STATUS_AUDITING.getRuleStatus().equals(ruleStatus)) {
+ return RuleStatus.RULE_STATUS_AUDITING.getDesc();
+ } else if (RuleStatus.RULE_STATUS_ONLINE.getRuleStatus().equals(ruleStatus)) {
+ return RuleStatus.RULE_STATUS_ONLINE.getDesc();
+ } else {
+ return RuleStatus.RULE_STATUS_TESTING.getDesc();
+ }
+ }
+
+ public static String getKeyByRuleStatus(Integer ruleStatus) {
+ if (RuleStatus.RULE_STATUS_OBSERVING.getRuleStatus().equals(ruleStatus)) {
+ return RuleStatus.RULE_STATUS_OBSERVING.getKey();
+ } else if (RuleStatus.RULE_STATUS_AUDITING.getRuleStatus().equals(ruleStatus)) {
+ return RuleStatus.RULE_STATUS_AUDITING.getKey();
+ } else if (RuleStatus.RULE_STATUS_ONLINE.getRuleStatus().equals(ruleStatus)) {
+ return RuleStatus.RULE_STATUS_ONLINE.getKey();
+ } else {
+ return RuleStatus.RULE_STATUS_TESTING.getKey();
+ }
+ }
+
+ public String getKey() {
+ return key;
+ }
+
+ public void setKey(String key) {
+ this.key = key;
+ }
+
+ public static void main(String args[]) {
+ // System.out.println(SasRuleEnum.getDescByRuleStatus(new Integer(1)));
+ System.out.println(RuleStatus.RULE_STATUS_OBSERVING.getRuleStatus());
+
+ Integer a = new Integer(1);
+ int b = 1;
+ System.out.println(a == b);
+
+ }
+
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/context/ContextConfigure.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/context/ContextConfigure.java
new file mode 100644
index 0000000..e5c2145
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/context/ContextConfigure.java
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.context;
+
+import java.util.Properties;
+
+public class ContextConfigure {
+
+ /**
+ * 正则表达式优化相关
+ */
+ protected volatile boolean isSupportRegexOptimization = true; // 是否支持正则表达式优化
+ protected volatile int regexTimeout = 1000; // 正则表达式超时时间，默认100ms
+
+ protected int actionPoolSize = 5;
+ /**
+ * 控制监控输出
+ */
+ protected int monitorPoolSize = 10000; // 监控对象池大小
+ private volatile String rulesLogsIps = "127.0.0.1";
+ protected volatile boolean fetchDataInThisHost = false; // 是否在本机采样，在采样模式下使用
+ protected volatile int longTimeAlert = 1000; // 规则超过500ms则打印输出
+ protected volatile int mode = MONITOR_STATISTICS; // 0：开启模式；1，采样模式；-1：关闭模式
+ public static final int MONITOR_OPEN = 1; // 开启模式
+ public static final int MONITOR_CLOSE = -1; // 关闭模式
+ public static final int MONITOR_SAMPLE = 0; // 关闭模式
+ public static final int MONITOR_STATISTICS = 2; // 只统计时间，不输出
+ /**
+ * 区分引擎运行的环境
+ */
+ protected String evnNamespace; // 区分压测环境和线上环境
+
+ /**
+ * 控制结果输出
+ */
+ protected boolean action2Online = true; // 是否执行在线的action，如果是false，不执行线上的action
+ protected boolean action2Observer = false; // 观察中的规则是否输出到观察表，如果为true，则写到观察表
+ protected boolean actionOnline2Observer = false; // 在线规则输出到观察表，如果为true，则写到观察表，此种情况是为了测试
+
+ /**
+ * 控制加载规则
+ */
+ protected String loadRuleStatuses = "3"; // 规则加载方式，根据规则的状态加载
+ protected String diyRuleCodes = null; // 如果规则状态选择自定义，则此处列出规则的状态，用逗号分隔
+ //
+ ///**
+ // * 分流用对象
+ // */
+ //protected transient SplitFlow splitFlow = new SplitFlow();
+
+ /**
+ * 分流用对象
+ */
+ protected boolean openUnRepeateScript = false;
+
+ protected boolean openObserverDatasource = true;
+
+ /**
+ * 是否使用缓存处理正则
+ */
+ protected boolean encacheFlag = false;
+
+ /**
+ * 升级中心 压测数据
+ */
+ protected int pressureTotalCount = 10000;
+
+ /**
+ * 恶意IP最大加载数
+ */
+ protected int maliciousIPMaxCount = 1000000;
+
+ /**
+ * 是否支持监控模块输出超时日志
+ *
+ * @return
+ */
+ public boolean canMonitor() {
+ if (mode == ContextConfigure.MONITOR_OPEN) {
+ return true;
+ }
+ if (mode == ContextConfigure.MONITOR_SAMPLE && isFetchDataInThisHost()) {
+ return true;
+ }
+ return false;
+ }
+
+ public ContextConfigure(Properties properties) {
+
+ }
+
+ /**
+ * 是否支持规则统计（统计慢规则）
+ *
+ * @return
+ */
+ public boolean canStatisticsMonitor() {
+ if (mode == ContextConfigure.MONITOR_STATISTICS) {
+ return true;
+ }
+ if (this.canMonitor()) {
+ return true;
+ }
+ return false;
+ }
+
+ public boolean isSupportRegexOptimization() {
+ return isSupportRegexOptimization;
+ }
+
+ public void setSupportRegexOptimization(boolean supportRegexOptimization) {
+ isSupportRegexOptimization = supportRegexOptimization;
+ }
+
+ public int getRegexTimeout() {
+ return regexTimeout;
+ }
+
+ public void setRegexTimeout(int regexTimeout) {
+ this.regexTimeout = regexTimeout;
+ }
+
+ public boolean isFetchDataInThisHost() {
+ return fetchDataInThisHost;
+ }
+
+ public void setFetchDataInThisHost(boolean fetchDataInThisHost) {
+ this.fetchDataInThisHost = fetchDataInThisHost;
+ }
+
+ public int getLongTimeAlert() {
+ return longTimeAlert;
+ }
+
+ public void setLongTimeAlert(int longTimeAlert) {
+ this.longTimeAlert = longTimeAlert;
+ }
+
+ public String getEvnNamespace() {
+ return evnNamespace;
+ }
+
+ public void setEvnNamespace(String evnNamespace) {
+ this.evnNamespace = evnNamespace;
+ }
+
+ public boolean isAction2Online() {
+ return action2Online;
+ }
+
+ public void setAction2Online(boolean action2Online) {
+ this.action2Online = action2Online;
+ }
+
+ public boolean isAction2Observer() {
+ return action2Observer;
+ }
+
+ public void setAction2Observer(boolean action2Observer) {
+ this.action2Observer = action2Observer;
+ }
+
+ public boolean isActionOnline2Observer() {
+ return actionOnline2Observer;
+ }
+
+ public void setActionOnline2Observer(boolean actionOnline2Observer) {
+ this.actionOnline2Observer = actionOnline2Observer;
+ }
+
+ public String getLoadRuleStatuses() {
+ return loadRuleStatuses;
+ }
+
+ public void setLoadRuleStatuses(String loadRuleStatuses) {
+ this.loadRuleStatuses = loadRuleStatuses;
+ }
+
+ public String getDiyRuleCodes() {
+ return diyRuleCodes;
+ }
+
+ public void setDiyRuleCodes(String diyRuleCodes) {
+ this.diyRuleCodes = diyRuleCodes;
+ }
+
+ public String getRulesLogsIps() {
+ return rulesLogsIps;
+ }
+
+ public void setRulesLogsIps(String rulesLogsIps) {
+ this.rulesLogsIps = rulesLogsIps;
+ }
+
+ public int getMonitorPoolSize() {
+ return monitorPoolSize;
+ }
+
+ public void setMonitorPoolSize(int monitorPoolSize) {
+ this.monitorPoolSize = monitorPoolSize;
+ }
+
+ public int getMode() {
+ return mode;
+ }
+
+ public void setMode(int mode) {
+ this.mode = mode;
+ }
+
+ //public SplitFlow getSplitFlow() {
+ // return splitFlow;
+ //}
+ //
+ //public void setSplitFlow(SplitFlow splitFlow) {
+ // this.splitFlow = splitFlow;
+ //}
+
+ public boolean isOpenUnRepeateScript() {
+ return openUnRepeateScript;
+ }
+
+ public void setOpenUnRepeateScript(boolean openUnRepeateScript) {
+ this.openUnRepeateScript = openUnRepeateScript;
+ }
+
+ public boolean isOpenObserverDatasource() {
+ return openObserverDatasource;
+ }
+
+ public void setOpenObserverDatasource(boolean openObserverDatasource) {
+ this.openObserverDatasource = openObserverDatasource;
+ }
+
+ public boolean isEncacheFlag() {
+ return encacheFlag;
+ }
+
+ public void setEncacheFlag(boolean encacheFlag) {
+ this.encacheFlag = encacheFlag;
+ }
+
+ public int getPressureTotalCount() {
+ return pressureTotalCount;
+ }
+
+ public void setPressureTotalCount(int pressureTotalCount) {
+ this.pressureTotalCount = pressureTotalCount;
+ }
+
+ public int getActionPoolSize() {
+ return actionPoolSize;
+ }
+
+ public void setActionPoolSize(int actionPoolSize) {
+ this.actionPoolSize = actionPoolSize;
+ }
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/context/RuleContext.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/context/RuleContext.java
new file mode 100644
index 0000000..26554b2
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/context/RuleContext.java
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.context;
+
+import java.io.Serializable;
+import java.util.Properties;
+import java.util.Vector;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+import com.alibaba.fastjson.JSONObject;
+
+import org.apache.rocketmq.streams.db.driver.JDBCDriver;
+import org.apache.rocketmq.streams.filter.operator.action.Action;
+import org.apache.rocketmq.streams.filter.function.expression.ExpressionFunction;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+import org.apache.rocketmq.streams.filter.operator.expression.Expression;
+import org.apache.rocketmq.streams.filter.operator.var.Var;
+import org.apache.rocketmq.streams.common.configurable.IConfigurableService;
+import org.apache.rocketmq.streams.script.function.model.FunctionConfigure;
+import org.apache.rocketmq.streams.script.function.service.impl.ScanFunctionService;
+import org.apache.rocketmq.streams.common.context.AbstractContext;
+import org.apache.rocketmq.streams.common.context.IMessage;
+import org.apache.rocketmq.streams.common.context.Message;
+import org.apache.rocketmq.streams.common.monitor.IMonitor;
+import org.apache.rocketmq.streams.common.monitor.TopologyFilterMonitor;
+import org.apache.rocketmq.streams.common.metadata.MetaData;
+import org.apache.rocketmq.streams.common.metadata.MetaDataAdapter;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+public class RuleContext extends AbstractContext<Message> implements Serializable {
+
+ private static final Log LOG = LogFactory.getLog(RuleContext.class);
+
+ /**
+ * 观察者对应的configure
+ */
+ public static final String OBSERVER_NAME = "observerDBAction";
+
+ private static RuleContext superRuleContext;
+
+ private ExecutorService actionExecutor = null;
+
+ /**
+ * 默认的命名空间
+ */
+ public static final String DEFALUT_NAME_SPACE = IConfigurableService.PARENT_CHANNEL_NAME_SPACE;
+
+ /**
+ * var的名称合值对应的map
+ */
+ private ConcurrentMap<String, Object> varValueMap = new ConcurrentHashMap<String, Object>();
+ private ConcurrentMap<String, Boolean> expressionValueMap = new ConcurrentHashMap<String, Boolean>();
+
+ /**
+ * 错误信息
+ */
+ private Vector<String> errorMessageList = new Vector<>();
+ /**
+ * 当前的命名空间
+ */
+ private String nameSpace;
+
+ /**
+ * 配置服务
+ */
+ // private transient IConfigurableService configureService;
+
+ private transient ScanFunctionService functionService = ScanFunctionService.getInstance();
+ private transient RuleContext parentContext;
+ private transient Rule rule;
+
+ /**
+ * 监控一条信息的运行情况
+ */
+ private transient volatile IMonitor ruleMonitor;
+
+ /**
+ * 一个规则对应的系统配置
+ */
+ private transient ContextConfigure contextConfigure;
+
+ // private transient static volatile AtomicBoolean initflag = new AtomicBoolean(false);
+
+ private static volatile boolean initflag = false;
+
+ protected TopologyFilterMonitor expressionMonitor = new TopologyFilterMonitor();
+
+ public static void initSuperRuleContext(ContextConfigure contextConfigure) {
+ if (!initflag) {
+ synchronized (RuleContext.class) {
+ if (!initflag) {
+ RuleContext staticRuleContext = new RuleContext(DEFALUT_NAME_SPACE, contextConfigure);
+ ExecutorService actionExecutor = Executors.newFixedThreadPool(contextConfigure.getActionPoolSize());
+ staticRuleContext.actionExecutor = actionExecutor;
+ staticRuleContext.functionService.scanePackage("org.apache.rocketmq.streams.filter.function");
+ superRuleContext = staticRuleContext;
+ initflag = true;
+ }
+ }
+ }
+
+ }
+
+ private RuleContext(String pnameSpace, ContextConfigure contextConfigure) {
+ this(pnameSpace, new JSONObject(), null, contextConfigure);
+ }
+
+ public RuleContext(String nameSpace, JSONObject pmessage, Rule rule, ContextConfigure contextConfigure) {
+ super(new Message(pmessage));
+ if (!DEFALUT_NAME_SPACE.equals(nameSpace)) {
+ this.parentContext = superRuleContext;
+ }
+ this.nameSpace = nameSpace;
+ this.rule = rule;
+ this.contextConfigure = contextConfigure;
+ if (contextConfigure == null && parentContext != null) {
+ this.contextConfigure = this.parentContext.getContextConfigure();
+ }
+ if (this.functionService == null && this.parentContext != null) {
+ this.functionService = this.parentContext.getFunctionService();
+ }
+
+ }
+
+ public RuleContext(JSONObject pmessage, Rule rule, Properties properties) {
+ this(rule.getNameSpace(), pmessage, rule, new ContextConfigure(properties));
+ }
+
+ public RuleContext(JSONObject pmessage, Rule rule) {
+ this(rule.getNameSpace(), pmessage, rule, new ContextConfigure(null));
+ }
+
+ public String getNameSpace() {
+ if (nameSpace != null) {
+ return nameSpace;
+ }
+ if (parentContext != null) {
+ return parentContext.getNameSpace();
+ }
+ return nameSpace;
+ }
+
+ /**
+ * 获取表达式值
+ *
+ * @param configureName
+ * @return
+ */
+ public Boolean getExpressionValue(String configureName) {
+ Boolean result = expressionValueMap.get(configureName);
+ return result;
+ }
+
+ /**
+ * 设置表达式值
+ *
+ * @param configureName
+ * @param result
+ */
+ public void putExpressionValue(String nameSpace, String configureName, Boolean result) {
+ if (rule.getExpressionMap().containsKey(configureName)) {
+ expressionValueMap.putIfAbsent(configureName, result);
+ }
+ }
+
+ /**
+ * 当规则产生错误时，记录错误信息
+ *
+ * @param message
+ */
+ public void addErrorMessage(Rule rule, String message) {
+ String messageInfo = message;
+ if (rule != null) {
+ messageInfo = rule.getRuleCode() + ":" + messageInfo;
+ }
+ errorMessageList.add(messageInfo);
+ }
+
+ public Var getVar(String ruleName, String name) {
+ return getVar(name);
+ }
+
+ public Var getVar(String name) {
+ Var var = rule.getVarMap().get(name);
+
+ return var;
+ }
+
+ public Expression getExpression(String name) {
+ Expression expression = rule.getExpressionMap().get(name);
+ return expression;
+ }
+
+ public Action getAction(String name) {
+ Action action = rule.getActionMap().get(name);
+ return action;
+ }
+
+ public ExpressionFunction getExpressionFunction(String name, Object... objects) {
+ try {
+ FunctionConfigure fc = functionService.getFunctionConfigure(name, objects);
+ if (fc == null) {
+ return null;
+ }
+ return (ExpressionFunction)fc.getBean();
+ } catch (Exception e) {
+ LOG.error("RuleContext getExpressionFunction error,name is: " + name, e);
+ return null;
+ }
+
+ }
+
+ public MetaData getMetaData(String name) {
+ MetaData metaData = rule.getMetaDataMap().get(name);
+ return metaData;
+ }
+
+ public JDBCDriver getDataSource(String name) {
+ return rule.getDataSourceMap().get(name);
+ }
+
+ /**
+ * @param name
+ * @return
+ */
+ public MetaDataAdapter getMetaDataAdapter(String name) {
+ MetaData md = getMetaData(name);
+ JDBCDriver dataSource = this.getDataSource(md.getDataSourceName());
+ MetaDataAdapter mda = new MetaDataAdapter(md, dataSource);
+ return mda;
+
+ }
+
+ public boolean containsVarName(String varName) {
+ if (varValueMap.containsKey(varName)) {
+ return true;
+ }
+ return false;
+ }
+
+ /**
+ * 获取变量值,内部使用，不能直接使用，获取变量的值需要用var.getVarValue()
+ *
+ * @param varName
+ * @return
+ */
+ public Object getVarCacheValue(String varName) {
+ Object value = varValueMap.get(varName);
+ if (value != null) {
+ return value;
+ }
+ if (parentContext != null) {
+ value = parentContext.getVarCacheValue(varName);
+ }
+ return value;
+
+ }
+
+ /**
+ * 设置变量值
+ *
+ * @param varName
+ * @param value
+ */
+ public void putVarValue(String nameSpace, String varName, Object value) {
+ if (varName == null || value == null) {
+ return;
+ }
+ if (rule.getVarMap().containsKey(varName)) {
+ varValueMap.putIfAbsent(varName, value);
+ }
+
+ }
+
+ public void setContextConfigure(ContextConfigure contextConfigure) {
+ this.contextConfigure = contextConfigure;
+ }
+
+ public ContextConfigure getContextConfigure() {
+ if (contextConfigure != null) {
+ return contextConfigure;
+ }
+ if (parentContext != null) {
+ return parentContext.getContextConfigure();
+ }
+ return null;
+ }
+
+ @Override
+ public IConfigurableService getConfigurableService() {
+ return rule.getConfigurableService();
+ }
+
+ @Override
+ public AbstractContext copy() {
+ IMessage message = this.message.copy();
+ RuleContext context = new RuleContext(nameSpace, message.getMessageBody(), rule, contextConfigure);
+ super.copyProperty(context);
+ context.actionExecutor = actionExecutor;
+ context.errorMessageList = errorMessageList;
+ context.expressionValueMap = expressionValueMap;
+ context.functionService = functionService;
+ context.ruleMonitor = ruleMonitor;
+ context.parentContext = parentContext;
+ context.varValueMap = varValueMap;
+ context.configurableService = configurableService;
+ return context;
+ }
+
+ public ExecutorService getActionExecutor() {
+ if (actionExecutor != null) {
+ return actionExecutor;
+ } else {
+ return parentContext.getActionExecutor();
+ }
+ }
+
+ public void setNameSpace(String nameSpace) {
+ this.nameSpace = nameSpace;
+ }
+
+ public ConcurrentMap<String, Object> getVarValueMap() {
+ return varValueMap;
+ }
+
+ public void setVarValueMap(ConcurrentMap<String, Object> varValueMap) {
+ this.varValueMap = varValueMap;
+ }
+
+ public ConcurrentMap<String, Boolean> getExpressionValueMap() {
+ return expressionValueMap;
+ }
+
+ public void setExpressionValueMap(ConcurrentMap<String, Boolean> expressionValueMap) {
+ this.expressionValueMap = expressionValueMap;
+ }
+
+ public Vector<String> getErrorMessageList() {
+ return errorMessageList;
+ }
+
+ public void setErrorMessageList(Vector<String> errorMessageList) {
+ this.errorMessageList = errorMessageList;
+ }
+
+ public ScanFunctionService getFunctionService() {
+ return functionService;
+ }
+
+ public RuleContext getParentContext() {
+ return parentContext;
+ }
+
+ public void setParentContext(RuleContext parentContext) {
+ this.parentContext = parentContext;
+ }
+
+ public Rule getRule() {
+ return rule;
+ }
+
+ public void setRule(Rule rule) {
+ this.rule = rule;
+ }
+
+ public IMonitor getRuleMonitor() {
+ return ruleMonitor;
+ }
+
+ public void setRuleMonitor(IMonitor ruleMonitor) {
+ this.ruleMonitor = ruleMonitor;
+ }
+
+ public TopologyFilterMonitor getExpressionMonitor() {
+ return expressionMonitor;
+ }
+
+ public void setExpressionMonitor(TopologyFilterMonitor expressionMonitor) {
+ this.expressionMonitor = expressionMonitor;
+ }
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/context/RuleMessage.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/context/RuleMessage.java
new file mode 100644
index 0000000..d7831f1
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/context/RuleMessage.java
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.context;
+
+import com.alibaba.fastjson.JSONObject;
+import org.apache.rocketmq.streams.filter.monitor.rule.MessageMonitor;
+import org.apache.rocketmq.streams.common.context.Message;
+
+public class RuleMessage extends Message {
+
+ protected MessageMonitor messageMonitor;
+
+ public RuleMessage(JSONObject message) {
+ super(message);
+ messageMonitor = createMessageMonitor(message.toJSONString());
+ }
+
+ public MessageMonitor getMessageMonitor() {
+ return messageMonitor;
+ }
+
+ /**
+ * 创建一条信息的监控对象
+ *
+ * @param message
+ * @return
+ */
+ protected MessageMonitor createMessageMonitor(String message) {
+ MessageMonitor messageMonitor = new MessageMonitor();
+ messageMonitor.begin(message);
+ return messageMonitor;
+ }
+
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/engine/IRuleEngine.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/engine/IRuleEngine.java
new file mode 100644
index 0000000..7db56a9
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/engine/IRuleEngine.java
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.engine;
+
+import java.util.List;
+
+import org.apache.rocketmq.streams.filter.context.RuleMessage;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+import org.apache.rocketmq.streams.common.context.AbstractContext;
+
+public interface IRuleEngine {
+
+ /**
+ * 把消息相关的上下文传递给规则引擎，规则引擎进行变量封装，资源检查，规则执行等操作
+ *
+ * @param message 包含变量相关信息和原始的message信息
+ * @return 触发的规则
+ */
+ List<Rule> executeRule(RuleMessage message, List<Rule> rules);
+
+ /**
+ * 把消息相关的上下文传递给规则引擎，规则引擎进行变量封装，资源检查，规则执行等操作
+ *
+ * @param message 包含变量相关信息和原始的message信息
+ * @return 触发的规则
+ */
+ List<Rule> executeRule(AbstractContext context, RuleMessage message, List<Rule> rules);
+
+ /**
+ * 把消息相关的上下文传递给规则引擎，规则引擎进行变量封装，资源检查，规则执行等操作
+ *
+ * @param message 包含变量相关信息和原始的message信息
+ * @return 触发的规则
+ */
+ List<Rule> executeRuleWithoutAction(RuleMessage message, List<Rule> rules);
+
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/engine/SplitFlow.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/engine/SplitFlow.java
new file mode 100644
index 0000000..ce5807f
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/engine/SplitFlow.java
@@ -0,0 +1,159 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.engine;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+import com.alibaba.fastjson.JSONObject;
+
+public class SplitFlow {
+
+ private static final Log LOG = LogFactory.getLog(SplitFlow.class);
+ private volatile String nameSpace;
+ private JSONObject uuidNameMap = new JSONObject();
+
+ // uuid尾号
+ private volatile String uuidEndNum = NUMBERS_ALL;
+ private volatile String ageis_split_namespaces = "topic_aegis_detect_msg_proc";
+ private volatile String sas_split_namespaces;
+
+ // 全部尾号
+ private static final String NUMBERS_ALL = "all";
+ // 没有尾号
+ private static final String NUMBERS_NONE = "none";
+
+ private void paserConfigToField(String jsonValue) {
+ JSONObject jsonObject = JSONObject.parseObject(jsonValue);
+ this.uuidEndNum = jsonObject.getString("uuidEndNum");
+ this.ageis_split_namespaces = jsonObject.getString("ageis_split_namespaces");
+ this.sas_split_namespaces = jsonObject.getString("sas_split_namespaces");
+ }
+
+ /**
+ * 验证安骑士数据是否满足分流后的条件
+ *
+ * @return
+ */
+ public boolean aegisUuidMatch(String uuid) {
+ if (uuidEndNum == null || "".equals(uuidEndNum)) {
+ LOG.warn("SplitFlow aegisUuidMatch,uuidEndNum is null");
+ return true;
+ }
+ String numbers = uuidEndNum;
+
+ if (NUMBERS_ALL.equals(numbers)) {
+ return true;
+ } else if (NUMBERS_NONE.equals(numbers)) {
+ return false;
+ } else {
+ return this.getNumberResult(uuid, numbers);
+ }
+ }
+
+ /**
+ * 验证安骑士数据是否满足分流后的条件
+ *
+ * @param message
+ * @return
+ */
+ public boolean sasMatch(JSONObject message) {
+ return true;
+ }
+
+ public String getUuidNameByNameSpace(String nameSpace) {
+ String uuid = uuidNameMap.getString(nameSpace);
+ if (uuid == null) {
+ uuid = "uuid";
+ }
+ return uuid;
+ }
+
+ public void setUuidNameJson(String uuidNameJson) {
+ try {
+ this.uuidNameMap = JSONObject.parseObject(uuidNameJson);
+ } catch (Exception e) {
+ LOG.error("SplitFlow setUuidNameJson error" + uuidNameJson, e);
+ }
+
+ }
+
+ /**
+ * 判断尾号是否满足条件
+ *
+ * @return
+ */
+ private boolean getNumberResult(String aliuid, String userNumbers) {
+ String[] numbers = userNumbers.split(",");
+ for (String number : numbers) {
+ if (aliuid.endsWith(number)) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ public String getNameSpace() {
+ return nameSpace;
+ }
+
+ public void setNameSpace(String nameSpace) {
+ this.nameSpace = nameSpace;
+ }
+
+ public String getAgeis_split_namespaces() {
+ return ageis_split_namespaces;
+ }
+
+ public void setAgeis_split_namespaces(String ageis_split_namespaces) {
+ this.ageis_split_namespaces = ageis_split_namespaces;
+ }
+
+ public String getSas_split_namespaces() {
+ return sas_split_namespaces;
+ }
+
+ public void setSas_split_namespaces(String sas_split_namespaces) {
+ this.sas_split_namespaces = sas_split_namespaces;
+ }
+
+ public void setSplitFlowJson(String splitFlowJson) {
+ try {
+ paserConfigToField(splitFlowJson);
+ } catch (Exception e) {
+ LOG.error("SplitFlow setSplitFlowJson error,splitFlowJson is:" + splitFlowJson, e);
+ }
+
+ }
+
+ public JSONObject getUuidNameMap() {
+ return uuidNameMap;
+ }
+
+ public void setUuidNameMap(JSONObject uuidNameMap) {
+ this.uuidNameMap = uuidNameMap;
+ }
+
+ public String getUuidEndNum() {
+ return uuidEndNum;
+ }
+
+ public void setUuidEndNum(String uuidEndNum) {
+ this.uuidEndNum = uuidEndNum;
+ }
+
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/engine/impl/DefaultRuleEngine.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/engine/impl/DefaultRuleEngine.java
new file mode 100644
index 0000000..6ec8f3a
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/engine/impl/DefaultRuleEngine.java
@@ -0,0 +1,280 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.engine.impl;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.rocketmq.streams.filter.context.RuleContext;
+import org.apache.rocketmq.streams.filter.engine.IRuleEngine;
+import org.apache.rocketmq.streams.filter.operator.action.Action;
+import org.apache.rocketmq.streams.filter.contants.RuleStatus;
+import org.apache.rocketmq.streams.filter.exception.RegexTimeoutException;
+import org.apache.rocketmq.streams.filter.context.RuleMessage;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+import org.apache.rocketmq.streams.filter.operator.expression.Expression;
+import org.apache.rocketmq.streams.filter.operator.expression.RelationExpression;
+import org.apache.rocketmq.streams.common.context.AbstractContext;
+import org.apache.rocketmq.streams.common.monitor.IMonitor;
+import org.apache.rocketmq.streams.common.monitor.group.MonitorCommander;
+import org.apache.rocketmq.streams.common.monitor.TopologyFilterMonitor;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+public class DefaultRuleEngine implements IRuleEngine {
+
+ private static final Log LOG = LogFactory.getLog(DefaultRuleEngine.class);
+
+ private static final Log RULEENGINE_MESSAGE_LOG = LogFactory.getLog("ruleengine_message");
+
+ @SuppressWarnings({"rawtypes", "unchecked"})
+ @Override
+ public List<Rule> executeRule(RuleMessage message, List<Rule> rules) {
+ return executeRule(null, message, rules, true);
+ }
+
+ @Override
+ public List<Rule> executeRule(AbstractContext context, RuleMessage message, List<Rule> rules) {
+ return executeRule(context, message, rules, true);
+ }
+
+ protected List<Rule> executeRule(AbstractContext context, RuleMessage message, List<Rule> rules, boolean isAction) {
+ IMonitor rulesMonitor = null;
+ IMonitor monitor = null;
+ if (context != null) {
+ monitor = context.getCurrentMonitorItem();
+ monitor = context.startMonitor("rules");
+ rulesMonitor = monitor;
+ }
+ List<Rule> fireRules = new ArrayList<>();
+ List<Rule> excuteRules = rules;
+ try {
+ if (excuteRules == null) {
+ return fireRules;
+ }
+ try {
+
+ for (Rule rule : excuteRules) {
+ RuleContext ruleContext = new RuleContext(message.getMessageBody(), rule);
+ if (context != null) {
+ context.syncSubContext(ruleContext);
+ }
+ IMonitor ruleMonitor = null;
+ if (monitor != null) {
+ ruleMonitor = monitor.createChildren(rule);
+ ruleContext.setRuleMonitor(ruleMonitor);
+ }
+
+ try {
+
+ boolean isFireRule = executeRule(ruleContext, rule);
+ if (isFireRule == false) {
+ TopologyFilterMonitor piplineExecutorMonitor = message.getHeader().getPiplineExecutorMonitor();
+ if (piplineExecutorMonitor != null) {
+ piplineExecutorMonitor.setNotFireRule(rule.getConfigureName());
+ piplineExecutorMonitor.setNotFireExpression2DependentFields(ruleContext.getExpressionMonitor().getNotFireExpression2DependentFields());
+ }
+ }
+ if (ruleMonitor != null) {
+ ruleMonitor.addContextMessage(isFireRule);
+
+ }
+
+ if (isFireRule) {
+ fireRules.add(rule);
+
+ //doUnRepeateScript(ruleContext, rule);// 执行规则去重脚本
+ if (isAction) {
+ fireAction(ruleContext, rule);
+ }
+ }
+ if (ruleMonitor != null) {
+ ruleMonitor.endMonitor();
+ if (ruleMonitor.isSlow()) {
+ ruleMonitor.setSampleData(ruleContext).put("rule_info", rule.toJsonObject());
+ }
+ }
+
+ } catch (Exception e) {
+ if (!RegexTimeoutException.class.isInstance(e)) {
+ LOG.error("DefaultRuleEngine executeRule for excuteRules error: rule name is: "
+ + rule.getConfigureName(), e);
+ }
+ if (ruleMonitor != null) {
+ ruleMonitor.occureError(e, e.getMessage());
+ ruleContext.addErrorMessage(rule, e.getMessage());
+ ruleMonitor.setSampleData(ruleContext).put("rule_info", rule.toJsonObject());
+ }
+
+ }
+ }
+ } catch (Exception e) {
+ LOG.error("DefaultRuleEngine executeRule error,excuteRules size is " + excuteRules.size()
+ + " ,fireRules size is :" + fireRules.size(), e);
+ }
+ } catch (Exception e) {
+ LOG.error(
+ "DefaultRuleEngine executeRule error: fireRules size is: " + fireRules.size() + "excuteRules size is : "
+ + excuteRules.size(), e);
+
+ }
+ if (rulesMonitor != null) {
+ rulesMonitor.setType(IMonitor.TYPE_DATAPROCESS);
+ rulesMonitor.endMonitor();
+ MonitorCommander.getInstance().finishMonitor(rulesMonitor.getName(), rulesMonitor);
+ }
+ return fireRules;
+
+ }
+
+ @Override
+ public List<Rule> executeRuleWithoutAction(RuleMessage message, List<Rule> rules) {
+ return executeRule(null, message, rules, false);
+ }
+
+ @SuppressWarnings("rawtypes")
+ private void fireAction(RuleContext context, Rule rule) {
+ if (rule == null) {
+ LOG.error("DefaultRuleEngine fireAction error: rules is null!");
+ return;
+ }
+ try {
+ /**
+ * 判断rule的status 如果是观察者模式，则写入到观察者库中的观察表里
+ */
+ if (rule.getRuleStatus().intValue() == RuleStatus.RULE_STATUS_ONLINE.getRuleStatus().intValue()) {// 上线状态
+ if (!context.getContextConfigure().isAction2Online()) {
+ // LOG.warn("DefaultRuleEngine fireAction ignore : configure action2Online false!");
+ if (context.getContextConfigure().isActionOnline2Observer()) {
+ Action action = context.getAction(RuleContext.OBSERVER_NAME);
+ if (action == null) {
+ return;
+ }
+ doAction(context, action, rule);
+ }
+ return;
+ }
+ try {
+ if (rule.getActionNames() == null || rule.getActionNames().size() == 0) {
+ return;
+ }
+ for (String actionName : rule.getActionNames()) {
+ Action action = context.getAction(actionName);
+ if (action == null) {
+ continue;
+ }
+ doAction(context, action, rule);
+ }
+ } catch (Exception e) {
+ LOG.error("DefaultRuleEngine fire atciton error: ruleName is" + rule.getConfigureName(), e);
+ context.addErrorMessage(rule, "DefaultRuleEngine fire atciton error: " + e.getMessage());
+ }
+ } else {
+ if (context.getContextConfigure() != null && !context.getContextConfigure().isAction2Observer()) {
+ return;
+ }
+ Action action = context.getAction(RuleContext.OBSERVER_NAME);
+ if (action == null) {
+ return;
+ }
+ doAction(context, action, rule);
+ }
+
+ } catch (Exception e) {
+ LOG.error("DefaultRuleEngine fireAction error: ruleName is" + rule.getConfigureName(), e);
+ }
+
+ }
+
+ /**
+ * 处理一个规则的一个action
+ *
+ * @param context
+ * @param rule
+ */
+ @SuppressWarnings("rawtypes")
+ protected void doAction(final RuleContext context, final Action action, final Rule rule) {
+ context.getActionExecutor().execute(new Runnable() {
+
+ @Override
+ public void run() {
+ IMonitor monitor = context.getRuleMonitor();
+ IMonitor actionMonitor = monitor.createChildren(action);
+ try {
+
+ action.doAction(context, rule);
+ if (monitor != null) {
+ actionMonitor.endMonitor();
+ if (actionMonitor.isSlow()) {
+ actionMonitor.setSampleData(context).put("action_info", action.toJsonObject());
+ }
+ }
+ } catch (Exception e) {
+ String errorMsg = "DefaultRuleEngine doAction error,rule: " + rule.getRuleCode() + " ,action: "
+ + action.getConfigureName();
+ // RULEENGINE_MESSAGE_LOG.warn(errorMsg
+ // , e);
+ actionMonitor.occureError(e, errorMsg, e.getMessage());
+ actionMonitor.setSampleData(context).put("action_info", action.toJsonObject());
+ }
+
+ }
+ });
+ }
+
+ protected boolean executeRule(RuleContext context, Rule rule) {
+ boolean match = processExpress(rule, context);
+ return match;
+ }
+
+ /**
+ * 处理Express
+ *
+ * @param rule
+ * @param context
+ * @return
+ */
+ @SuppressWarnings("rawtypes")
+ private boolean processExpress(Rule rule, RuleContext context) {
+ try {
+ if (rule.getExpressionName() == null) {
+ return false;
+ }
+ // rule.getGroupExpressionManager().matchAndSetResult(context,rule);
+ Expression expression = context.getExpression(rule.getExpressionName());
+ if (expression == null) {
+ return false;
+ }
+
+ boolean match = expression.getExpressionValue(context, rule);
+ if (!RelationExpression.class.isInstance(expression)) {
+ TopologyFilterMonitor piplineExecutorMonitor = new TopologyFilterMonitor();
+ piplineExecutorMonitor.addNotFireExpression(expression.toString(), expression.getDependentFields(rule.getExpressionMap()));
+ context.setExpressionMonitor(piplineExecutorMonitor);
+ }
+ if (!match) {
+ return false;
+ }
+ } catch (Exception e) {
+ e.printStackTrace();
+ LOG.error("DefaultRuleEngine processExpress error,rule is: " + rule.getConfigureName(), e);
+ return false;
+ }
+ return true;
+ }
+
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/exception/RegexTimeoutException.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/exception/RegexTimeoutException.java
new file mode 100644
index 0000000..c69aa2f
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/exception/RegexTimeoutException.java
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.exception;
+
+public class RegexTimeoutException extends RuntimeException {
+ private String regex;
+ private String context;
+ private long timeout;
+
+ public RegexTimeoutException(String message) {
+ super(message);
+ }
+
+ public RegexTimeoutException(String regex, String context, long timeout) {
+ // super("正则表达式执行超时，超时的正则为："+regex+"。匹配的内容为："+context+"。对应的超时时间为："+timeout);
+ super("The regular expression executes timeout, and the timeout is:" + regex + ". The matching content is "
+ + context + ". The corresponding timeout time is: " + timeout);
+ this.regex = regex;
+ this.context = context;
+ this.timeout = timeout;
+ }
+
+ public String getRegex() {
+ return regex;
+ }
+
+ public String getContext() {
+ return context;
+ }
+
+ public long getTimeout() {
+ return timeout;
+ }
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/etl/RenameFunction.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/etl/RenameFunction.java
new file mode 100644
index 0000000..880af7b
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/etl/RenameFunction.java
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.function.etl;
+
+import com.alibaba.fastjson.JSONObject;
+import org.apache.rocketmq.streams.script.annotation.Function;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethod;
+import org.apache.rocketmq.streams.common.utils.ReflectUtil;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+
+@Function
+public class RenameFunction {
+
+ @FunctionMethod("rename")
+ public void rename(JSONObject msg, String newFieldName, String oldFieldName) {
+ String value = ReflectUtil.getBeanFieldOrJsonValue(msg, oldFieldName);
+ if (StringUtil.isEmpty(value)) {
+ return;
+ }
+ msg.remove(oldFieldName);
+ msg.put(newFieldName, value);
+ }
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/AbstractExpressionFunction.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/AbstractExpressionFunction.java
new file mode 100644
index 0000000..74a3adc
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/AbstractExpressionFunction.java
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.function.expression;
+
+import org.apache.rocketmq.streams.filter.context.RuleContext;
+import org.apache.rocketmq.streams.filter.exception.RegexTimeoutException;
+import org.apache.rocketmq.streams.filter.operator.expression.Expression;
+import org.apache.rocketmq.streams.common.monitor.IMonitor;
+import org.apache.rocketmq.streams.common.utils.MapKeyUtil;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+import org.apache.rocketmq.streams.filter.operator.Rule;
+
+public abstract class AbstractExpressionFunction implements ExpressionFunction {
+
+ private static final Log LOG = LogFactory.getLog(AbstractExpressionFunction.class);
+ private static final Log RULEENGINE_MESSAGE_LOG = LogFactory.getLog("ruleengine_message");
+
+ @Override
+ public Boolean doFunction(Expression expression, RuleContext context, Rule rule) {
+ String name = MapKeyUtil.createKey(Expression.TYPE, expression.getConfigureName());
+ IMonitor monitor = null;
+ if (context != null && context.getRuleMonitor() != null) {
+ monitor = context.getRuleMonitor().createChildren(name);
+ }
+ try {
+ Boolean result = doExpressionFunction(expression, context, rule);
+ if (monitor != null) {
+ monitor.setResult(result);
+ monitor.endMonitor();
+ if (monitor.isSlow()) {
+ monitor.setSampleData(context).put("expression_info", expression.toJson());
+ }
+ }
+ return result;
+ } catch (RegexTimeoutException e) {
+ LOG.error("AbstractExpressionFunction RegexTimeoutException", e);
+ RULEENGINE_MESSAGE_LOG.warn("AbstractExpressionFunction doFunction error", e);
+ if (monitor != null) {
+ monitor.occureError(e, "AbstractExpressionFunction doFunction error", e.getMessage());
+ monitor.setSampleData(context).put("expression_info", expression.toJsonObject());
+ }
+
+ throw e;
+ }
+
+ }
+
+ @SuppressWarnings("rawtypes")
+ protected abstract Boolean doExpressionFunction(Expression expression, RuleContext context, Rule rule);
+
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/CompareFunction.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/CompareFunction.java
new file mode 100644
index 0000000..a643882
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/CompareFunction.java
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.function.expression;
+
+import org.apache.rocketmq.streams.filter.context.RuleContext;
+import org.apache.rocketmq.streams.filter.operator.var.Var;
+import org.apache.rocketmq.streams.script.utils.FunctionUtils;
+import org.apache.rocketmq.streams.common.utils.ReflectUtil;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+import org.apache.rocketmq.streams.filter.operator.Rule;
+import org.apache.rocketmq.streams.filter.operator.expression.Expression;
+
+public abstract class CompareFunction extends AbstractExpressionFunction {
+
+ private static final Log LOG = LogFactory.getLog(CompareFunction.class);
+
+ @Override
+ public Boolean doExpressionFunction(Expression expression, RuleContext context, Rule rule) {
+ if (!expression.volidate()) {
+ return false;
+ }
+ Object varValue = null;
+ String varName = expression.getVarName();
+ Var var = context.getVar(varName);
+ varValue = var.getVarValue(context, rule);
+ /**
+ * 两个数字比较的情况
+ */
+ if ((FunctionUtils.isNumber(varName) || FunctionUtils.isConstant(varName)) && varValue == null) {
+ varValue = varName;
+ }
+
+ if (varValue == null || expression.getValue() == null) {
+ return false;
+ }
+ Object basicVarValue = expression.getDataType().getData(varValue.toString());
+ Object basicValue = expression.getDataType().getData(expression.getValue().toString());
+ if (varValue == null || expression.getValue() == null) {
+ return false;
+ }
+ boolean match = false;
+ if (basicValue == null || basicVarValue == null) {
+ return false;
+ }
+
+ Class varClass = basicVarValue == null ? expression.getDataType().getDataClass() : basicVarValue.getClass();
+ Class valueClass = basicValue == null ? expression.getDataType().getDataClass() : basicValue.getClass();
+ try {
+ match = (Boolean)ReflectUtil.invoke(this, "compare",
+ new Class[] {varClass, valueClass},
+ new Object[] {basicVarValue, basicValue});
+ } catch (Exception e) {
+ LOG.error("CompareFunction doFunction ReflectUtil.invoke error: ", e);
+ }
+
+ return match;
+ }
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/ContainsCaseInsensitiveFunction.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/ContainsCaseInsensitiveFunction.java
new file mode 100644
index 0000000..215ac8a
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/ContainsCaseInsensitiveFunction.java
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.function.expression;
+
+import org.apache.rocketmq.streams.filter.context.RuleContext;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+import org.apache.rocketmq.streams.filter.operator.expression.Expression;
+import org.apache.rocketmq.streams.filter.operator.var.Var;
+import org.apache.rocketmq.streams.script.annotation.Function;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethod;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethodAilas;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+@Function
+public class ContainsCaseInsensitiveFunction extends AbstractExpressionFunction {
+
+ private static final Log LOG = LogFactory.getLog(ContainsCaseInsensitiveFunction.class);
+
+ @Override
+ @FunctionMethod("containsCaseInsensitiveFunction")
+ @FunctionMethodAilas("包含(忽略大小写)")
+ public Boolean doExpressionFunction(Expression expression, RuleContext context, Rule rule) {
+
+ try {
+ if (!expression.volidate()) {
+ return false;
+ }
+
+ Var var = context.getVar(rule.getConfigureName(), expression.getVarName());
+ if (var == null) {
+ return false;
+ }
+ Object varObject = null;
+ Object valueObject = null;
+ varObject = var.getVarValue(context, rule);
+ valueObject = expression.getValue();
+
+ if (varObject == null || valueObject == null) {
+ return false;
+ }
+
+ String varString = "";
+ String regex = "";
+ varString = String.valueOf(varObject).trim();
+ regex = String.valueOf(valueObject).trim();
+
+ if (varString.toLowerCase().contains(regex.toLowerCase())) {
+ return true;
+ } else {
+ return false;
+ }
+ } catch (Exception e) {
+ LOG.error("ContainsCaseInsensitiveFunction error: rule name is: " + rule.getConfigureName(), e);
+ return false;
+ }
+
+ }
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/ContainsFunction.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/ContainsFunction.java
new file mode 100644
index 0000000..f3f36ed
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/ContainsFunction.java
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.function.expression;
+
+import org.apache.rocketmq.streams.filter.context.RuleContext;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+import org.apache.rocketmq.streams.filter.operator.expression.Expression;
+import org.apache.rocketmq.streams.filter.operator.var.Var;
+import org.apache.rocketmq.streams.script.annotation.Function;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethod;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethodAilas;
+
+@Function
+public class ContainsFunction extends AbstractExpressionFunction {
+
+ @Override
+ @FunctionMethod("contains")
+ @FunctionMethodAilas("包含")
+ public Boolean doExpressionFunction(Expression expression, RuleContext context, Rule rule) {
+ if (!expression.volidate()) {
+ return false;
+ }
+
+ Var var = context.getVar(rule.getConfigureName(), expression.getVarName());
+ if (var == null) {
+ return false;
+ }
+ Object varObject = null;
+ Object valueObject = null;
+ varObject = var.getVarValue(context, rule);
+ valueObject = expression.getValue();
+
+ if (varObject == null || valueObject == null) {
+ return false;
+ }
+
+ String varString = "";
+ String valueString = "";
+ varString = String.valueOf(varObject).trim();
+ valueString = String.valueOf(valueObject).trim();
+
+ if (varString.contains(valueString)) {
+ return true;
+ } else {
+ return false;
+ }
+ }
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/Equals.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/Equals.java
new file mode 100644
index 0000000..4744c8a
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/Equals.java
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.function.expression;
+
+import org.apache.rocketmq.streams.filter.context.RuleContext;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+import org.apache.rocketmq.streams.filter.operator.expression.Expression;
+import org.apache.rocketmq.streams.common.datatype.DateDataType;
+import org.apache.rocketmq.streams.script.annotation.Function;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethod;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethodAilas;
+import org.apache.rocketmq.streams.script.utils.FunctionUtils;
+
+import java.util.Date;
+
+@Function
+public class Equals extends CompareFunction {
+
+ private final static double MIN_VALUE = 0.000001;
+ private static final DateDataType dateDataType = new DateDataType(Date.class);
+
+ /**
+ * 是否是equals的函数名
+ *
+ * @param functionName
+ * @return
+ */
+ public static boolean isEqualFunction(String functionName) {
+ if ("=".equals(functionName) || "==".equals(functionName) || "等于".equals(functionName)) {
+ return true;
+ }
+ return false;
+ }
+
+ @FunctionMethod(value = "=", alias = "==")
+ @FunctionMethodAilas("等于")
+ @Override
+ public Boolean doFunction(Expression expression, RuleContext context, Rule rule) {
+ return super.doFunction(expression, context, rule);
+ }
+
+ public boolean compare(int a, int b) {
+ return a == b;
+ }
+
+ public boolean compare(String a, int b) {
+ try {
+ a = FunctionUtils.getConstant(a);
+ int aa = new Integer(a).intValue();
+ return aa == b;
+ } catch (Exception e) {
+ return false;
+ }
+ }
+
+ public boolean compare(Integer a, Integer b) {
+ try {
+ Integer aa = new Integer(a);
+ return aa.equals(b);
+ } catch (Exception e) {
+ return false;
+ }
+
+ }
+
+ public boolean compare(String a, String b) {
+ // a = StringUtils.defaultString(a);
+ // b = StringUtils.defaultString(b);
+ // return a.trim().equals(b.trim());
+ a = FunctionUtils.getConstant(a);
+ b = FunctionUtils.getConstant(b);
+ return a.equals(b);
+ }
+
+ public boolean compare(boolean a, boolean b) {
+ return a == b;
+ }
+
+ public boolean compare(Boolean a, Boolean b) {
+ return a.equals(b);
+ }
+
+ public boolean compare(long a, long b) {
+ return a == b;
+ }
+
+ public boolean compare(double a, double b) {
+ if (a - b > -MIN_VALUE && a - b < MIN_VALUE) {
+ return true;
+ }
+ return false;
+ }
+
+ public boolean compare(float a, float b) {
+ if (a - b > -MIN_VALUE && a - b < MIN_VALUE) {
+ return true;
+ }
+ return false;
+ }
+
+ public boolean compare(Long a, Long b) {
+ try {
+ Long aa = new Long(a);
+ return aa.equals(b);
+ } catch (Exception e) {
+ return false;
+ }
+ }
+
+ public boolean compare(Double a, Double b) {
+ if (a - b > -MIN_VALUE && a - b < MIN_VALUE) {
+ return true;
+ }
+ return false;
+ }
+
+ public boolean compare(Float a, Float b) {
+ if (a - b > -MIN_VALUE && a - b < MIN_VALUE) {
+ return true;
+ }
+ return false;
+ }
+
+ public boolean compare(Date a, Date b) {
+ String ad = dateDataType.toDataJson(a);
+ String bd = dateDataType.toDataJson(b);
+ return compare(ad, bd);
+ }
+
+ public static void main(String args[]) {
+ Equals equals = new Equals();
+
+ String a = "/usr/sbin/sshd";
+ String b = "/usr/sbin/zabbix_agentd";
+
+ long startTime1 = System.currentTimeMillis();
+ for (int i = 0; i < 10000; i++) {
+ a.equals(b);
+ }
+
+ long endTime1 = System.currentTimeMillis() - startTime1;
+ System.out.println("time is:" + endTime1);
+
+ long startTime2 = System.currentTimeMillis();
+ for (int i = 0; i < 10000; i++) {
+ a.trim().equals(b.trim());
+ }
+ long endTime2 = System.currentTimeMillis() - startTime2;
+ System.out.println("time is:" + endTime2);
+
+ }
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/ExpressionFunction.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/ExpressionFunction.java
new file mode 100644
index 0000000..1e5687d
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/ExpressionFunction.java
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.function.expression;
+
+import org.apache.rocketmq.streams.filter.context.RuleContext;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+import org.apache.rocketmq.streams.filter.operator.expression.Expression;
+
+public interface ExpressionFunction {
+ /**
+ * FunctionServiceImpl 函数接口，通过集成此方法声明函数
+ *
+ * @param t
+ * @param context
+ * @param rule
+ * @return
+ */
+ Boolean doFunction(Expression expression, RuleContext context, Rule rule);
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/GreaterEquals.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/GreaterEquals.java
new file mode 100644
index 0000000..8a4fd86
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/GreaterEquals.java
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.function.expression;
+
+import org.apache.rocketmq.streams.filter.context.RuleContext;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+import org.apache.rocketmq.streams.filter.operator.expression.Expression;
+import org.apache.rocketmq.streams.common.datatype.DateDataType;
+import org.apache.rocketmq.streams.script.annotation.Function;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethod;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethodAilas;
+import org.apache.rocketmq.streams.script.utils.FunctionUtils;
+
+import java.util.Date;
+
+@Function
+public class GreaterEquals extends CompareFunction {
+
+ private static final DateDataType dateDataType = new DateDataType(Date.class);
+
+ @FunctionMethod(value = ">=", alias = ">=")
+ @FunctionMethodAilas("大于等于")
+ @Override
+ public Boolean doFunction(Expression expression, RuleContext context, Rule rule) {
+ return super.doFunction(expression, context, rule);
+ }
+
+ public boolean compare(int a, int b) {
+ return a >= b;
+ }
+
+ public boolean compare(Integer a, Integer b) {
+ return a >= b;
+ }
+
+ public boolean compare(String a, int b) {
+ try {
+ a = FunctionUtils.getConstant(a);
+ int aa = new Integer(a).intValue();
+ return aa >= b;
+ } catch (Exception e) {
+ return false;
+ }
+
+ }
+
+ public boolean compare(String a, Integer b) {
+ try {
+ a = FunctionUtils.getConstant(a);
+ Integer aa = new Integer(a);
+ return aa >= b;
+ } catch (Exception e) {
+ return false;
+ }
+ }
+
+ public boolean compare(String a, String b) {
+ a = FunctionUtils.getConstant(a);
+ b = FunctionUtils.getConstant(b);
+ return a.compareTo(b) >= 0;
+ }
+
+ public boolean compare(long a, long b) {
+ return a >= b;
+ }
+
+ public boolean compare(Long a, Long b) {
+ return a >= b;
+ }
+
+ public boolean compare(double a, double b) {
+ return a - b >= 0;
+ }
+
+ public boolean compare(Double a, Double b) {
+ return a - b >= 0;
+ }
+
+ public boolean compare(float a, float b) {
+ return a - b >= 0;
+ }
+
+ public boolean compare(Float a, Float b) {
+ return a - b >= 0;
+ }
+
+ public boolean compare(String a, long b) {
+ try {
+ a = FunctionUtils.getConstant(a);
+ long aa = new Long(a).longValue();
+ return aa >= b;
+ } catch (Exception e) {
+ return false;
+ }
+
+ }
+
+ public boolean compare(String a, Long b) {
+ try {
+ Long aa = new Long(a);
+ return aa >= b;
+ } catch (Exception e) {
+ return false;
+ }
+ }
+
+ public boolean compare(String a, double b) {
+ try {
+ double aa = new Double(a).doubleValue();
+ return aa - b >= 0;
+ } catch (Exception e) {
+ return false;
+ }
+
+ }
+
+ public boolean compare(String a, Double b) {
+ try {
+ Double aa = new Double(a);
+ return aa - b >= 0;
+ } catch (Exception e) {
+ return false;
+ }
+ }
+
+ public boolean compare(String a, float b) {
+ try {
+ float aa = new Float(a).floatValue();
+ return aa - b >= 0;
+ } catch (Exception e) {
+ return false;
+ }
+ }
+
+ public boolean compare(String a, Float b) {
+ try {
+ Float aa = new Float(a);
+ return aa - b >= 0;
+ } catch (Exception e) {
+ return false;
+ }
+ }
+
+ public boolean compare(Date a, Date b) {
+ String ad = dateDataType.toDataJson(a);
+ String bd = dateDataType.toDataJson(b);
+ return compare(ad, bd);
+ }
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/GreaterThan.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/GreaterThan.java
new file mode 100644
index 0000000..185f91a
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/GreaterThan.java
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.function.expression;
+
+import org.apache.rocketmq.streams.filter.context.RuleContext;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+import org.apache.rocketmq.streams.filter.operator.expression.Expression;
+import org.apache.rocketmq.streams.common.datatype.DateDataType;
+import org.apache.rocketmq.streams.script.annotation.Function;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethod;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethodAilas;
+import org.apache.rocketmq.streams.script.utils.FunctionUtils;
+
+import java.util.Date;
+
+@Function
+
+public class GreaterThan extends CompareFunction {
+
+ private static final DateDataType dateDataType = new DateDataType(Date.class);
+
+ @FunctionMethod(value = ">", alias = ">")
+ @FunctionMethodAilas("大于")
+ @Override
+ public Boolean doFunction(Expression expression, RuleContext context, Rule rule) {
+ return super.doFunction(expression, context, rule);
+ }
+
+ public boolean compare(int a, int b) {
+ return a > b;
+ }
+
+ public boolean compare(Integer a, Integer b) {
+
+ return a > b;
+ }
+
+ public boolean compare(String a, Integer b) {
+ try {
+ a = FunctionUtils.getConstant(a);
+ Integer aa = new Integer(a);
+ return aa > b;
+ } catch (Exception e) {
+ return false;
+ }
+ }
+
+ public boolean compare(String a, String b) {
+ a = FunctionUtils.getConstant(a);
+ b = FunctionUtils.getConstant(b);
+ return a.compareTo(b) > 0;
+ }
+
+ public boolean compare(long a, long b) {
+ return a > b;
+ }
+
+ public boolean compare(double a, double b) {
+ return a - b > 0;
+ }
+
+ public boolean compare(float a, float b) {
+ return a - b > 0;
+ }
+
+ public boolean compare(Long a, Long b) {
+ return a > b;
+ }
+
+ public boolean compare(String a, Long b) {
+ try {
+ a = FunctionUtils.getConstant(a);
+ Long aa = new Long(a);
+ return aa > b;
+ } catch (Exception e) {
+ return false;
+ }
+ }
+
+ public boolean compare(Double a, Double b) {
+ return a - b > 0;
+ }
+
+ public boolean compare(String a, Double b) {
+ try {
+ a = FunctionUtils.getConstant(a);
+ Double aa = new Double(a);
+ return aa - b > 0;
+ } catch (Exception e) {
+ return false;
+ }
+ }
+
+ public boolean compare(Float a, Float b) {
+ return a - b > 0;
+ }
+
+ public boolean compare(String a, Float b) {
+ try {
+ a = FunctionUtils.getConstant(a);
+ Float aa = new Float(a);
+ return aa - b > 0;
+ } catch (Exception e) {
+ return false;
+ }
+ }
+
+ public boolean compare(Date a, Date b) {
+ String ad = dateDataType.toDataJson(a);
+ String bd = dateDataType.toDataJson(b);
+ return compare(ad, bd);
+ }
+
+ public static void main(String args[]) {
+ GreaterThan gt = new GreaterThan();
+ System.out.println(gt.compare("1201", "1200"));
+ }
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/InFunction.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/InFunction.java
new file mode 100644
index 0000000..25e7e04
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/InFunction.java
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.function.expression;
+
+import org.apache.rocketmq.streams.filter.context.RuleContext;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+import org.apache.rocketmq.streams.filter.operator.expression.Expression;
+import org.apache.rocketmq.streams.filter.operator.var.Var;
+import org.apache.rocketmq.streams.common.cache.softreference.ICache;
+import org.apache.rocketmq.streams.common.cache.softreference.impl.SoftReferenceCache;
+import org.apache.rocketmq.streams.script.annotation.Function;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethod;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethodAilas;
+import org.apache.rocketmq.streams.script.utils.FunctionUtils;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+
+import java.util.HashSet;
+import java.util.Set;
+
+@Function
+public class InFunction extends AbstractExpressionFunction {
+
+ /**
+ * in 的字符串形成set，可以最快o（1）来提高效率
+ */
+ private ICache<Object, Set<String>> cache = new SoftReferenceCache();
+
+ @Override
+ @FunctionMethod(value = "in", alias = "~in")
+ @FunctionMethodAilas("包含")
+ public Boolean doExpressionFunction(Expression expression, RuleContext context, Rule rule) {
+ if (!expression.volidate()) {
+ return false;
+ }
+
+ Var var = context.getVar(rule.getConfigureName(), expression.getVarName());
+ if (var == null) {
+ return false;
+ }
+ Object varObject = null;
+ Object valueObject = null;
+ varObject = var.getVarValue(context, rule);
+ valueObject = expression.getValue();
+
+ if (varObject == null || valueObject == null) {
+ return false;
+ }
+ String varString = String.valueOf(varObject).trim();
+ Set<String> valueSet = cache.get(valueObject);
+ if (valueSet != null) {
+ return valueSet.contains(varString);
+ }
+
+ String valueString = "";
+
+ valueString = String.valueOf(valueObject).trim();
+ if (StringUtil.isEmpty(valueString)) {
+ return false;
+ }
+ String[] values = valueString.split(",");
+ Set<String> set = new HashSet<>();
+ for (String value : values) {
+ value = FunctionUtils.getConstant(value);
+ set.add(value);
+
+ }
+ cache.put(valueObject, set);
+
+ return set.contains(varString);
+ }
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/InMetaData.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/InMetaData.java
new file mode 100644
index 0000000..27aaed1
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/InMetaData.java
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.function.expression;
+
+import org.apache.rocketmq.streams.filter.context.RuleContext;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+import org.apache.rocketmq.streams.filter.operator.expression.Expression;
+import org.apache.rocketmq.streams.filter.operator.var.Var;
+import org.apache.rocketmq.streams.script.annotation.Function;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethod;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethodAilas;
+
+@Function
+public class InMetaData extends AbstractExpressionFunction {
+
+ @FunctionMethod("engine_in")
+ @FunctionMethodAilas("in(英文逗号分隔)")
+ @Override
+ public Boolean doExpressionFunction(Expression expression, RuleContext context, Rule rule) {
+ if (!expression.volidate()) {
+ return false;
+ }
+
+ Var var = context.getVar(rule.getConfigureName(), expression.getVarName());
+ if (var == null) {
+ return false;
+ }
+ Object varObject = null;
+ Object valueObject = null;
+ varObject = var.getVarValue(context, rule);
+ valueObject = expression.getValue();
+
+ if (varObject == null || valueObject == null) {
+ return false;
+ }
+
+ String varString = "";
+ String inStrings = "";
+ varString = String.valueOf(varObject).trim();
+ inStrings = String.valueOf(valueObject).trim();
+ String[] instr = inStrings.split(",");
+ for (String str : instr) {
+ if (str.trim().equals(varString)) {
+ return true;
+ }
+ }
+
+ return false;
+
+ }
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/IpContainsFunction.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/IpContainsFunction.java
new file mode 100644
index 0000000..d72cbff
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/IpContainsFunction.java
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.function.expression;
+
+import org.apache.rocketmq.streams.filter.context.RuleContext;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+import org.apache.rocketmq.streams.filter.operator.expression.Expression;
+import org.apache.rocketmq.streams.filter.utils.IPUtil;
+import org.apache.rocketmq.streams.filter.operator.var.Var;
+import org.apache.rocketmq.streams.script.annotation.Function;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethod;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethodAilas;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+@Function
+public class IpContainsFunction extends AbstractExpressionFunction {
+
+ private static final Log LOG = LogFactory.getLog(IpContainsFunction.class);
+
+ private String ip;
+ private long start;
+ private long end;
+ private String startIp;
+ private String endIp;
+
+ @SuppressWarnings("rawtypes")
+ @Override
+ @FunctionMethod("ipContains")
+ @FunctionMethodAilas("ip包含")
+ public Boolean doExpressionFunction(Expression expression, RuleContext context, Rule rule) {
+ try {
+ if (!expression.volidate()) {
+ return false;
+ }
+
+ Var var = context.getVar(rule.getConfigureName(), expression.getVarName());
+ if (var == null) {
+ return false;
+ }
+ Object varObject = null;
+ Object valueObject = null;
+ varObject = var.getVarValue(context, rule);
+ valueObject = expression.getValue();
+
+ if (varObject == null || valueObject == null) {
+ return false;
+ }
+
+ String varString = "";
+ String regex = "";
+ varString = String.valueOf(varObject).trim();
+ regex = String.valueOf(valueObject).trim();
+
+ if (StringUtil.isEmpty(varString) || StringUtil.isEmpty(regex)) {
+ return false;
+ }
+ parseRegexIp(regex);
+ return contains(varString);
+ } catch (Exception e) {
+ LOG.error("IpContainsFunction doExpressionFunction error", e);
+ return false;
+ }
+
+ }
+
+ private void parseRegexIp(String ip) {
+ if (null == ip || "".equals(ip)) {
+ return;
+ }
+
+ int n = ip.indexOf("/");
+ String preIp = ip;
+ int mask = 32;
+ if (n > 0) {
+ preIp = ip.substring(0, n);
+ mask = Integer.parseInt(ip.substring(n + 1));
+ if (!IPUtil.checkMask(mask)) {
+ return;
+ }
+ }
+
+ if (!IPUtil.checkIpFormat(preIp)) {
+ return;
+ }
+ this.ip = ip;
+ this.start = getStart(preIp, mask);
+ this.end = getEnd(preIp, mask);
+ this.startIp = IPUtil.ipToString(this.start);
+ this.endIp = IPUtil.ipToString(this.end);
+ }
+
+ private long getStart(String ip, int mask) {
+ return IPUtil.ipToInt(ip) & IPUtil.getPrefixIp(mask);
+ }
+
+ private long getEnd(String ip, int mask) {
+ return getStart(ip, mask) | IPUtil.getSuffixIp(mask);
+ }
+
+ /*
+ * 判断IP是否包含
+ */
+ private boolean contains(String ip) {
+ // 先判断是否为ip段
+ int n = ip.indexOf("/");
+ if (n <= 0) {
+ if (!IPUtil.checkIpFormat(ip)) {
+ return false;
+ }
+
+ long intIp = IPUtil.ipToInt(ip);
+ return this.start <= intIp && this.end >= intIp;
+ }
+
+ String preIp = ip.substring(0, n);
+ int mask = Integer.parseInt(ip.substring(n + 1));
+ if (!IPUtil.checkMask(mask) || !IPUtil.checkIpFormat(preIp)) {
+ return false;
+ }
+
+ return this.start <= this.getStart(preIp, mask) && this.end >= this.getEnd(preIp, mask);
+ }
+
+ public String getIp() {
+ return ip;
+ }
+
+ public void setIp(String ip) {
+ this.ip = ip;
+ }
+
+ public long getStart() {
+ return start;
+ }
+
+ public void setStart(long start) {
+ this.start = start;
+ }
+
+ public long getEnd() {
+ return end;
+ }
+
+ public void setEnd(long end) {
+ this.end = end;
+ }
+
+ public String getStartIp() {
+ return startIp;
+ }
+
+ public void setStartIp(String startIp) {
+ this.startIp = startIp;
+ }
+
+ public String getEndIp() {
+ return endIp;
+ }
+
+ public void setEndIp(String endIp) {
+ this.endIp = endIp;
+ }
+
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/IsNotNull.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/IsNotNull.java
new file mode 100644
index 0000000..306fcd3
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/IsNotNull.java
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.function.expression;
+
+import org.apache.rocketmq.streams.filter.context.RuleContext;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+import org.apache.rocketmq.streams.filter.operator.expression.Expression;
+import org.apache.rocketmq.streams.filter.operator.var.Var;
+import org.apache.rocketmq.streams.script.annotation.Function;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethod;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethodAilas;
+
+@Function
+public class IsNotNull extends AbstractExpressionFunction {
+
+ @Override
+ @FunctionMethod(value = "isNotNull", alias = "notNull")
+ @FunctionMethodAilas("不为空")
+ public Boolean doExpressionFunction(Expression expression, RuleContext context, Rule rule) {
+ Var var = context.getVar(rule.getConfigureName(), expression.getVarName());
+ if (var == null) {
+ return false;
+ }
+ Object varObject = null;
+ varObject = var.getVarValue(context, rule);
+ if (varObject == null) {
+ return false;
+ }
+
+ String varString = String.valueOf(varObject).trim();
+ if ("".equals(varString)) {
+ return false;
+ }
+ return true;
+ }
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/IsNull.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/IsNull.java
new file mode 100644
index 0000000..7c8a963
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/IsNull.java
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.function.expression;
+
+import org.apache.rocketmq.streams.filter.context.RuleContext;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+import org.apache.rocketmq.streams.filter.operator.expression.Expression;
+import org.apache.rocketmq.streams.filter.operator.var.Var;
+import org.apache.rocketmq.streams.script.annotation.Function;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethod;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethodAilas;
+
+@Function
+public class IsNull extends AbstractExpressionFunction {
+
+ @Override
+ @FunctionMethod(value = "isNull", alias = "null")
+ @FunctionMethodAilas("为空")
+ public Boolean doExpressionFunction(Expression expression, RuleContext context, Rule rule) {
+ Var var = context.getVar(rule.getConfigureName(), expression.getVarName());
+ if (var == null) {
+ return true;
+ }
+ Object varObject = null;
+ varObject = var.getVarValue(context, rule);
+ if (varObject == null) {
+ return true;
+ }
+ String varString = String.valueOf(varObject).trim();
+ if ("".equals(varString)) {
+ return true;
+ }
+ return false;
+ }
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/LessEquals.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/LessEquals.java
new file mode 100644
index 0000000..da0e5b0
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/LessEquals.java
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.function.expression;
+
+import org.apache.rocketmq.streams.filter.context.RuleContext;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+import org.apache.rocketmq.streams.filter.operator.expression.Expression;
+import org.apache.rocketmq.streams.common.datatype.DateDataType;
+import org.apache.rocketmq.streams.script.annotation.Function;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethod;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethodAilas;
+
+import java.util.Date;
+
+@Function
+public class LessEquals extends CompareFunction {
+
+ private static final DateDataType dateDataType = new DateDataType(Date.class);
+
+ @FunctionMethod(value = "<=", alias = "<=")
+ @FunctionMethodAilas("小于等于")
+ @Override
+ public Boolean doFunction(Expression expression, RuleContext context, Rule rule) {
+ return super.doFunction(expression, context, rule);
+ }
+
+ public boolean compare(int a, int b) {
+ return a <= b;
+ }
+
+ public boolean compare(Integer a, Integer b) {
+ return a <= b;
+ }
+
+ public boolean compare(String a, Integer b) {
+ try {
+ Integer aa = new Integer(a);
+ return aa <= b;
+ } catch (Exception e) {
+ return false;
+ }
+ }
+
+ public boolean compare(String a, String b) {
+ return a.compareTo(b) <= 0;
+ }
+
+ public boolean compare(long a, long b) {
+ return a <= b;
+ }
+
+ public boolean compare(double a, double b) {
+ return a - b <= 0;
+ }
+
+ public boolean compare(float a, float b) {
+ return a - b <= 0;
+ }
+
+ public boolean compare(Long a, Long b) {
+ return a <= b;
+ }
+
+ public boolean compare(String a, Long b) {
+ try {
+ Long aa = new Long(a);
+ return aa <= b;
+ } catch (Exception e) {
+ return false;
+ }
+ }
+
+ public boolean compare(Double a, Double b) {
+ return a - b <= 0;
+ }
+
+ public boolean compare(String a, Double b) {
+ try {
+ Double aa = new Double(a);
+ return aa - b <= 0;
+ } catch (Exception e) {
+ return false;
+ }
+ }
+
+ public boolean compare(Float a, Float b) {
+ return a - b <= 0;
+ }
+
+ public boolean compare(String a, Float b) {
+ try {
+ Float aa = new Float(a);
+ return aa - b <= 0;
+ } catch (Exception e) {
+ return false;
+ }
+ }
+
+ public boolean compare(Date a, Date b) {
+ String ad = dateDataType.toDataJson(a);
+ String bd = dateDataType.toDataJson(b);
+ return compare(ad, bd);
+ }
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/LessThan.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/LessThan.java
new file mode 100644
index 0000000..14683eb
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/LessThan.java
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.function.expression;
+
+import org.apache.rocketmq.streams.filter.context.RuleContext;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+import org.apache.rocketmq.streams.filter.operator.expression.Expression;
+import org.apache.rocketmq.streams.common.datatype.DateDataType;
+import org.apache.rocketmq.streams.script.annotation.Function;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethod;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethodAilas;
+
+import java.util.Date;
+
+@Function
+
+public class LessThan extends CompareFunction {
+
+ private static final DateDataType dateDataType = new DateDataType(Date.class);
+
+ @Override
+ @FunctionMethod(value = "<", alias = "<")
+ @FunctionMethodAilas("小于")
+ public Boolean doFunction(Expression expression, RuleContext context, Rule rule) {
+ return super.doFunction(expression, context, rule);
+ }
+
+ public boolean compare(int a, int b) {
+ return a < b;
+ }
+
+ public boolean compare(Integer a, Integer b) {
+ return a < b;
+ }
+
+ public boolean compare(String a, Integer b) {
+ try {
+ Integer aa = new Integer(a);
+ return aa < b;
+ } catch (Exception e) {
+ return false;
+ }
+ }
+
+ public boolean compare(String a, String b) {
+ return a.compareTo(b) < 0;
+ }
+
+ public boolean compare(long a, long b) {
+ return a < b;
+ }
+
+ public boolean compare(double a, double b) {
+ return a - b < 0;
+ }
+
+ public boolean compare(float a, float b) {
+ return a - b < 0;
+ }
+
+ public boolean compare(Long a, Long b) {
+ return a < b;
+ }
+
+ public boolean compare(String a, Long b) {
+ try {
+ Long aa = new Long(a);
+ return aa < b;
+ } catch (Exception e) {
+ return false;
+ }
+ }
+
+ public boolean compare(Double a, Double b) {
+ return a - b < 0;
+ }
+
+ public boolean compare(String a, Double b) {
+ try {
+ Double aa = new Double(a);
+ return aa - b < 0;
+ } catch (Exception e) {
+ return false;
+ }
+ }
+
+ public boolean compare(Float a, Float b) {
+ return a - b < 0;
+ }
+
+ public boolean compare(String a, Float b) {
+ try {
+ Float aa = new Float(a);
+ return aa - b < 0;
+ } catch (Exception e) {
+ return false;
+ }
+ }
+
+ public boolean compare(Date a, Date b) {
+ String ad = dateDataType.toDataJson(a);
+ String bd = dateDataType.toDataJson(b);
+ return compare(ad, bd);
+ }
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/LikeFunction.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/LikeFunction.java
new file mode 100644
index 0000000..e864202
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/LikeFunction.java
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.function.expression;
+
+import org.apache.rocketmq.streams.filter.context.RuleContext;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+import org.apache.rocketmq.streams.filter.operator.expression.Expression;
+import org.apache.rocketmq.streams.filter.operator.var.Var;
+import org.apache.rocketmq.streams.common.cache.softreference.ICache;
+import org.apache.rocketmq.streams.common.cache.softreference.impl.SoftReferenceCache;
+import org.apache.rocketmq.streams.script.annotation.Function;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethod;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethodAilas;
+import org.apache.rocketmq.streams.common.optimization.LikeRegex;
+import org.apache.rocketmq.streams.script.utils.FunctionUtils;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+
+@Function
+public class LikeFunction extends AbstractExpressionFunction {
+ private transient ICache<String, LikeRegex> likeCache = new SoftReferenceCache<>();
+
+ private transient ICache<String, LikeCache> cache = new SoftReferenceCache<>();
+
+ public static boolean isLikeFunciton(String functionName) {
+ return "like".equals(functionName);
+ }
+
+ private class LikeCache {
+ public LikeCache(String containStr, String regexStr, boolean isPrefix) {
+ this.containStr = containStr;
+ this.regexStr = regexStr;
+ this.isPrefix = isPrefix;
+
+ }
+
+ protected String containStr;//预处理like中一定包含的字符串
+ protected String regexStr;//把like转换成regex对应的字符串
+ protected boolean isPrefix = false;//是否是前缀匹配
+
+ }
+
+ @Override
+ @FunctionMethod("like")
+ @FunctionMethodAilas("包含")
+ public Boolean doExpressionFunction(Expression expression, RuleContext context, Rule rule) {
+ if (!expression.volidate()) {
+ return false;
+ }
+
+ Var var = context.getVar(rule.getConfigureName(), expression.getVarName());
+ if (var == null) {
+ return false;
+ }
+ Object varObject = null;
+ Object valueObject = null;
+ varObject = var.getVarValue(context, rule);
+ valueObject = expression.getValue();
+
+ if (varObject == null || valueObject == null) {
+ return false;
+ }
+
+ String varString = "";
+ String valueString = "";
+ varString = String.valueOf(varObject).trim();
+ valueString = String.valueOf(valueObject).trim();
+ if (StringUtil.isEmpty(valueString)) {
+ return false;
+ }
+ valueString = FunctionUtils.getConstant(valueString);
+ LikeRegex likeRegex = likeCache.get(valueString);
+ if (likeRegex == null) {
+ likeRegex = new LikeRegex(valueString);
+ likeCache.put(valueString, likeRegex);
+ }
+ boolean likeResult = likeRegex.match(varString);
+ // System.out.println(likeResult);
+ return likeResult;
+
+ //LikeCache likeCache=sinkcache.get(valueString);
+ //if(likeCache==null){
+ // String containStr=parseContainStr(valueString);
+ // String regexStr=convertRegex(valueString);
+ // likeCache=new LikeCache(containStr,regexStr,!valueString.startsWith("%"));
+ // sinkcache.put(valueString,likeCache);
+ //}
+ //if(likeCache.containStr!=null){
+ // if(likeCache.isPrefix&&!varString.startsWith(likeCache.containStr)){
+ // return false;
+ // }
+ // if(varString.indexOf(likeCache.containStr)==-1){
+ // return false;
+ // }
+ //}
+ //boolean result= StringUtil.matchRegex(varString,likeCache.regexStr);
+ //return result;
+ }
+
+ protected static String[] regexSpecialWords = {"\\", "$", "(", ")", "*", "+", ".", "[", "]", "?", "^", "{", "}", "|"};
+
+ /**
+ * 把like中一定包含的字符串抽取出来，再正则匹配前，先做预处理
+ *
+ * @param likeStr like的语句
+ * @return 如果能够抽取，返回抽取的字符串，否则返回null
+ */
+ protected static String parseContainStr(String likeStr) {
+ String tmp = likeStr;
+ if (tmp.startsWith("%")) {
+ tmp = tmp.substring(1);
+ }
+ if (tmp.endsWith("%")) {
+ tmp = tmp.substring(0, tmp.length() - 1);
+ }
+ for (String word : regexSpecialWords) {
+ if (tmp.contains(word)) {
+ return null;
+ }
+ }
+ return null;
+ }
+
+ /**
+ * regexWord.add("."); regexWord.add("*"); regexWord.add("?"); regexWord.add("{"); regexWord.add("}"); // regexWord.add("\\d"); regexWord.add(")"); // regexWord.add("("); regexWord.add("+"); regexWord.add("\\"); regexWord.add("["); regexWord.add("]"); regexWord.add("^"); regexWord.add("$");
+ *
+ * @param likeStr
+ * @return
+ */
+
+ protected static String convertRegex(String likeStr) {
+ likeStr = likeStr.replace("\\", "\\\\");
+ likeStr = likeStr.replace("[!", "[^");
+ likeStr = likeStr.replace(".", "\\.");
+ likeStr = likeStr.replace("*", "\\*");
+ likeStr = likeStr.replace("_", ".");
+ likeStr = likeStr.replace("+", "\\+");
+
+ if (!likeStr.trim().startsWith("%")) {
+ likeStr = "^" + likeStr;
+ }
+ if (!likeStr.trim().endsWith("%")) {
+ likeStr = likeStr + "$";
+ }
+ likeStr = likeStr.replace("%", ".*");
+ return likeStr;
+ }
+
+ public static void main(String[] args) {
+ String valueString = convertRegex("%/php%");
+ System.out.println(valueString);
+ System.out.println(StringUtil.matchRegex("/www/server/php/70/sbin/php-fpm", valueString));
+ }
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/NotContainsCaseInsensitiveFunction.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/NotContainsCaseInsensitiveFunction.java
new file mode 100644
index 0000000..1a8d815
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/NotContainsCaseInsensitiveFunction.java
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.function.expression;
+
+import org.apache.rocketmq.streams.filter.context.RuleContext;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+import org.apache.rocketmq.streams.filter.operator.expression.Expression;
+import org.apache.rocketmq.streams.filter.operator.var.Var;
+import org.apache.rocketmq.streams.script.annotation.Function;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethod;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethodAilas;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+@Function
+
+public class NotContainsCaseInsensitiveFunction extends AbstractExpressionFunction {
+
+ private static final Log LOG = LogFactory.getLog(NotContainsCaseInsensitiveFunction.class);
+
+ @Override
+ @FunctionMethod(value = "notContainsCaseInsensitiveFunction", alias = "~notContains")
+ @FunctionMethodAilas("不包含(忽略大小写)")
+ public Boolean doExpressionFunction(Expression expression, RuleContext context, Rule rule) {
+
+ try {
+ if (!expression.volidate()) {
+ return false;
+ }
+
+ Var var = context.getVar(rule.getConfigureName(), expression.getVarName());
+ if (var == null) {
+ return false;
+ }
+ Object varObject = null;
+ Object valueObject = null;
+ varObject = var.getVarValue(context, rule);
+ valueObject = expression.getValue();
+ if (varObject == null) {
+ return true;
+ }
+ if (valueObject == null) {
+ return false;
+ }
+
+ String varString = "";
+ String valueString = "";
+ varString = String.valueOf(varObject).trim();
+ valueString = String.valueOf(valueObject).trim();
+ if (varString.toLowerCase().contains(valueString.toLowerCase())) {
+ return false;
+ } else {
+ return true;
+ }
+ } catch (Exception e) {
+ LOG.error("ContainsCaseInsensitiveFunction error: rule name is: " + rule.getConfigureName(), e);
+ return false;
+ }
+
+ }
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/NotContainsFunction.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/NotContainsFunction.java
new file mode 100644
index 0000000..1e4974c
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/NotContainsFunction.java
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.function.expression;
+
+import org.apache.rocketmq.streams.filter.context.RuleContext;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+import org.apache.rocketmq.streams.filter.operator.expression.Expression;
+import org.apache.rocketmq.streams.filter.operator.var.Var;
+import org.apache.rocketmq.streams.script.annotation.Function;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethod;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethodAilas;
+
+@Function
+public class NotContainsFunction extends AbstractExpressionFunction {
+
+ @Override
+ @FunctionMethod("notContains")
+ @FunctionMethodAilas("不包含")
+ public Boolean doExpressionFunction(Expression expression, RuleContext context, Rule rule) {
+ if (!expression.volidate()) {
+ return false;
+ }
+
+ Var var = context.getVar(rule.getConfigureName(), expression.getVarName());
+ if (var == null) {
+ return false;
+ }
+ Object varObject = null;
+ Object valueObject = null;
+ varObject = var.getVarValue(context, rule);
+ valueObject = expression.getValue();
+ if (varObject == null) {
+ return true;
+ }
+ if (valueObject == null) {
+ return false;
+ }
+ String varString = "";
+ String valueString = "";
+ varString = String.valueOf(varObject).trim();
+ valueString = String.valueOf(valueObject).trim();
+
+ if (varString.contains(valueString)) {
+ return false;
+ } else {
+ return true;
+ }
+ }
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/NotEquals.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/NotEquals.java
new file mode 100644
index 0000000..8a57072
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/NotEquals.java
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.function.expression;
+
+import org.apache.rocketmq.streams.filter.context.RuleContext;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+import org.apache.rocketmq.streams.filter.operator.expression.Expression;
+import org.apache.rocketmq.streams.common.datatype.DateDataType;
+import org.apache.rocketmq.streams.script.annotation.Function;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethod;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethodAilas;
+import org.apache.rocketmq.streams.script.utils.FunctionUtils;
+
+import java.util.Date;
+
+@Function
+
+public class NotEquals extends CompareFunction {
+
+ private final static double MIN_VALUE = 0.000001;
+ private static final DateDataType dateDataType = new DateDataType(Date.class);
+
+ @FunctionMethod(value = "!=", alias = "<>")
+ @FunctionMethodAilas("不等于")
+ @Override
+ public Boolean doFunction(Expression expression, RuleContext context, Rule rule) {
+ return super.doFunction(expression, context, rule);
+ }
+
+ public boolean compare(int a, int b) {
+ return a != b;
+ }
+
+ public boolean compare(Integer a, Integer b) {
+ return !a.equals(b);
+ }
+
+ public boolean compare(String a, Integer b) {
+ try {
+ Integer aa = new Integer(a);
+ return !aa.equals(b);
+ } catch (Exception e) {
+ return false;
+ }
+ }
+
+ public boolean compare(String a, String b) {
+ a = FunctionUtils.getConstant(a);
+ b = FunctionUtils.getConstant(b);
+ return !a.equals(b);
+ }
+
+ public boolean compare(boolean a, boolean b) {
+ return a != b;
+ }
+
+ public boolean compare(long a, long b) {
+ return a != b;
+ }
+
+ public boolean compare(double a, double b) {
+ if (a - b > -MIN_VALUE && a - b < MIN_VALUE) {
+ return false;
+ } else {
+ return true;
+ }
+
+ }
+
+ public boolean compare(float a, float b) {
+ if (a - b > -MIN_VALUE && a - b < MIN_VALUE) {
+ return true;
+ } else {
+ return false;
+ }
+
+ }
+
+ public boolean compare(Boolean a, Boolean b) {
+ return !a.equals(b);
+ }
+
+ public boolean compare(Long a, Long b) {
+ return !a.equals(b);
+ }
+
+ public boolean compare(String a, Long b) {
+ try {
+ a = FunctionUtils.getConstant(a);
+ Long aa = new Long(a);
+ return !aa.equals(b);
+ } catch (Exception e) {
+ return false;
+ }
+ }
+
+ public boolean compare(Double a, Double b) {
+ if (a - b > -MIN_VALUE && a - b < MIN_VALUE) {
+ return false;
+ } else {
+ return true;
+ }
+
+ }
+
+ public boolean compare(String a, Double b) {
+ try {
+ a = FunctionUtils.getConstant(a);
+ Double aa = new Double(a);
+ if (aa - b > -MIN_VALUE && aa - b < MIN_VALUE) {
+ return false;
+ } else {
+ return true;
+ }
+ } catch (Exception e) {
+ return false;
+ }
+ }
+
+ public boolean compare(Float a, Float b) {
+ if (a - b > -MIN_VALUE && a - b < MIN_VALUE) {
+ return true;
+ } else {
+ return false;
+ }
+
+ }
+
+ public boolean compare(String a, Float b) {
+ try {
+ a = FunctionUtils.getConstant(a);
+ Float aa = new Float(a);
+ if (aa - b > -MIN_VALUE && aa - b < MIN_VALUE) {
+ return false;
+ } else {
+ return true;
+ }
+ } catch (Exception e) {
+ return false;
+ }
+ }
+
+ public boolean compare(Date a, Date b) {
+ String ad = dateDataType.toDataJson(a);
+ String bd = dateDataType.toDataJson(b);
+ return compare(ad, bd);
+ }
+
+ public static void main(String[] args) {
+ Boolean a = true;
+ Boolean b = false;
+ NotEquals not = new NotEquals();
+ System.out.println(not.compare(a, b));
+ }
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/NotInFunction.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/NotInFunction.java
new file mode 100644
index 0000000..602fe30
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/NotInFunction.java
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.function.expression;
+
+import org.apache.rocketmq.streams.filter.context.RuleContext;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+import org.apache.rocketmq.streams.filter.operator.expression.Expression;
+import org.apache.rocketmq.streams.script.annotation.Function;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethod;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethodAilas;
+
+@Function
+public class NotInFunction extends InFunction {
+
+ @Override
+ @FunctionMethod(value = "!in", alias = "~!in")
+ @FunctionMethodAilas("包含")
+ public Boolean doExpressionFunction(Expression expression, RuleContext context, Rule rule) {
+ return !super.doExpressionFunction(expression, context, rule);
+ }
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/NotLikeFunction.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/NotLikeFunction.java
new file mode 100644
index 0000000..e22f4a4
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/NotLikeFunction.java
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.function.expression;
+
+import org.apache.rocketmq.streams.filter.context.RuleContext;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+import org.apache.rocketmq.streams.filter.operator.expression.Expression;
+import org.apache.rocketmq.streams.script.annotation.Function;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethod;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethodAilas;
+
+@Function
+public class NotLikeFunction extends LikeFunction {
+
+ @Override
+ @FunctionMethod(value = "notLike", alias = "!like")
+ @FunctionMethodAilas("not like")
+ public Boolean doExpressionFunction(Expression expression, RuleContext context, Rule rule) {
+ return !super.doExpressionFunction(expression, context, rule);
+ }
+
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/NotRegexCaseInsensitiveFunction.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/NotRegexCaseInsensitiveFunction.java
new file mode 100644
index 0000000..9422b90
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/NotRegexCaseInsensitiveFunction.java
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.function.expression;
+
+import org.apache.rocketmq.streams.filter.context.RuleContext;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+import org.apache.rocketmq.streams.filter.operator.expression.Expression;
+import org.apache.rocketmq.streams.script.annotation.Function;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethod;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethodAilas;
+
+@Function
+
+public class NotRegexCaseInsensitiveFunction extends RegexCaseInsensitiveFunction {
+
+ @Override
+ @FunctionMethod(value = "notRegexCaseInsensitive", alias = "~!regex")
+ @FunctionMethodAilas("正则不匹配(忽略大小写)")
+ public Boolean doExpressionFunction(Expression expression, RuleContext context, Rule rule) {
+ return !super.doExpressionFunction(expression, context, rule);
+ }
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/NotRegexFunction.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/NotRegexFunction.java
new file mode 100644
index 0000000..6d28fdc
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/NotRegexFunction.java
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.function.expression;
+
+import org.apache.rocketmq.streams.filter.context.RuleContext;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+import org.apache.rocketmq.streams.filter.operator.expression.Expression;
+import org.apache.rocketmq.streams.script.annotation.Function;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethod;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethodAilas;
+
+@Function
+
+public class NotRegexFunction extends RegexFunction {
+
+ @Override
+ @FunctionMethod(value = "notRegex", alias = "!regex")
+ @FunctionMethodAilas("正则不匹配")
+ public Boolean doExpressionFunction(Expression expression, RuleContext context, Rule rule) {
+ return !super.doExpressionFunction(expression, context, rule);
+ }
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/RegexCaseInsensitiveFunction.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/RegexCaseInsensitiveFunction.java
new file mode 100644
index 0000000..1b8003f
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/RegexCaseInsensitiveFunction.java
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.function.expression;
+
+import org.apache.rocketmq.streams.filter.context.RuleContext;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+import org.apache.rocketmq.streams.filter.operator.expression.Expression;
+import org.apache.rocketmq.streams.script.annotation.Function;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethod;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethodAilas;
+
+@Function
+
+public class RegexCaseInsensitiveFunction extends RegexFunction {
+
+ @Override
+ @FunctionMethod(value = "regexCaseInsensitive", alias = "~regex")
+ @FunctionMethodAilas("正则匹配(忽略大小写)")
+ public Boolean doExpressionFunction(Expression expression, RuleContext context, Rule rule) {
+ return super.doExpressionFunction(expression, context, rule);
+ }
+
+ @Override
+ protected boolean caseInsensitive() {
+ return true;
+ }
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/RegexFunction.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/RegexFunction.java
new file mode 100644
index 0000000..591dbc7
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/RegexFunction.java
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.function.expression;
+
+import org.apache.rocketmq.streams.filter.context.RuleContext;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+import org.apache.rocketmq.streams.filter.operator.expression.Expression;
+import org.apache.rocketmq.streams.filter.operator.var.Var;
+import org.apache.rocketmq.streams.script.annotation.Function;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethod;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethodAilas;
+import org.apache.rocketmq.streams.common.context.Message;
+import org.apache.rocketmq.streams.common.optimization.CalculationResultCache;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+@Function
+
+public class RegexFunction extends AbstractExpressionFunction {
+
+ protected static CalculationResultCache calculationResultCache = CalculationResultCache.getInstance();
+
+ private static final Log LOG = LogFactory.getLog(RegexFunction.class);
+ private static final int REGEX_TIME_OUT = -1;
+
+ public static boolean isRegex(String functionName) {
+ if ("regex".equals(functionName) || "regexCaseInsensitive".equals(functionName) || "~regex".equals(functionName)) {
+ return true;
+ }
+ return false;
+ }
+
+ public static boolean isNotRegex(String functionName) {
+ if ("notRegex".equals(functionName) || "!regex".equals(functionName) || "notRegexCaseInsensitive".equals(functionName) || "~!regex".equals(functionName)) {
+ return true;
+ }
+ return false;
+ }
+
+ @SuppressWarnings("rawtypes")
+ @Override
+ @FunctionMethod("regex")
+ @FunctionMethodAilas("正则匹配")
+ public Boolean doExpressionFunction(Expression expression, RuleContext context, Rule rule) {
+ if (!expression.volidate()) {
+ return false;
+ }
+ Message message = context.getMessage();
+
+ Var var = context.getVar(rule.getConfigureName(), expression.getVarName());
+ if (var == null) {
+ return false;
+ }
+ Object varObject = null;
+ Object valueObject = null;
+ varObject = var.getVarValue(context, rule);
+
+ valueObject = expression.getValue();
+
+ if (varObject == null || valueObject == null) {
+ return false;
+ }
+
+ String varString = "";
+ String regex = "";
+ varString = String.valueOf(varObject).trim();
+ regex = String.valueOf(valueObject).trim();
+ Boolean isMatch = calculationResultCache.match(regex, varString);
+ if (isMatch != null) {
+ return isMatch;
+ }
+ boolean value = false;
+ if (caseInsensitive()) {
+ value = StringUtil.matchRegexCaseInsensitive(varString, regex);
+ } else {
+ value = StringUtil.matchRegex(varString, regex);
+ }
+ if (isMatch == null) {
+ calculationResultCache.registeRegex(regex, varString, value);
+ }
+ return value;
+ }
+
+ @SuppressWarnings({"rawtypes"})
+ protected boolean doPreProcess(RuleContext context, Expression expression, String varString) {
+
+ try {
+ String keyword = expression.getKeyword();
+ if (StringUtil.isEmpty(keyword) || StringUtil.isEmpty(varString)) {
+ return true;
+ }
+ boolean result = true;
+ if (varString.toLowerCase().indexOf(keyword) == -1) {
+ result = false;
+ }
+
+ return result;
+ } catch (Exception e) {
+ LOG.error("AbstractExpressionFunction doPreProcess error", e);
+ return true;
+ }
+
+ }
+
+ protected boolean caseInsensitive() {
+ return false;
+ }
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/ScriptFunction.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/ScriptFunction.java
new file mode 100644
index 0000000..e20ff2a
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/expression/ScriptFunction.java
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.function.expression;
+
+import com.alibaba.fastjson.JSONObject;
+
+import org.apache.rocketmq.streams.filter.builder.ExpressionBuilder;
+import org.apache.rocketmq.streams.filter.context.RuleContext;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+import org.apache.rocketmq.streams.filter.operator.expression.Expression;
+import org.apache.rocketmq.streams.filter.operator.expression.SimpleExpression;
+import org.apache.rocketmq.streams.script.ScriptComponent;
+import org.apache.rocketmq.streams.script.context.FunctionContext;
+import org.apache.rocketmq.streams.script.annotation.Function;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethod;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethodAilas;
+import org.apache.rocketmq.streams.common.context.Message;
+import org.apache.rocketmq.streams.script.utils.FunctionUtils;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+import org.apache.rocketmq.streams.script.parser.imp.FunctionParser;
+import org.apache.rocketmq.streams.script.service.IScriptExpression;
+
+import java.util.List;
+
+/**
+ * 可以把script脚本发布成规则引擎的表达式格式如下（"随便写会被忽略"，"operator"，"equals(fieldname,'3232')"）; 函数必须返回的是boolean值
+ */
+
+@Function
+public class ScriptFunction extends AbstractExpressionFunction {
+ ScriptComponent scriptComponent = ScriptComponent.getInstance();
+ public static final String SPLIT_SIGN = "######";//对参数进行分隔
+ public static final String QUOTATION_CONVERT = "^^^^";//单引号转换
+
+ @Override
+ @FunctionMethod("operator")
+ @FunctionMethodAilas("执行脚本")
+ public Boolean doExpressionFunction(Expression expression, RuleContext context, Rule rule) {
+
+ Object valueObject = expression.getValue();
+ String valueString = "";
+
+ valueString = String.valueOf(valueObject).trim();
+ if (StringUtil.isEmpty(valueString)) {
+ return false;
+ }
+
+ String script = createScript(valueString, context, expression);
+ List<IScriptExpression> expressions = FunctionParser.getInstance().parse(script);
+ if (expressions == null || expressions.size() == 0) {
+ throw new RuntimeException("execute operator function error，parse function error " + valueString);
+ }
+ IScriptExpression scriptExpression = expressions.get(0);
+ FunctionContext functionContext = new FunctionContext(context.getMessage());
+ context.syncSubContext(functionContext);
+ Object object = scriptExpression.executeExpression(context.getMessage(), functionContext);
+ if (object == null) {
+ throw new RuntimeException("execute scriptFunction error, expect return boolean value ,real is null");
+ }
+ if (Boolean.class.isInstance(object)) {
+ return (Boolean)object;
+ }
+ throw new RuntimeException("execute scriptFunction error, expect return boolean value ,real is " + object.getClass().getName());
+ }
+
+ /**
+ * 解析函数中的参数，如果参数是表达式，先进行表达式解析，得到参数值
+ *
+ * @param valueString
+ * @param context
+ * @param expression
+ * @return
+ */
+ private String createScript(String valueString, RuleContext context, Expression expression) {
+ valueString = FunctionUtils.getConstant(valueString);
+ String functionStr = valueString;
+
+ if (valueString.endsWith(";")) {
+ functionStr = functionStr.substring(0, functionStr.length() - 1);
+ }
+ int index = functionStr.indexOf("(");
+ String functionName = functionStr.substring(0, index);
+ functionStr = functionStr.substring(index + 1, valueString.length() - 1);
+ String[] values = functionStr.split(SPLIT_SIGN);
+ StringBuilder stringBuilder = new StringBuilder();
+ stringBuilder.append(functionName + "(");
+ boolean isFirst = true;
+ for (String value : values) {
+ if (isFirst) {
+ isFirst = false;
+ } else {
+ stringBuilder.append(",");
+ }
+ if (isExpression(value)) {
+ boolean paramter = ExpressionBuilder.executeExecute(expression.getNameSpace(), value, context.getMessage().getMessageBody());
+ stringBuilder.append(paramter);
+ } else {
+ stringBuilder.append(value);
+ }
+ }
+ stringBuilder.append(");");
+ return stringBuilder.toString();
+ }
+
+ protected boolean isExpression(String paramter) {
+ if (StringUtil.isEmpty(paramter)) {
+ return false;
+ }
+ paramter = paramter.trim();
+ if (paramter.startsWith("(") && paramter.endsWith(")")) {
+ return true;
+ }
+ return false;
+ }
+
+ public static void main(String[] args) {
+ ScriptFunction scriptFunction = new ScriptFunction();
+ Expression expression = new SimpleExpression("inner_message", "operator", "'~((___lower_proc_path_1,regex,\'"
+ + "(rdesktop|filebeat|\\/tmp\\/go-build|headless_shell|dongxingrpc|流量|scheduler|check|gpg-agent"
+ + "|/fireball|magneticod|portainer|kube|/vfs|jingling|/netstat|busybox|/tomcat|server|ping|bitcoind"
+ + "|/desktop|docker|aliprobe|safe|update|service|query|svchost\\"
+ + ".exe|mozilla|firefox|ss-server|parity|aria2c|daemon|software|/perl|/redis|/gnome|/bin/ssh|msinfo\\"
+ + ".exe|chrome\\.exe|/bin/ping|/ali|/usr/local/cloudmonitor/|sock|w3wp\\"
+ + ".exe|/usr/local/aegis|/phantomjs|filenet|tunnel|probe)\'))'");
+
+ Message message = new Message(new JSONObject());
+ Rule rule = new Rule();
+ rule.setNameSpace("3r3");
+ scriptFunction.doExpressionFunction(expression, new RuleContext(new JSONObject(), rule), rule);
+ }
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/script/CaseFunction.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/script/CaseFunction.java
new file mode 100644
index 0000000..c63b724
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/function/script/CaseFunction.java
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.function.script;
+
+import java.util.List;
+
+import com.alibaba.fastjson.JSONObject;
+
+import org.apache.rocketmq.streams.filter.builder.ExpressionBuilder;
+import org.apache.rocketmq.streams.script.ScriptComponent;
+import org.apache.rocketmq.streams.script.context.FunctionContext;
+import org.apache.rocketmq.streams.script.annotation.Function;
+import org.apache.rocketmq.streams.script.annotation.FunctionMethod;
+import org.apache.rocketmq.streams.common.context.IMessage;
+import org.apache.rocketmq.streams.script.utils.FunctionUtils;
+
+@Function
+public class CaseFunction {
+
+ /**
+ * if(((((___compare_1&___compare_2)&___compare_3)&___compare_4)&___in_1)){___case_1='可疑编码命令';};
+ *
+ * @param message
+ * @param context
+ * @param value
+ * @return
+ */
+ @FunctionMethod(value = "if", alias = "case", comment = "支持内嵌函数")
+ public Boolean match(IMessage message, FunctionContext context, String value) {
+ String tmp = value;
+ value = FunctionUtils.getValueString(message, context, value);
+ if (value == null) {
+ value = tmp;
+ }
+ if (value.length() <= 5) {
+ String lowValue = value.trim().toLowerCase();
+ if (lowValue.equals("true") || lowValue.equals("false")) {
+ return Boolean.valueOf(value);
+ }
+ }
+ if (value.startsWith("(") && value.endsWith(")")) {
+ String expression = value;
+ Boolean result = ExpressionBuilder.executeExecute(System.currentTimeMillis() + "", expression,
+ message.getMessageBody());
+ //message.getHeader().getRegex2Value().put(value,result);
+ return result;
+ }
+ return false;
+ }
+
+ @FunctionMethod(value = "!", alias = "!if", comment = "支持内嵌函数")
+ public Boolean notma(IMessage message, FunctionContext context, String value) {
+ return !match(message, context, value);
+ }
+
+ //(___in_2&((___compare_5|__fdfddf)|___compare_7))
+ //(___in_2&((___compare_5|(a,==,b))|___compare_7))
+ public static void main(String[] args) {
+ String scriptValue
+ = "if((___in_2&((___compare_5|___compare_6)|___compare_7))){___case_1=___cast_java应用执行可疑指令_1;};";
+ ScriptComponent scriptComponent = ScriptComponent.getInstance();
+ JSONObject msg = new JSONObject();
+ msg.put("___in_2", true);
+ msg.put("___compare_5", true);
+ msg.put("___compare_6", true);
+ msg.put("___compare_7", false);
+ msg.put("___cast_java应用执行可疑指令_1", "fdsfdsdsff");
+ List<IMessage> messages = scriptComponent.getService().executeScript(msg, scriptValue);
+ System.out.println(messages.get(0).getMessageBody().toJSONString());
+ }
+
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/monitor/Monitor.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/monitor/Monitor.java
new file mode 100644
index 0000000..25392e9
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/monitor/Monitor.java
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.monitor;
+
+import com.alibaba.fastjson.JSONArray;
+import com.alibaba.fastjson.JSONObject;
+import org.apache.rocketmq.streams.filter.exception.RegexTimeoutException;
+
+import java.util.ArrayList;
+import java.util.List;
+
+@SuppressWarnings("rawtypes")
+public abstract class Monitor<T> {
+
+ protected long startTime;
+ protected long endTime;
+ protected String message;
+ protected T result;
+ protected boolean isException;
+ protected Exception e;
+
+ protected List<Monitor> children = new ArrayList<>();
+
+ public long costTime() {
+ // endTime=0时，Action还没结束，故不统计
+ if (startTime <= 0 || endTime <= 0) {
+ return 0;
+ }
+ return endTime - startTime;
+ }
+
+ public void begin(String message) {
+ this.message = message;
+ startTime = System.currentTimeMillis();
+ }
+
+ public void end() {
+ endTime = System.currentTimeMillis();
+ }
+
+ public void end(T result) {
+ end();
+ if (result != null) {
+ this.result = result;
+ }
+ }
+
+ public void endException(Exception e) {
+ end();
+ this.e = e;
+ this.isException = true;
+ }
+
+ public abstract String getType();
+
+ public void addChildren(Monitor monitor) {
+ children.add(monitor);
+ }
+
+ @Override
+ public String toString() {
+ JSONObject jsonObject = toJson();
+ if (jsonObject == null) {
+ return "";
+ }
+ return jsonObject.toJSONString();
+ }
+
+ public JSONObject toJson() {
+ if (message == null) {
+ return null;
+ }
+ JSONObject jsonObject = new JSONObject();
+ jsonObject.put("message", message);
+ jsonObject.put("result", resultToString());
+ jsonObject.put("cost_time", costTime());
+ jsonObject.put("type", getType());
+ if (children != null) {
+ JSONArray jsonArray = new JSONArray();
+ for (Monitor monitor : children) {
+ jsonArray.add(monitor.toString());
+ }
+ jsonObject.put("children", jsonArray);
+ }
+
+ return jsonObject;
+ }
+
+ protected String resultToString() {
+ if (result == null) {
+ return "error";
+ }
+ return result.toString();
+ }
+
+ public long getStartTime() {
+ return startTime;
+ }
+
+ public void setStartTime(long startTime) {
+ this.startTime = startTime;
+ }
+
+ public long getEndTime() {
+ return endTime;
+ }
+
+ public void setEndTime(long endTime) {
+ this.endTime = endTime;
+ }
+
+ public String getMessage() {
+ return message;
+ }
+
+ public List<Monitor> getChildren() {
+ return children;
+ }
+
+ public boolean isRegexTimeout() {
+ if (!isException) {
+ return false;
+ }
+ if (RegexTimeoutException.class.isInstance(e)) {
+ return true;
+ }
+ return false;
+ }
+
+ public Exception getException() {
+ return e;
+ }
+
+ public boolean isException() {
+ return isException;
+ }
+
+ public T getResult() {
+ return result;
+ }
+
+ public void setResult(T result) {
+ this.result = result;
+ }
+
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/monitor/contants/MonitorType.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/monitor/contants/MonitorType.java
new file mode 100644
index 0000000..8228d9a
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/monitor/contants/MonitorType.java
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.monitor.contants;
+
+import java.util.ArrayList;
+import java.util.List;
+
+public enum MonitorType {
+ MESSAGE,
+ RULE,
+ ACTION,
+ VAR,
+ EXPRESSION,
+ NULL;
+
+ public static List<MonitorType> getAllMonitorTypes() {
+ List<MonitorType> monitorTypes = new ArrayList<>();
+ monitorTypes.add(MESSAGE);
+ monitorTypes.add(RULE);
+ monitorTypes.add(ACTION);
+ monitorTypes.add(VAR);
+ monitorTypes.add(EXPRESSION);
+ monitorTypes.add(NULL);
+ return monitorTypes;
+ }
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/monitor/rule/ActionMonitor.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/monitor/rule/ActionMonitor.java
new file mode 100644
index 0000000..91578b9
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/monitor/rule/ActionMonitor.java
@@ -0,0 +1,27 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.monitor.rule;
+
+import org.apache.rocketmq.streams.filter.monitor.Monitor;
+import org.apache.rocketmq.streams.filter.monitor.contants.MonitorType;
+
+public class ActionMonitor extends Monitor<Object> {
+ @Override
+ public String getType() {
+ return MonitorType.ACTION.name();
+ }
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/monitor/rule/ExpressionMonitor.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/monitor/rule/ExpressionMonitor.java
new file mode 100644
index 0000000..652a6ce
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/monitor/rule/ExpressionMonitor.java
@@ -0,0 +1,27 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.monitor.rule;
+
+import org.apache.rocketmq.streams.filter.monitor.Monitor;
+import org.apache.rocketmq.streams.filter.monitor.contants.MonitorType;
+
+public class ExpressionMonitor extends Monitor<Boolean> {
+ @Override
+ public String getType() {
+ return MonitorType.EXPRESSION.name();
+ }
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/monitor/rule/MessageMonitor.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/monitor/rule/MessageMonitor.java
new file mode 100644
index 0000000..4d20a92
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/monitor/rule/MessageMonitor.java
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.monitor.rule;
+
+import org.apache.rocketmq.streams.filter.monitor.Monitor;
+import org.apache.rocketmq.streams.filter.monitor.contants.MonitorType;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+@SuppressWarnings("rawtypes")
+public class MessageMonitor extends Monitor<List<Rule>> {
+ public static final String KEY = MessageMonitor.class.getName();
+ private static final NullMonitor NULL_MONITOR = new NullMonitor();
+ private Map<String, Monitor> ruleMonitors = new HashMap<String, Monitor>();
+ private volatile boolean monitorSwtich = true;
+
+ public MessageMonitor(boolean monitorSwtich) {
+ this.monitorSwtich = monitorSwtich;
+ }
+
+ public MessageMonitor() {
+ }
+
+ public Monitor createRuleMonitor(String ruleName) {
+ if (!monitorSwtich) {
+ return NULL_MONITOR;
+ }
+ RuleMonitor ruleMonitor = new RuleMonitor();
+ ruleMonitor.begin(ruleName);
+ this.addChildren(ruleMonitor);
+ return ruleMonitor;
+ }
+
+ public Monitor createExpressionMonitor(String expressionName, String ruleName) {
+ if (!monitorSwtich) {
+ return NULL_MONITOR;
+ }
+ Monitor ruleMonitor = getRuleMonitor(ruleName);
+ if (ruleMonitor == null) {
+ return null;
+ }
+ ExpressionMonitor expressionMonitor = new ExpressionMonitor();
+ expressionMonitor.begin(expressionName);
+ ruleMonitor.addChildren(expressionMonitor);
+ return expressionMonitor;
+ }
+
+ public Monitor createActionMonitor(String actionName, String ruleName) {
+ return NULL_MONITOR;
+ }
+
+ @Override
+ public void addChildren(Monitor monitor) {
+ if (!monitorSwtich) {
+ return;
+ }
+ super.addChildren(monitor);
+ ruleMonitors.put(monitor.getMessage(), monitor);
+ }
+
+ public Monitor getRuleMonitor(String ruleName) {
+ if (!monitorSwtich) {
+ return NULL_MONITOR;
+ }
+ Monitor monitor = ruleMonitors.get(ruleName);
+ if (monitor == null) {
+ return null;
+ }
+ return (RuleMonitor)monitor;
+ }
+
+ @Override
+ protected String resultToString() {
+ if (result == null) {
+ return "";
+ }
+ StringBuilder sb = new StringBuilder();
+ boolean isStart = false;
+ for (Rule rule : result) {
+ if (isStart) {
+ isStart = false;
+ } else {
+ sb.append(",");
+ }
+ sb.append(rule.getConfigureName());
+ }
+ return sb.toString();
+ }
+
+ @Override
+ public String getType() {
+ return MonitorType.MESSAGE.name();
+ }
+
+ public boolean isMonitorSwtich() {
+ return monitorSwtich;
+ }
+
+ public void setMonitorSwtich(boolean monitorSwtich) {
+ this.monitorSwtich = monitorSwtich;
+ }
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/monitor/rule/NullMonitor.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/monitor/rule/NullMonitor.java
new file mode 100644
index 0000000..fcaa652
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/monitor/rule/NullMonitor.java
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.monitor.rule;
+
+import com.alibaba.fastjson.JSONObject;
+import org.apache.rocketmq.streams.filter.monitor.Monitor;
+import org.apache.rocketmq.streams.filter.monitor.contants.MonitorType;
+
+@SuppressWarnings("rawtypes")
+public class NullMonitor extends Monitor {
+ @Override
+ public void begin(String message) {
+ }
+
+ @Override
+ public void end() {
+ }
+
+ @Override
+ public void end(Object result) {
+ }
+
+ @Override
+ public void addChildren(Monitor monitor) {
+ }
+
+ @Override
+ public String toString() {
+ return null;
+ }
+
+ @Override
+ public JSONObject toJson() {
+ return null;
+ }
+
+ @Override
+ public String getType() {
+ return MonitorType.NULL.name();
+ }
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/monitor/rule/RuleMonitor.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/monitor/rule/RuleMonitor.java
new file mode 100644
index 0000000..450fb99
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/monitor/rule/RuleMonitor.java
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.monitor.rule;
+
+import com.alibaba.fastjson.JSONObject;
+import org.apache.rocketmq.streams.filter.monitor.Monitor;
+import org.apache.rocketmq.streams.filter.monitor.contants.MonitorType;
+
+import java.util.*;
+
+@SuppressWarnings("rawtypes")
+public class RuleMonitor extends Monitor<Boolean> {
+
+ private Map<String, List<Monitor>> groupByMonitorClassName = new HashMap<>();
+
+ @Override
+ public void addChildren(Monitor monitor) {
+ super.addChildren(monitor);
+ List<Monitor> monitors = groupByMonitorClassName.get(monitor.getClass().getName());
+ if (monitors == null) {
+ monitors = new ArrayList<>();
+ groupByMonitorClassName.put(monitor.getClass().getName(), monitors);
+ }
+ monitors.add(monitor);
+ }
+
+ public List<Monitor> getExpressionMonitors() {
+ return groupByMonitorClassName.get(ExpressionMonitor.class.getName());
+ }
+
+ public List<Monitor> getActionMonitors() {
+ return groupByMonitorClassName.get(ActionMonitor.class.getName());
+ }
+
+ public List<Monitor> getVarMonitors() {
+ return groupByMonitorClassName.get(VarMonitor.class.getName());
+ }
+
+ @Override
+ public JSONObject toJson() {
+ return super.toJson();
+ }
+
+ @Override
+ public String getType() {
+ return MonitorType.RULE.name();
+ }
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/monitor/rule/VarMonitor.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/monitor/rule/VarMonitor.java
new file mode 100644
index 0000000..5156041
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/monitor/rule/VarMonitor.java
@@ -0,0 +1,27 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.monitor.rule;
+
+import org.apache.rocketmq.streams.filter.monitor.Monitor;
+import org.apache.rocketmq.streams.filter.monitor.contants.MonitorType;
+
+public class VarMonitor extends Monitor<Object> {
+ @Override
+ public String getType() {
+ return MonitorType.VAR.name();
+ }
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/FilterOperator.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/FilterOperator.java
new file mode 100644
index 0000000..3f8d3e9
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/FilterOperator.java
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.operator;
+
+public class FilterOperator extends Rule {
+
+ public FilterOperator(String namespace, String name, String expression) {
+ super(namespace, name, expression);
+ }
+
+ public FilterOperator() {}
+
+ public FilterOperator(String expression) {
+ super(expression);
+ }
+
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/Rule.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/Rule.java
new file mode 100644
index 0000000..4fb2311
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/Rule.java
@@ -0,0 +1,538 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.operator;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+
+import com.alibaba.fastjson.JSONArray;
+import com.alibaba.fastjson.JSONObject;
+
+import org.apache.rocketmq.streams.common.channel.sink.ISink;
+import org.apache.rocketmq.streams.filter.operator.action.Action;
+import org.apache.rocketmq.streams.filter.operator.action.impl.ChannelAction;
+import org.apache.rocketmq.streams.filter.optimization.ExpressionOptimization;
+import org.apache.rocketmq.streams.filter.context.RuleMessage;
+import org.apache.rocketmq.streams.filter.operator.expression.Expression;
+import org.apache.rocketmq.streams.filter.operator.expression.GroupExpression;
+import org.apache.rocketmq.streams.filter.operator.expression.GroupExpressionManager;
+import org.apache.rocketmq.streams.filter.operator.expression.RelationExpression;
+import org.apache.rocketmq.streams.filter.operator.var.ContextVar;
+import org.apache.rocketmq.streams.filter.operator.var.InnerVar;
+import org.apache.rocketmq.streams.filter.operator.var.Var;
+import org.apache.rocketmq.streams.filter.FilterComponent;
+import org.apache.rocketmq.streams.common.configurable.IConfigurable;
+import org.apache.rocketmq.streams.common.configurable.IAfterConfiguableRefreshListerner;
+import org.apache.rocketmq.streams.common.configurable.IConfigurableService;
+import org.apache.rocketmq.streams.common.context.AbstractContext;
+import org.apache.rocketmq.streams.common.context.IMessage;
+import org.apache.rocketmq.streams.common.topology.ChainStage;
+import org.apache.rocketmq.streams.common.topology.stages.FilterChainStage;
+import org.apache.rocketmq.streams.common.topology.builder.IStageBuilder;
+import org.apache.rocketmq.streams.common.topology.builder.PipelineBuilder;
+import org.apache.rocketmq.streams.common.metadata.MetaData;
+import org.apache.rocketmq.streams.common.metadata.MetaDataField;
+import org.apache.rocketmq.streams.common.topology.model.AbstractRule;
+import org.apache.rocketmq.streams.db.driver.JDBCDriver;
+
+public class Rule extends AbstractRule implements IAfterConfiguableRefreshListerner,
+ IStageBuilder<ChainStage> {
+ private transient volatile Map<String, Var> varMap = new HashMap<>();
+ private transient volatile Map<String, Expression> expressionMap = new HashMap<>();
+ @Deprecated
+ private transient Map<String, Action> actionMap = new HashMap<>();
+ private transient Map<String, MetaData> metaDataMap = new HashMap<>();
+ private transient volatile Map<String, JDBCDriver> dataSourceMap = new HashMap<>();
+ private String expressionStr;//表达式
+
+ protected transient GroupExpressionManager groupExpressionManager;
+ /**
+ * 如果已经完成varmap和expressionmap的初始化,主要是用于兼容老版本规则数据，新规则可以忽略这个字段，值设置为true
+ */
+ private transient boolean isFinishVarAndExpression = false;
+
+ public Rule() {groupExpressionManager = new GroupExpressionManager(this);}
+
+ public Rule(String namespace, String name, String expression) {
+ this();
+ FilterComponent filterComponent = FilterComponent.getInstance();
+ Rule rule = filterComponent.createRule(namespace, name, expression);
+ this.varMap = rule.getVarMap();
+ this.actionMap = rule.getActionMap();
+ this.expressionMap = rule.getExpressionMap();
+ this.metaDataMap = rule.getMetaDataMap();
+ this.dataSourceMap = rule.getDataSourceMap();
+ this.setMsgMetaDataName(rule.getMsgMetaDataName());
+ this.setExpressionStr(rule.getExpressionStr());
+ this.setVarNames(rule.getVarNames());
+ this.setExpressionName(rule.getExpressionName());
+ }
+
+ public Rule(String expression) {
+ this(null, null, expression);
+ }
+
+ public Rule copy() {
+ Rule rule = new Rule();
+ rule.setNameSpace(getNameSpace());
+ rule.setType(getType());
+ rule.setConfigureName(getConfigureName());
+ rule.varMap = varMap;
+ rule.expressionMap = expressionMap;
+ rule.actionMap = actionMap;
+ rule.metaDataMap = metaDataMap;
+ rule.dataSourceMap = dataSourceMap;
+ rule.setActionNames(actionNames);
+ rule.setVarNames(varNames);
+ rule.setExpressionName(expressionName);
+ rule.setMsgMetaDataName(getMsgMetaDataName());
+ rule.setRuleCode(ruleCode);
+ rule.setRuleDesc(ruleDesc);
+ rule.setRuleStatus(ruleStatus);
+ rule.setRuleTitle(ruleTitle);
+ rule.setRuleStatus(ruleStatus);
+ rule.setConfigurableService(configurableService);
+ rule.setPrivateDatas(privateDatas);
+ return rule;
+ }
+
+ @Override
+ public void doProcessAfterRefreshConfigurable(IConfigurableService configurableService) {
+
+ if (isFinishVarAndExpression == false) {
+ this.dataSourceMap = configurableService.queryConfigurableMapByType(ISink.TYPE);
+
+ }
+ initExpression(configurableService);
+ initVar(configurableService);
+ initAction(configurableService);
+ initMetaData(configurableService);
+ //this.optimize();
+ // groupExpressionManager.compile();
+ }
+
+ public void addAction(ChannelAction action) {
+ actionMap.put(action.getConfigureName(), action);
+ this.getActionNames().add(action.getConfigureName());
+ }
+
+ private void initVar(IConfigurableService configurableService) {
+ Map<String, Var> varMap = new HashMap<>();
+ if (isFinishVarAndExpression == false) {
+ varMap = configurableService.queryConfigurableMapByType(Var.TYPE);
+ }
+ if (expressionMap != null) {
+ Iterator<Entry<String, Expression>> it = expressionMap.entrySet().iterator();
+ while (it.hasNext()) {
+ Entry<String, Expression> entry = it.next();
+ Expression expression = entry.getValue();
+ if (RelationExpression.class.isInstance(expression)) {
+ continue;
+ }
+ String varName = expression.getVarName();
+ Var var = varMap.get(varName);
+ if (var == null) {
+ ContextVar contextVar = new ContextVar();
+ contextVar.setNameSpace(expression.getNameSpace());
+ contextVar.setConfigureName(varName);
+ contextVar.setVarName(varName);
+ contextVar.setFieldName(varName);
+ varMap.put(varName, contextVar);
+ var = contextVar;
+ }
+ this.varMap.put(varName, var);
+ }
+ }
+ InnerVar innerVar = new InnerVar();
+ innerVar.setNameSpace(getNameSpace());
+ innerVar.setConfigureName(innerVar.getClass().getSimpleName());
+ innerVar.setVarName(InnerVar.ORIG_MESSAGE);
+ this.varMap.put(innerVar.getVarName(), innerVar);
+ }
+
+ @Override
+ public Set<String> getDependentFields() {
+ Expression expression = expressionMap.get(getExpressionName());
+ return expression.getDependentFields(expressionMap);
+ }
+
+ protected void initExpression(IConfigurableService configurableService) {
+ Map<String, Expression> expressionMap = new HashMap<>();
+ if (isFinishVarAndExpression == false) {
+ expressionMap = configurableService.queryConfigurableMapByType(Expression.TYPE);
+ }
+ String expressionName = getExpressionName();
+ fetchExpression(expressionName, expressionMap);
+ }
+
+ /**
+ * 递归获取所有依赖的expression
+ *
+ * @param expressionName
+ * @param expressionMap
+ */
+ protected void fetchExpression(String expressionName, Map<String, Expression> expressionMap) {
+ Expression expression = expressionMap.get(expressionName);
+ if (expression == null) {
+ return;
+ }
+ this.expressionMap.put(expression.getConfigureName(), expression);
+ if (!RelationExpression.class.isInstance(expression)) {
+ return;
+ }
+ RelationExpression relationExpression = (RelationExpression)expression;
+ List<String> expressionNames = relationExpression.getValue();
+ if (expressionNames != null) {
+ for (String name : expressionNames) {
+ fetchExpression(name, expressionMap);
+ }
+ }
+ }
+
+ protected void initMetaData(IConfigurableService configurableService) {
+ Map<String, MetaData> metaDataMap = new HashMap<>();
+ this.metaDataMap = configurableService.queryConfigurableMapByType(MetaData.TYPE);
+ // MetaData metaData=metaDataMap.get(getMsgMetaDataName());
+ // if(metaData!=null){
+ // this.metaDataMap.put(getMsgMetaDataName(),metaData);
+ // }
+ }
+
+ /**
+ * 赋值action给rule
+ *
+ * @param configurableService
+ */
+ protected void initAction(IConfigurableService configurableService) {
+ Map<String, Action> actionMap = new HashMap<>();
+ actionMap = configurableService.queryConfigurableMapByType(Action.TYPE);
+ if (actionNames != null) {
+ for (String actionName : actionNames) {
+ Action action = actionMap.get(actionName);
+ if (action != null) {
+ this.actionMap.put(actionName, action);
+ }
+ }
+ }
+ }
+
+ public void putConfigurableMap(IConfigurable configurable, String type) {
+ if (Var.TYPE.equals(type)) {
+ varMap.put(configurable.getConfigureName(), (Var)configurable);
+ } else if (Expression.TYPE.equals(type)) {
+ expressionMap.put(configurable.getConfigureName(), (Expression)configurable);
+ } else if (Action.TYPE.equals(type)) {
+ actionMap.put(configurable.getConfigureName(), (Action)configurable);
+ } else if (MetaData.TYPE.equals(type)) {
+ metaDataMap.put(configurable.getConfigureName(), (MetaData)configurable);
+ } else if (ISink.TYPE.equals(type)) {
+ dataSourceMap.put(configurable.getConfigureName(), (JDBCDriver)configurable);
+ }
+ }
+
+ public void insertElement(IConfigurableService ruleEngineConfigurableService) {
+ insertOrUpdate(ruleEngineConfigurableService, varMap.values());
+ insertOrUpdate(ruleEngineConfigurableService, expressionMap.values());
+ insertOrUpdate(ruleEngineConfigurableService, actionMap.values());
+ insertOrUpdate(ruleEngineConfigurableService, metaDataMap.values());
+ insertOrUpdate(ruleEngineConfigurableService, dataSourceMap.values());
+ }
+
+ private <T extends IConfigurable> void insertOrUpdate(IConfigurableService ruleEngineConfigurableService,
+ Collection<T> configurables) {
+ if (configurables == null) {
+ return;
+ }
+ for (IConfigurable configurable : configurables) {
+ ruleEngineConfigurableService.insert(configurable);
+ }
+ }
+
+ public List<IConfigurable> getDependConfigurables() {
+ List<IConfigurable> configurableList = new ArrayList<>();
+ if (varMap != null) {
+ configurableList.addAll(varMap.values());
+ }
+ if (expressionMap != null) {
+ configurableList.addAll(expressionMap.values());
+ }
+ if (actionMap != null) {
+ configurableList.addAll(actionMap.values());
+ }
+ if (metaDataMap != null) {
+ configurableList.addAll(metaDataMap.values());
+ }
+ if (dataSourceMap != null) {
+ configurableList.addAll(dataSourceMap.values());
+ }
+ return configurableList;
+ }
+
+ public Map<String, Var> getVarMap() {
+ return varMap;
+ }
+
+ public void setVarMap(Map<String, Var> varMap) {
+ this.varMap = varMap;
+ }
+
+ public Map<String, Expression> getExpressionMap() {
+ return expressionMap;
+ }
+
+ public void setExpressionMap(Map<String, Expression> expressionMap) {
+ this.expressionMap = expressionMap;
+ }
+
+ public Map<String, Action> getActionMap() {
+ return actionMap;
+ }
+
+ public void setActionMap(Map<String, Action> actionMap) {
+ this.actionMap = actionMap;
+ }
+
+ public Map<String, MetaData> getMetaDataMap() {
+ return metaDataMap;
+ }
+
+ public void setMetaDataMap(Map<String, MetaData> metaDataMap) {
+ this.metaDataMap = metaDataMap;
+ }
+
+ public Map<String, JDBCDriver> getDataSourceMap() {
+ return dataSourceMap;
+ }
+
+ public void setDataSourceMap(Map<String, JDBCDriver> dataSourceMap) {
+ this.dataSourceMap = dataSourceMap;
+ }
+
+ private transient static FilterComponent filterComponent = FilterComponent.getInstance();
+
+ @Override
+ public Boolean doMessage(IMessage message, AbstractContext context) {
+ return execute(message.getMessageBody());
+ }
+
+ public static final String FIRE_RULES = "fireRules";
+
+ public boolean execute(JSONObject msg) {
+ RuleMessage ruleMessage = new RuleMessage(msg);
+ List<Rule> fireRules = filterComponent.excuteRule(ruleMessage, this);
+ msg.put(FIRE_RULES, createJsonArray(fireRules));
+ if (fireRules == null || fireRules.size() == 0) {
+ return false;
+ }
+ return true;
+ }
+
+ private JSONArray createJsonArray(List<Rule> fireRules) {
+ JSONArray jsonArray = new JSONArray();
+ if (fireRules == null) {
+ return jsonArray;
+ }
+ for (Rule rule : fireRules) {
+ jsonArray.add(rule.getNameSpace() + ":" + rule.getConfigureName());
+ }
+ return jsonArray;
+ }
+
+ public String toExpressionString() {
+ Expression rootExpression = expressionMap.get(expressionName);
+ return rootExpression.toExpressionString(expressionMap);
+ }
+
+ public String toMetaDataString() {
+ MetaData metaData = metaDataMap.get(getMsgMetaDataName());
+ if (metaData == null) {
+ return null;
+ }
+ List<MetaDataField> metaDataFields = metaData.getMetaDataFields();
+ StringBuilder sb = new StringBuilder();
+ boolean isFirst = true;
+ for (MetaDataField metaDataField : metaDataFields) {
+ if (metaDataField.getDataType().matchClass(String.class)) {
+ continue;
+ }
+ if (isFirst) {
+ isFirst = false;
+ } else {
+ sb.append(",");
+ }
+ String line = metaDataField.getFieldName() + ";" + metaDataField.getDataType().getDataTypeName();
+ sb.append(line);
+ }
+ return sb.toString();
+ }
+
+ public boolean isFinishVarAndExpression() {
+ return isFinishVarAndExpression;
+ }
+
+ public void setFinishVarAndExpression(boolean finishVarAndExpression) {
+ isFinishVarAndExpression = finishVarAndExpression;
+ }
+
+ public String getExpressionStr() {
+ return expressionStr;
+ }
+
+ public void setExpressionStr(String expressionStr) {
+ this.expressionStr = expressionStr;
+ }
+
+ @Override
+ public ChainStage createStageChain(PipelineBuilder pipelineBuilder) {
+ FilterChainStage filterChainStage = new FilterChainStage();
+ pipelineBuilder.addConfigurables(this);
+ filterChainStage.setRule(this);
+ filterChainStage.setEntityName("filter");
+ filterChainStage.setLabel(this.getConfigureName());
+ return filterChainStage;
+ }
+
+ @Override
+ public void addConfigurables(PipelineBuilder pipelineBuilder) {
+ if (varMap.values() != null) {
+ pipelineBuilder.addConfigurables(varMap.values());
+ }
+ if (expressionMap.values() != null) {
+ pipelineBuilder.addConfigurables(expressionMap.values());
+ }
+ if (actionMap.values() != null) {
+ pipelineBuilder.addConfigurables(actionMap.values());
+ }
+ if (metaDataMap.values() != null) {
+ pipelineBuilder.addConfigurables(metaDataMap.values());
+ }
+ if (dataSourceMap.values() != null) {
+ pipelineBuilder.addConfigurables(dataSourceMap.values());
+ }
+ }
+
+ /**
+ * 做优化，把相同变量的表达式，用hyperscan执行
+ */
+ public void optimize() {
+ Expression root = createOptimizationRule();
+ if (!RelationExpression.class.isInstance(root)) {
+ return;
+ }
+ groupByChildrenExpression((RelationExpression)root);
+ }
+
+ /**
+ * 在同一层关系中，相同变量名的分到一组，统一做处理
+ *
+ * @param root
+ */
+ protected void groupByChildrenExpression(RelationExpression root) {
+ List<String> expressionNames = root.getValue();
+ if (expressionNames == null || expressionNames.size() == 0) {
+ return;
+ }
+ Map<String, GroupExpression> varName2ExpressionNames = new HashMap<>();
+ List<String> newExpressionNames = new ArrayList<>();
+ for (String name : expressionNames) {
+ Expression expression = expressionMap.get(name);
+ //是直接值，不需要再次计算
+ if (expression == null) {
+ newExpressionNames.add(name);
+ continue;
+ }
+
+ //递归分组
+ if (RelationExpression.class.isInstance(expression)) {
+ newExpressionNames.add(name);
+ RelationExpression relationExpression = (RelationExpression)expression;
+ groupByChildrenExpression(relationExpression);
+ continue;
+ }
+ //按变量名分组
+ String varName = expression.getVarName();
+ GroupExpression groupExpression = varName2ExpressionNames.get(varName);
+ if (groupExpression == null) {
+ groupExpression = new GroupExpression(this, varName, root.getRelation().equalsIgnoreCase("or"));
+ varName2ExpressionNames.put(varName, groupExpression);
+
+ }
+ groupExpression.addExpressionName(expression);
+ }
+
+ //如果某个变量的个数>5，去除掉原来多个表达式，换成分组表达式，否则保持不变
+ Iterator<Entry<String, GroupExpression>> it = varName2ExpressionNames.entrySet().iterator();
+ while (it.hasNext()) {
+ Entry<String, GroupExpression> entry = it.next();
+ String varName = entry.getKey();
+ GroupExpression groupExpression = entry.getValue();
+ if (groupExpression.size() < 2) {
+ newExpressionNames.addAll(groupExpression.getAllExpressionNames());
+ } else {
+ groupExpressionManager.addGroupExpression(groupExpression);
+ expressionMap.put(groupExpression.getConfigureName(), groupExpression);
+ newExpressionNames.add(groupExpression.getConfigureName());
+ }
+ }
+ root.setValue(newExpressionNames);
+ }
+
+ /**
+ * 默认规则的解析是多层嵌套，优化的方案是尽量展开
+ */
+ protected Expression createOptimizationRule() {
+ Expression root = expressionMap.get(expressionName);
+ if (!RelationExpression.class.isInstance(root)) {
+ return root;
+ }
+ List<Expression> expressions = new ArrayList<>();
+ List<RelationExpression> relationExpressions = new ArrayList<>();
+ Iterator<Expression> it = expressionMap.values().iterator();
+ while (it.hasNext()) {
+ Expression expression = it.next();
+ if (RelationExpression.class.isInstance(expression)) {
+ relationExpressions.add((RelationExpression)expression);
+ } else {
+ expressions.add(expression);
+ }
+ }
+ ExpressionOptimization expressionOptimization = new ExpressionOptimization(root, expressions, relationExpressions);
+ List<Expression> list = expressionOptimization.optimizate();
+ expressionMap.clear();
+ expressions.clear();
+ relationExpressions.clear();
+ for (Expression express : list) {
+ expressionMap.put(express.getConfigureName(), express);
+ if (RelationExpression.class.isInstance(express)) {
+ relationExpressions.add((RelationExpression)express);
+ } else {
+ expressions.add(express);
+ }
+ }
+ return root;
+ }
+
+ public GroupExpressionManager getGroupExpressionManager() {
+ return groupExpressionManager;
+ }
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/action/Action.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/action/Action.java
new file mode 100644
index 0000000..5233c44
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/action/Action.java
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.operator.action;
+
+import org.apache.rocketmq.streams.common.configurable.BasedConfigurable;
+import org.apache.rocketmq.streams.common.configurable.IConfigurable;
+
+public abstract class Action<T> extends BasedConfigurable implements IConfigurableAction<T>, IConfigurable {
+ public static final String TYPE = "action";
+
+ public Action() {
+ setType(TYPE);
+ }
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/action/IConfigurableAction.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/action/IConfigurableAction.java
new file mode 100644
index 0000000..ddf76cd
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/action/IConfigurableAction.java
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.operator.action;
+
+import org.apache.rocketmq.streams.filter.context.RuleContext;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+
+public interface IConfigurableAction<T> {
+
+ /**
+ * 根据规则上下文执行configurable组件的默认动作
+ *
+ * @param context 规则上下文，一条数据一个
+ * @param rule 当前正在执行的规则
+ * @return
+ */
+ T doAction(RuleContext context, Rule rule);
+
+ /**
+ * 验证configurable组件的输入参数是否有效
+ *
+ * @param context 规则上下文，一条数据一个
+ * @param rule 当前正在执行的规则
+ * @return
+ */
+ boolean volidate(RuleContext context, Rule rule);
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/action/impl/ChannelAction.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/action/impl/ChannelAction.java
new file mode 100644
index 0000000..0f29dea
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/action/impl/ChannelAction.java
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.operator.action.impl;
+
+import org.apache.rocketmq.streams.filter.context.RuleContext;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+import org.apache.rocketmq.streams.common.configurable.IAfterConfiguableRefreshListerner;
+import org.apache.rocketmq.streams.common.configurable.IConfigurableService;
+import org.apache.rocketmq.streams.common.channel.IChannel;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.rocketmq.streams.filter.operator.action.Action;
+
+public class ChannelAction extends Action<Boolean> implements IAfterConfiguableRefreshListerner {
+
+ private static final Log LOG = LogFactory.getLog(ChannelAction.class);
+ protected String channelName;
+ protected transient IChannel channel;
+
+ public ChannelAction() {
+ setType(Action.TYPE);
+ }
+
+ public ChannelAction setChannel(IChannel channel) {
+ this.channelName = channel.getConfigureName();
+ this.channel = channel;
+ return this;
+ }
+
+ @Override
+ public Boolean doAction(RuleContext context, Rule rule) {
+ channel.batchAdd(context.getMessage());
+ return true;
+ }
+
+ @Override
+ public boolean volidate(RuleContext context, Rule rule) {
+ return true;
+ }
+
+ @Override
+ public void doProcessAfterRefreshConfigurable(IConfigurableService configurableService) {
+ IChannel channel = configurableService.queryConfigurable(IChannel.TYPE, channelName);
+ channel.openAutoFlush();
+ this.channel = channel;
+ }
+
+ public String getChannelName() {
+ return channelName;
+ }
+
+ public void setChannelName(String channelName) {
+ this.channelName = channelName;
+ }
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/action/impl/MetaDataAction.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/action/impl/MetaDataAction.java
new file mode 100644
index 0000000..47620e2
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/action/impl/MetaDataAction.java
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.operator.action.impl;
+
+import com.alibaba.fastjson.JSONObject;
+import org.apache.rocketmq.streams.filter.context.RuleContext;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+import org.apache.rocketmq.streams.filter.operator.action.Action;
+import org.apache.rocketmq.streams.filter.operator.var.Var;
+import org.apache.rocketmq.streams.common.datatype.DataType;
+import org.apache.rocketmq.streams.common.metadata.MetaData;
+import org.apache.rocketmq.streams.common.metadata.MetaDataAdapter;
+import org.apache.rocketmq.streams.common.metadata.MetaDataField;
+import org.apache.rocketmq.streams.common.utils.DataTypeUtil;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+public class MetaDataAction extends Action<Boolean> {
+
+ private static final Log LOG = LogFactory.getLog(MetaDataAction.class);
+ protected String metaDataName;
+ protected Map<String, String> varName2FieldName;
+
+ @Override
+ public Boolean doAction(RuleContext context, Rule rule) {
+ if (!volidate(context, rule)) {
+ return null;
+ }
+ Map<String, Object> fieldName2Value = getQuery(context, rule);
+
+ // /**
+ // * 为开启killchain的规则设置killchainId
+ // */
+ // if (rule.getKillChainFlag() != null && rule.getKillChainFlag().intValue() == 1) {
+ // fieldName2Value.put("killchainId", System.currentTimeMillis() + Math.round(Math.random() * 9999));
+ // }
+ Boolean result = doAction(context, rule, fieldName2Value);
+ return result;
+ }
+
+ protected Boolean doAction(RuleContext context, Rule rule, Map<String, Object> fieldName2Value) {
+ MetaDataAdapter metaDataAdapter = context.getMetaDataAdapter(metaDataName);
+ if (metaDataAdapter == null) {
+ throw new RuntimeException("MetaDataAction doAction metaDataAdapter is null");
+ }
+ return metaDataAdapter.insert(fieldName2Value);
+ }
+
+ public String getMetaDataName() {
+ return metaDataName;
+ }
+
+ public void setMetaDataName(String metaDataName) {
+ this.metaDataName = metaDataName;
+ }
+
+ public Map<String, String> getVarName2FieldName() {
+ return varName2FieldName;
+ }
+
+ public void setVarName2FieldName(Map<String, String> varName2FieldName) {
+ this.varName2FieldName = varName2FieldName;
+ }
+
+ @Override
+ public boolean volidate(RuleContext context, Rule rule) {
+ if (StringUtil.isEmpty(metaDataName)) {
+ return false;
+ }
+ MetaDataAdapter adapter = context.getMetaDataAdapter(metaDataName);
+ if (adapter == null || adapter.getMetaData() == null) {
+ return false;
+ }
+ if (varName2FieldName == null || varName2FieldName.size() == 0) {
+ return false;
+ }
+ return true;
+ }
+
+ /**
+ * 查询数据
+ *
+ * @param context
+ * @param rule
+ * @return
+ */
+ public Map<String, Object> getQuery(RuleContext context, Rule rule) {
+ Map<String, Object> query = new HashMap<String, Object>();
+ MetaData metaData = context.getMetaData(metaDataName);
+ Iterator<Map.Entry<String, String>> it = varName2FieldName.entrySet().iterator();
+ while (it.hasNext()) {
+ try {
+ Map.Entry<String, String> entry = it.next();
+ String varName = entry.getKey();
+ String fieldName = entry.getValue();
+ Var var = context.getVar(rule.getConfigureName(), varName);
+ List<String> values = new ArrayList<>();
+ /**
+ * 需要MetaDataField配置最终输出的字段
+ */
+ try {
+ MetaDataField field = metaData.getMetaDataField(fieldName);
+ if (field == null) {
+ continue;
+ }
+ DataType dataType = field.getDataType();
+ if (query.containsKey(fieldName)) {
+ Object value = query.get(fieldName);
+ if (DataTypeUtil.isList(value.getClass())) {
+ values = (List)value;
+ } else {
+ query.put(fieldName, values);
+ values.add(dataType.toDataJson(value));
+ }
+ values.add(dataType.toDataJson(var.getVarValue(context, rule)));
+ } else {
+ query.put(fieldName, var.getVarValue(context, rule));
+ }
+ } catch (Exception e) {
+ LOG.error("MetaDataAction getQuery error,rule is :" + rule.getRuleCode() + " ,action is: "
+ + rule.getActionNames() + " ,metaDataName is: " + metaDataName + " ,varName is: " + varName
+ + " ,fieldName is: " + fieldName, e);
+ }
+
+ } catch (Exception e) {
+ LOG.error("MetaDataAction getQuery error,rule :" + rule.getRuleCode() + " ,action is:"
+ + rule.getActionNames(), e);
+ }
+
+ }
+ return query;
+ }
+
+ /**
+ * 把action的结果打印出来
+ *
+ * @param context
+ * @param rule
+ * @return
+ */
+ public String toString(RuleContext context, Rule rule) {
+ Map<String, Object> fieldName2Value = getQuery(context, rule);
+
+ JSONObject jsonObject = new JSONObject();
+ Iterator<Map.Entry<String, Object>> it = fieldName2Value.entrySet().iterator();
+ while (it.hasNext()) {
+ Map.Entry<String, Object> entry = it.next();
+ String value = String.valueOf(entry.getValue());
+ try {
+ if (value.contains(":")) {
+ jsonObject.put(entry.getKey(), JSONObject.parseObject(value));
+ } else {
+ jsonObject.put(entry.getKey(), entry.getValue());
+ }
+ } catch (Exception e) {
+ jsonObject.put(entry.getKey(), entry.getValue());
+ }
+
+ }
+ return jsonObject.toJSONString();
+ }
+
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/expression/Expression.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/expression/Expression.java
new file mode 100644
index 0000000..4a111c3
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/expression/Expression.java
@@ -0,0 +1,405 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.operator.expression;
+
+import java.io.Serializable;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.rocketmq.streams.common.configurable.BasedConfigurable;
+import org.apache.rocketmq.streams.common.configurable.IConfigurableService;
+import org.apache.rocketmq.streams.common.utils.AESUtil;
+import org.apache.rocketmq.streams.common.utils.ContantsUtil;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+import org.apache.rocketmq.streams.script.utils.FunctionUtils;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import com.alibaba.fastjson.JSONObject;
+import org.apache.rocketmq.streams.filter.operator.action.IConfigurableAction;
+import org.apache.rocketmq.streams.filter.context.RuleContext;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+import org.apache.rocketmq.streams.filter.function.expression.ExpressionFunction;
+import org.apache.rocketmq.streams.filter.operator.var.Var;
+import org.apache.rocketmq.streams.common.configurable.IConfigurable;
+import org.apache.rocketmq.streams.common.datatype.DataType;
+import org.apache.rocketmq.streams.common.datatype.ListDataType;
+import org.apache.rocketmq.streams.common.datatype.StringDataType;
+import org.apache.rocketmq.streams.common.metadata.MetaDataField;
+import org.apache.rocketmq.streams.common.utils.DataTypeUtil;
+
+public class Expression<T> extends BasedConfigurable
+ implements IConfigurable, IConfigurableAction<Boolean>, Serializable {
+ public static final String TYPE = "express";
+ private static final long serialVersionUID = 4610495074511059465L;
+ private static final Log LOG = LogFactory.getLog(Expression.class);
+ private static final String FIELD_COMPARE = "field";
+ private static final String AES_KEY = "baicheng.cbc";
+ private String varName;
+ private String functionName;
+ @SuppressWarnings("rawtypes")
+ private DataType dataType = new StringDataType(String.class);
+ protected T value;
+
+ public Expression copy() {
+ Expression expression = new Expression();
+ expression.setNameSpace(getNameSpace());
+ expression.setType(getType());
+ expression.setConfigureName(getConfigureName());
+ expression.setVarName(varName);
+ expression.setFunctionName(functionName);
+ expression.setDataType(dataType);
+ expression.setValue(value);
+ expression.setPrivateDatas(privateDatas);
+ expression.setAesFlag(aesFlag);
+ return expression;
+ }
+
+ // 前端处理使用
+ private String dataTypestr;
+
+ // 表达式中的关键词 正则之前进行字符串判断
+ private String keyword;
+
+ // 表达式的值是否加密 0未加密 1加密
+ private int aesFlag = 0;
+
+ /**
+ * 表达式是否是字段之间的比较，dataTypestr为field时为true
+ */
+ private boolean fieldFlag = false;
+ private volatile boolean initFlag = false;
+
+ public Expression() {
+ setType(TYPE);
+ }
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public Boolean doAction(RuleContext context, Rule rule) {
+ ExpressionFunction function = context.getExpressionFunction(functionName, this, context, rule);
+ if (function == null) {
+ return false;
+ }
+ /**
+ * 表达式新增fieldFlag 如：src_port=desc_port fieldFlag为true时，设置value为对应的字段值
+ */
+ if (fieldFlag && !initFlag) {
+ synchronized (Expression.class) {
+ if (fieldFlag && !initFlag) {
+ Var var = context.getVar(rule.getConfigureName(), String.valueOf(value));
+ if (var == null) {
+ LOG.error("Expression context.getVar var is null,expressName is: " + this.getConfigureName()
+ + " ,varName is: " + String.valueOf(value));
+ return false;
+ }
+ Object varObject = null;
+ varObject = var.getVarValue(context, rule);
+ if (varObject == null) {
+ LOG.error(
+ "Expression context.getVar varObject is null,expressName is: " + this.getConfigureName()
+ + " ,varName is: " + String.valueOf(value));
+ return false;
+ }
+ this.value = (T)String.valueOf(varObject).trim();
+ }
+ }
+ }
+ boolean result = false;
+ try {
+ result = function.doFunction(this, context, rule);
+ } catch (Exception e) {
+ LOG.error(
+ "Expression doAction function.doFunction error,rule is: " + rule.getConfigureName() + " ,express is:"
+ + getConfigureName(), e);
+ }
+ return result;
+ }
+
+ public Boolean getExpressionValue(RuleContext context, Rule rule) {
+ Boolean result = context.getExpressionValue(getConfigureName());
+ if (result != null) {
+ return result;
+ }
+ try {
+ result = doAction(context, rule);
+ if (result != null) {
+ context.putExpressionValue(getNameSpace(), getConfigureName(), result);
+ }
+ } catch (Exception e) {
+ LOG.error("Expression getExpressionValue error,rule is:" + rule.getConfigureName() + " ,express is: "
+ + getConfigureName(), e);
+ return false;
+ }
+ //if(result==false){
+ // context.setNotFireExpression(this.toString());
+ //}
+ return result;
+ }
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public String toJson() {
+ String valueStr = String.valueOf(value);
+ try {
+ // 适配fastjson的bug，parseArray时会对"-"进行分割
+ if (ListDataType.class.isInstance(dataType) && valueStr.contains("-") && valueStr.contains(",")) {
+ this.value = (T)value;
+ } else {
+ // value很多时候会是String类型，这样dataType.toDataJson会报错，所以先转为dataType类型,
+ // list类型JsonArray.tojson会报错，不做处理（[pfile_path_rule,proc_vul_exploit_rce_rule]）
+ if (!ListDataType.class.isInstance(dataType)) {
+ this.value = (T)dataType.getData(valueStr);
+ }
+ }
+ } catch (Exception e) {
+ e.printStackTrace();
+ }
+ JSONObject jsonObject = new JSONObject();
+ jsonObject.put(IConfigurableService.CLASS_NAME, this.getClass().getName());
+ jsonObject.put("varName", varName);
+ jsonObject.put("functionName", functionName);
+
+ if (FIELD_COMPARE.equals(dataTypestr)) {
+ fieldFlag = true;
+ }
+
+ // 如果是String，且不是字段比较，则加密
+ if (!fieldFlag) {
+ if (StringDataType.class.isInstance(dataType)) {
+ try {
+ jsonObject.put("value", AESUtil.aesEncrypt(valueStr, AES_KEY));
+ jsonObject.put("aesFlag", 1);
+ } catch (Exception e) {
+ jsonObject.put("value", dataType.toDataJson(value));
+ }
+ } else {
+ jsonObject.put("value", dataType.toDataJson(value));
+ }
+ } else {
+ jsonObject.put("value", valueStr);
+ }
+ jsonObject.put("dataType", dataType.toJson());
+ jsonObject.put("keyword", keyword == null ? "" : keyword.toLowerCase().trim());
+ jsonObject.put("fieldFlag", fieldFlag);
+
+ getJsonValue(jsonObject);
+ return jsonObject.toJSONString();
+ }
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void toObject(String jsonString) {
+ JSONObject jsonObject = JSONObject.parseObject(jsonString);
+ this.varName = jsonObject.getString("varName");
+ this.functionName = jsonObject.getString("functionName");
+ String dataTypeJson = jsonObject.getString("dataType");
+ this.dataType = DataTypeUtil.createDataType(dataTypeJson);
+ String valueJson = jsonObject.getString("value");
+ this.aesFlag = jsonObject.getIntValue("aesFlag");
+ this.fieldFlag = jsonObject.getBooleanValue("fieldFlag");
+
+ // 前端显示用
+ String dataTypestr = "";
+ try {
+ if (dataType != null) {
+ dataTypestr = MetaDataField.getDataTypeStrByType(dataType);
+ }
+ } catch (Exception e) {
+ dataTypestr = "String";
+ }
+
+ if (!fieldFlag) {
+ if (StringDataType.class.isInstance(dataType)) {
+ // value经过加密 需要解密
+ if (aesFlag == 1) {
+ try {
+ this.value = (T)AESUtil.aesDecrypt(valueJson, AES_KEY);
+ } catch (Exception e) {
+ this.value = (T)this.dataType.getData(valueJson);
+ }
+ } else {
+ this.value = (T)this.dataType.getData(valueJson);
+ }
+ } else {
+ this.value = (T)this.dataType.getData(valueJson);
+ }
+ } else {
+ dataTypestr = FIELD_COMPARE;
+ this.value = (T)valueJson;
+ }
+
+ this.dataTypestr = dataTypestr;
+ this.keyword =
+ jsonObject.getString("keyword") == null ? "" : jsonObject.getString("keyword").toLowerCase().trim();
+ setJsonValue(jsonObject);
+ }
+
+ protected void getJsonValue(JSONObject jsonObject) {
+ }
+
+ protected void setJsonValue(JSONObject jsonObject) {
+ }
+
+ public Set<String> getDependentFields(Map<String, Expression> expressionMap) {
+ Set<String> set = new HashSet<>();
+ if (StringUtil.isNotEmpty(varName) && !ContantsUtil.isContant(varName) && !FunctionUtils.isNumber(varName) && !FunctionUtils.isBoolean(varName)) {
+ set.add(varName);
+ }
+ return set;
+ }
+
+ public String getVarName() {
+ return varName;
+ }
+
+ public void setVarName(String varName) {
+ this.varName = varName;
+ }
+
+ public String getFunctionName() {
+ return functionName;
+ }
+
+ public void setFunctionName(String functionName) {
+ this.functionName = functionName;
+ }
+
+ public DataType getDataType() {
+ return dataType;
+ }
+
+ public void setDataType(DataType dataType) {
+ this.dataType = dataType;
+ }
+
+ public T getValue() {
+ return value;
+ }
+
+ public void setValue(T value) {
+ this.value = value;
+ }
+
+ @Override
+ public boolean init() {
+ return true;
+ }
+
+ /**
+ * 是否支持快速匹配，快速匹配的意思是var不需要io
+ *
+ * @param expression
+ * @param context
+ * @param rule
+ * @return
+ */
+ public boolean supportQuickMatch(Expression expression, RuleContext context, Rule rule) {
+ String varName = expression.getVarName();
+ Var var = context.getVar(rule.getConfigureName(), varName);
+ if (var == null || var.canLazyLoad()) {
+ return false;
+ }
+ return true;
+ }
+
+ /**
+ * 合法性验证
+ *
+ * @return
+ */
+ public boolean volidate() {
+ if (StringUtil.isEmpty(varName) || StringUtil.isEmpty(functionName) || dataType == null) {
+ return false;
+ }
+ return true;
+ }
+
+ @Override
+ public boolean volidate(RuleContext context, Rule rule) {
+ return true;
+ }
+
+ public String getDataTypestr() {
+ return dataTypestr;
+ }
+
+ public void setDataTypestr(String dataTypestr) {
+ this.dataTypestr = dataTypestr;
+ if (FIELD_COMPARE.equals(dataTypestr)) {
+ fieldFlag = true;
+ }
+
+ DataType dt = MetaDataField.getDataTypeByStr(dataTypestr);
+ this.dataType = dt;
+ }
+
+ public String getKeyword() {
+ return keyword;
+ }
+
+ public void setKeyword(String keyword) {
+ if (keyword != null) {
+ keyword = keyword.toLowerCase().trim();
+ }
+ this.keyword = keyword;
+ }
+
+ public int getAesFlag() {
+ return aesFlag;
+ }
+
+ public void setAesFlag(int aesFlag) {
+ this.aesFlag = aesFlag;
+ }
+
+ /**
+ * 返回expressCode
+ *
+ * @return
+ */
+ @SuppressWarnings("rawtypes")
+ public String toExpressionString(Map<String, Expression> name2Expression,
+ String... expressionNamePrefixs) {
+ return toString();
+ }
+
+ @Override
+ public String toString() {
+ String dataType = null;
+ String result = "(" + varName + "," + getFunctionName() + ",";
+ if (!getDataType().matchClass(String.class)) {
+ dataType = getDataType().getDataTypeName();
+ result += dataType + ",";
+ }
+ String value = getDataType().toDataJson(getValue());
+ if (getDataType().matchClass(String.class) && needContants(value)) {
+ value = "'" + value + "'";
+ }
+ result += value + ")";
+
+ return result;
+ }
+
+ private boolean needContants(String value) {
+ if (value.indexOf("(") != -1 || value.indexOf(",") != -1) {
+ return true;
+ }
+ return false;
+ }
+
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/expression/ExpressionPerformance.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/expression/ExpressionPerformance.java
new file mode 100644
index 0000000..b583cad
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/expression/ExpressionPerformance.java
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.operator.expression;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+public class ExpressionPerformance implements Runnable {
+ protected static List<ExpressionPerformance> expressionPerformances = new ArrayList<>();
+
+ protected transient volatile List<String> expressionNames = new ArrayList<>();
+ protected transient Map<String, ExpressionStatistic> expressionStatisticMap = new HashMap<>();
+ protected transient long lastTime = System.currentTimeMillis();//最后一次的优化时间
+ protected transient final List<String> values;
+ private static ScheduledExecutorService scheduledExecutorService = Executors.newScheduledThreadPool(10);
+
+ static {
+
+ scheduledExecutorService.scheduleWithFixedDelay(new Runnable() {
+ @Override
+ public void run() {
+ for (ExpressionPerformance expressionPerformance : expressionPerformances) {
+ expressionPerformance.run();
+ }
+ }
+ }, 10, 3, TimeUnit.SECONDS);
+ }
+
+ @Override
+ public void run() {
+ List<String> newExpressionNames = new ArrayList<>();
+ List<ExpressionStatistic> statistics = new ArrayList<>();
+ statistics.addAll(expressionStatisticMap.values());
+ Collections.sort(statistics, new Comparator<ExpressionStatistic>() {
+ @Override
+ public int compare(ExpressionStatistic o1, ExpressionStatistic o2) {
+ return o2.count.get() - o1.count.get();
+ }
+ });
+ for (ExpressionStatistic statistic : statistics) {
+ newExpressionNames.add(statistic.name);
+ }
+ this.expressionNames = newExpressionNames;
+ this.lastTime = System.currentTimeMillis();
+ }
+
+ protected static class ExpressionStatistic {
+ protected String name;
+ protected AtomicInteger count = new AtomicInteger(0);
+ }
+
+ public ExpressionPerformance(List<String> values) {
+ this.expressionNames = values;
+ this.values = values;
+ for (String name : expressionNames) {
+ ExpressionStatistic expressionStatistic = new ExpressionStatistic();
+ expressionStatistic.name = name;
+ expressionStatistic.count.set(0);
+ expressionStatisticMap.put(name, expressionStatistic);
+ }
+ expressionPerformances.add(this);
+ }
+
+ public Boolean optimizate(String expressionName, Boolean value) {
+ ExpressionStatistic expressionStatistic = expressionStatisticMap.get(expressionName);
+ expressionStatistic.count.incrementAndGet();
+ return value;
+ }
+
+ public Iterator<String> iterator() {
+ if (expressionNames == null) {
+ return values.iterator();
+ }
+ return expressionNames.iterator();//expressionNames 会做优化，会给快速失效的表达式，加权中
+ }
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/expression/ExpressionRelationParser.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/expression/ExpressionRelationParser.java
new file mode 100644
index 0000000..ff813e5
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/expression/ExpressionRelationParser.java
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.operator.expression;
+
+import org.apache.rocketmq.streams.common.model.NameCreator;
+
+import java.util.ArrayList;
+import java.util.List;
+
+public class ExpressionRelationParser {
+ public static final String OR = "|";
+ public static final String AND = "&";
+
+ /**
+ * 创建表达式
+ *
+ * @param namespace
+ * @param str
+ * @param groupList
+ * @return
+ */
+ public static RelationExpression createRelations(String namespace, String ruleName, String str, List<RelationExpression> groupList) {
+ if (str.indexOf(")") == -1) {
+ return createMixRelation(namespace, ruleName, str, groupList);
+ }
+ int endIndex = str.indexOf(")");
+ int startIndex = 0;
+ for (int i = endIndex; i > 0; i--) {
+ String word = str.substring(i - 1, i);
+ if ("(".equals(word)) {
+ startIndex = i;
+ break;
+ }
+ }
+ String expression = str.substring(startIndex, endIndex);
+ RelationExpression relationExpression = createMixRelation(namespace, ruleName, expression, groupList);
+ str = str.replace("(" + expression + ")", relationExpression.getConfigureName());
+ return createRelations(namespace, ruleName, str, groupList);
+ }
+
+ /**
+ * 混合表达关系处理
+ *
+ * @param namespace
+ * @param str
+ * @return
+ */
+ private static RelationExpression createMixRelation(String namespace, String ruleName, String str,
+ List<RelationExpression> groupList) {
+ if (str.indexOf(OR) == -1) {
+ return createSignleRelation(namespace, ruleName, str, AND, groupList);
+ }
+ String[] values = str.split("\\" + OR);
+ for (String value : values) {
+ String sign = AND;
+ if (value.indexOf(sign) != -1) {
+ RelationExpression relationExpression = createSignleRelation(namespace, ruleName, value, sign, groupList);
+ str = str.replace(value, relationExpression.getConfigureName());
+ }
+ }
+ str = str.replace(" ", "");
+ return createSignleRelation(namespace, ruleName, str, OR, groupList);
+ }
+
+ /**
+ * 纯粹表达关系处理
+ *
+ * @param namespace
+ * @param str
+ * @param sign
+ * @return
+ */
+ private static RelationExpression createSignleRelation(String namespace, String ruleName, String str, String sign,
+ List<RelationExpression> groupList) {
+ NameCreator nameCreator = NameCreator.createOrGet(ruleName);
+ String expressionName = nameCreator.createName(ruleName);
+ RelationExpression relationExpression = new RelationExpression();
+ relationExpression.setNameSpace(namespace);
+ relationExpression.setType(Expression.TYPE);
+ relationExpression.setValue(new ArrayList<String>());
+ relationExpression.setConfigureName(expressionName);
+
+ relationExpression.setRelation(OR.equals(sign) ? "or" : "and");
+ String[] values = str.split("\\" + sign);
+ for (String value : values) {
+ relationExpression.addExpression(value);
+ }
+ groupList.add(relationExpression);
+
+ return relationExpression;
+ }
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/expression/ExpressionRelationPaser.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/expression/ExpressionRelationPaser.java
new file mode 100644
index 0000000..942ef67
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/expression/ExpressionRelationPaser.java
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.streams.filter.operator.expression;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.rocketmq.streams.common.model.NameCreator;
+
+public class ExpressionRelationPaser {
+ public static final String OR = "|";
+ public static final String AND = "&";
+
+
+ /**
+ * 创建表达式
+ *
+ * @param namespace
+ * @param str
+ * @param groupList
+ * @return
+ */
+ public static RelationExpression createRelations(String namespace,String ruleName, String str, List<RelationExpression> groupList) {
+ if (str.indexOf(")") == -1) {
+ return createMixRelation(namespace, ruleName,str, groupList);
+ }
+ int endIndex = str.indexOf(")");
+ int startIndex = 0;
+ for (int i = endIndex; i > 0; i--) {
+ String word = str.substring(i - 1, i);
+ if ("(".equals(word)) {
+ startIndex = i;
+ break;
+ }
+ }
+ String expression = str.substring(startIndex, endIndex);
+ RelationExpression relationExpression = createMixRelation(namespace, ruleName,expression, groupList);
+ str = str.replace("(" + expression + ")", relationExpression.getConfigureName());
+ return createRelations(namespace,ruleName, str, groupList);
+ }
+
+ /**
+ * 混合表达关系处理
+ *
+ * @param namespace
+ * @param str
+ * @return
+ */
+ private static RelationExpression createMixRelation(String namespace,String ruleName, String str,
+ List<RelationExpression> groupList) {
+ if (str.indexOf(OR) == -1) {
+ return createSignleRelation(namespace,ruleName, str, AND, groupList);
+ }
+ String[] values = str.split("\\" + OR);
+ for (String value : values) {
+ String sign = AND;
+ if (value.indexOf(sign) != -1) {
+ RelationExpression relationExpression = createSignleRelation(namespace, ruleName, value, sign, groupList);
+ str = str.replace(value, relationExpression.getConfigureName());
+ }
+ }
+ str = str.replace(" ", "");
+ return createSignleRelation(namespace,ruleName, str, OR, groupList);
+ }
+
+ /**
+ * 纯粹表达关系处理
+ *
+ * @param namespace
+ * @param str
+ * @param sign
+ * @return
+ */
+ private static RelationExpression createSignleRelation(String namespace,String ruleName, String str, String sign,
+ List<RelationExpression> groupList) {
+ NameCreator nameCreator=NameCreator.createOrGet(ruleName);
+ String expressionName=nameCreator.createName(ruleName);
+ RelationExpression relationExpression = new RelationExpression();
+ relationExpression.setNameSpace(namespace);
+ relationExpression.setType(Expression.TYPE);
+ relationExpression.setValue(new ArrayList<String>());
+ relationExpression.setConfigureName(expressionName);
+
+ relationExpression.setRelation(OR.equals(sign) ? "or" : "and");
+ String[] values = str.split("\\" + sign);
+ for (String value : values) {
+ relationExpression.addExpression(value);
+ }
+ groupList.add(relationExpression);
+
+ return relationExpression;
+ }
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/expression/GroupExpression.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/expression/GroupExpression.java
new file mode 100644
index 0000000..69a61e5
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/expression/GroupExpression.java
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.operator.expression;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.rocketmq.streams.common.model.NameCreator;
+import org.apache.rocketmq.streams.filter.context.RuleContext;
+import org.apache.rocketmq.streams.filter.function.expression.RegexFunction;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+import org.apache.rocketmq.streams.common.cache.compress.impl.IntValueKV;
+import org.apache.rocketmq.streams.common.utils.MapKeyUtil;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+
+public class GroupExpression extends Expression<List<Expression>> {
+ protected Rule rule;
+ protected String varName;
+ protected static IntValueKV cache = new IntValueKV(3000000);
+ protected boolean isOrRelation = true;//是否是or关系
+ protected Map<String, Boolean> expressionName2Result = new HashMap<>();//正则类表达式的名字和结果的映射
+ protected Set<String> regeExpressionNameSet = new HashSet<>();//正则类表达式的名字
+
+ public GroupExpression(Rule rule, String varName, boolean isOrRelation) {
+ this.rule = rule;
+ this.varName = varName;
+ this.isOrRelation = isOrRelation;
+ this.setConfigureName(NameCreator.createNewName("expression.group"));
+ value = new ArrayList<>();
+ this.setNameSpace(rule.getNameSpace());
+ }
+
+ @Override
+ public Boolean doAction(RuleContext context, Rule rule) {
+ String varValue = context.getMessage().getMessageBody().getString(varName);
+ String key = MapKeyUtil.createKey(getNameSpace(), rule.getConfigureName(), getConfigureName(), varValue);
+ Integer value = cache.get(key);
+ if (value == null) {
+ Boolean isMatch = executeMatch(context, rule);
+ if (isMatch) {
+ value = 1;
+ } else {
+ value = 0;
+ }
+ cache.put(key, value);
+ }
+ return value > 0;
+ }
+
+ @Override
+ public String toString() {
+ return "";
+ }
+
+ @Override
+ public Set<String> getDependentFields(Map<String, Expression> expressionMap) {
+ Set<String> set = new HashSet<>();
+ set.add(varName);
+
+ return set;
+ }
+
+ private Boolean executeMatch(RuleContext context, Rule rule) {
+ for (Expression expression : getValue()) {
+ Boolean result = expressionName2Result.get(expression.getConfigureName());
+ boolean isMatch = false;
+ if (result != null) {
+ isMatch = result;
+ } else {
+ isMatch = expression.doAction(context, rule);
+ }
+ if (isOrRelation) {
+ if (isMatch) {
+ return true;
+ }
+ } else {
+ if (!isMatch) {
+ return false;
+ }
+ }
+ }
+ if (isOrRelation) {
+ return false;
+ } else {
+ return true;
+ }
+ }
+
+ public void addExpressionName(Expression expression) {
+ if (RegexFunction.isRegex(expression.getFunctionName())) {
+ regeExpressionNameSet.add(expression.getConfigureName());
+ }
+ getValue().add(expression);
+ }
+
+ public void setRegexResult(Set<String> allRegexResult) {
+ Iterator<String> it = regeExpressionNameSet.iterator();
+ while (it.hasNext()) {
+ String name = it.next();
+ Boolean value = allRegexResult.contains(name);
+ if (value == null) {
+ value = false;
+ }
+ expressionName2Result.put(name, value);
+
+ }
+ }
+
+ public int size() {
+ return getValue().size();
+ }
+
+ public Collection<? extends String> getAllExpressionNames() {
+ Set<String> names = new HashSet<>();
+ for (Expression expression : getValue()) {
+ names.add(expression.getConfigureName());
+ }
+ return names;
+ }
+
+ public static void main(String[] args) {
+ String content = "abdfdfd";
+ String regex = "ab.*fd";
+ System.out.println(StringUtil.matchRegex(content, regex));
+ }
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/expression/GroupExpressionManager.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/expression/GroupExpressionManager.java
new file mode 100644
index 0000000..09e34e9
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/expression/GroupExpressionManager.java
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.operator.expression;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+
+import com.alibaba.fastjson.JSONObject;
+
+import org.apache.rocketmq.streams.filter.context.RuleContext;
+import org.apache.rocketmq.streams.filter.function.expression.RegexFunction;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+import org.apache.rocketmq.streams.common.optimization.HyperscanRegex;
+
+/**
+ * hyperscan和规则的结合，目前暂不启用。
+ */
+public class GroupExpressionManager {
+ protected Rule rule;
+ protected Map<String, HyperscanRegex> hyperscanRegexMap = new HashMap<>();
+ protected List<GroupExpression> groupExpressions = new ArrayList<>();
+
+ public GroupExpressionManager(Rule rule) {
+ this.rule = rule;
+
+ }
+
+ public void compile() {
+ for (Expression expression : rule.getExpressionMap().values()) {
+ if (SimpleExpression.class.isInstance(expression) && RegexFunction.isRegex(expression.getFunctionName())) {
+ String varName = expression.getVarName();
+ HyperscanRegex hyperscanRegex = hyperscanRegexMap.get(varName);
+ if (hyperscanRegex == null) {
+ hyperscanRegex = new HyperscanRegex();
+ hyperscanRegexMap.put(varName, hyperscanRegex);
+ }
+ hyperscanRegex.addRegex((String)expression.getValue(), expression.getConfigureName());
+ }
+ }
+ for (HyperscanRegex hyperscanRegex : hyperscanRegexMap.values()) {
+ hyperscanRegex.compile();
+ }
+ }
+
+ public void matchAndSetResult(RuleContext context, Rule rule) {
+ Set<String> allRegexResult = new HashSet<>();
+ JSONObject msg = context.getMessage().getMessageBody();
+ Iterator<Entry<String, HyperscanRegex>> it = hyperscanRegexMap.entrySet().iterator();
+ while (it.hasNext()) {
+ Entry<String, HyperscanRegex> entry = it.next();
+ String varName = entry.getKey();
+ String varValue = msg.getString(varName);
+ HyperscanRegex hyperscanRegex = entry.getValue();
+ Set<String> expressionNames = hyperscanRegex.matchExpression(varValue);
+ allRegexResult.addAll(expressionNames);
+ }
+ for (GroupExpression groupExpression : groupExpressions) {
+ groupExpression.setRegexResult(allRegexResult);
+ }
+ }
+
+ public void addGroupExpression(GroupExpression groupExpression) {
+ groupExpressions.add(groupExpression);
+ }
+
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/expression/OptimizationRule.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/expression/OptimizationRule.java
new file mode 100644
index 0000000..28f4001
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/expression/OptimizationRule.java
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.operator.expression;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.rocketmq.streams.filter.operator.Rule;
+
+public class OptimizationRule {
+ protected Rule rule;
+ protected Map<String, Expression> expressionMap = new HashMap<>();
+ protected String rootExpressionName;
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/expression/RelationExpression.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/expression/RelationExpression.java
new file mode 100644
index 0000000..13b8589
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/expression/RelationExpression.java
@@ -0,0 +1,347 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.operator.expression;
+
+import com.alibaba.fastjson.JSONObject;
+import org.apache.rocketmq.streams.filter.context.RuleContext;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.rocketmq.streams.common.context.Message;
+import org.apache.rocketmq.streams.common.monitor.TopologyFilterMonitor;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+import org.apache.rocketmq.streams.script.utils.FunctionUtils;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+public class RelationExpression extends Expression<List<String>> {
+
+ private static final long serialVersionUID = -3213091464347965570L;
+ private static final Log LOG = LogFactory.getLog(RelationExpression.class);
+ private String relation = "and"; // and or
+ // 前端处理使用
+ private String expressions; // @隔开的表达式名称
+
+ protected volatile ExpressionPerformance expressionPerformance;
+
+ @Override
+ protected void getJsonValue(JSONObject jsonObject) {
+ jsonObject.put("relation", relation);
+ }
+
+ @Override
+ protected void setJsonValue(JSONObject jsonObject) {
+ this.relation = jsonObject.getString("relation");
+ }
+
+ public RelationExpression() {
+
+ }
+
+ @Override
+ protected boolean initConfigurable() {
+ boolean success = super.initConfigurable();
+ expressionPerformance = new ExpressionPerformance(getValue());
+ return success;
+ }
+
+ public void addExpression(String expressionName) {
+ value = this.getValue();
+ if (value == null) {
+ value = new ArrayList<String>();
+ }
+ value.add(expressionName);
+ }
+
+ public Iterator<String> iterator() {
+ if (value == null) {
+ return null;
+ }
+ if (expressionPerformance != null) {
+ return expressionPerformance.iterator();//expressionNames 会做优化，会给快速失效的表达式，加权中
+ } else {
+ return value.iterator();
+ }
+
+ }
+
+ public boolean isOrRelation() {
+ return "or".equals(relation);
+ }
+
+ @Override
+ public boolean supportQuickMatch(Expression expression, RuleContext context, Rule rule) {
+ Iterator<String> it = iterator();
+ if (it == null) {
+ return true;
+ }
+ while (it.hasNext()) {
+ String expressionName = it.next();
+ Expression e = context.getExpression(expressionName);
+ if (e == null) {
+ continue;
+ }
+ if (!e.supportQuickMatch(e, context, rule)) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ @Override
+ public Boolean doAction(RuleContext context, Rule rule) {
+
+ Iterator<String> it = iterator();
+ /**
+ * 如果表达式组的值为空，则返回false
+ */
+ if (!it.hasNext()) {
+ return false;
+ }
+
+ boolean flag = true;
+ if (StringUtil.isEmpty(relation)) {
+ return false;
+ }
+ if ("and".equals(this.relation)) {
+ while (it.hasNext()) {
+ if (flag) {
+ String expressionName = it.next();
+ Expression exp = context.getExpression(expressionName);
+ if (exp == null) {
+ Message message = context.getMessage();
+ Boolean result = message.getMessageBody().getBoolean(expressionName);
+ if (result != null) {
+ if (result == false) {
+ TopologyFilterMonitor piplineExecutorMonitor = new TopologyFilterMonitor();
+ piplineExecutorMonitor.addNotFireExpression(expressionName, expressionName);
+ context.setExpressionMonitor(piplineExecutorMonitor);
+ return optimizate(expressionName, false);
+ } else {
+ continue;
+ }
+ } else {
+ throw new RuntimeException("expect exist expression, but not " + expressionName);
+ }
+
+ }
+
+ if (RelationExpression.class.isInstance(exp)) {
+ Boolean foreachResult = exp.doAction(context, rule);
+ if (foreachResult != null && !foreachResult) {
+ // expressions.add(exp.toString());
+ return optimizate(expressionName, false);
+ }
+ } else {
+ try {
+ flag = exp.getExpressionValue(context, rule);
+ } catch (Exception e) {
+ LOG.error("RelationExpression and function.doFunction error,rule is: "
+ + rule.getConfigureName() + " ,express is: " + exp.getConfigureName(), e);
+ e.printStackTrace();
+ return false;
+ }
+ if (!flag) {
+ TopologyFilterMonitor piplineExecutorMonitor = new TopologyFilterMonitor();
+ piplineExecutorMonitor.addNotFireExpression(exp.toString(), exp.getDependentFields(rule.getExpressionMap()));
+ context.setExpressionMonitor(piplineExecutorMonitor);
+ return optimizate(expressionName, false);
+ }
+ }
+ }
+ }
+ return true;
+ } else {// or
+ flag = false;
+ TopologyFilterMonitor piplineExecutorMonitor = new TopologyFilterMonitor();
+ while (it.hasNext()) {
+ String expressionName = it.next();
+ Expression exp = context.getExpression(expressionName);
+ if (exp == null) {
+ Message message = context.getMessage();
+ Boolean result = message.getMessageBody().getBoolean(expressionName);
+ if (result != null) {
+ if (result == true) {
+ return optimizate(expressionName, true);
+ } else {
+ continue;
+ }
+ } else {
+ throw new RuntimeException("expect exist expression, but not " + expressionName);
+ }
+ }
+ if (RelationExpression.class.isInstance(exp)) {
+ Boolean foreachResult = exp.doAction(context, rule);
+ if (foreachResult != null && foreachResult) {
+ // expressions.add(exp.toString());
+ return optimizate(expressionName, true);
+ } else {
+ //如果关系表达式未触发，则检测context中，有没有因为and失败的条件
+ if (context.getExpressionMonitor() != null && context.getExpressionMonitor().getNotFireExpression2DependentFields().size() > 0) {
+ piplineExecutorMonitor.addNotFireExpression(context.getExpressionMonitor().getNotFireExpression2DependentFields());
+ }
+ }
+ } else {
+
+ try {
+ flag = exp.getExpressionValue(context, rule);
+ } catch (Exception e) {
+ LOG.error(
+ "RelationExpression or function.doFunction error,rule is: " + rule.getConfigureName()
+ + " ,express is: " + exp.getConfigureName(), e);
+ }
+ if (flag) {
+ return optimizate(expressionName, true);
+ }
+ }
+
+ }
+ context.setExpressionMonitor(piplineExecutorMonitor);
+ return false;
+ }
+
+ }
+
+ public Boolean optimizate(String exprssionName, Boolean value) {
+ if (expressionPerformance == null) {
+ synchronized (this) {
+ if (expressionPerformance == null) {
+ expressionPerformance = new ExpressionPerformance(getValue());
+ }
+ }
+
+ }
+ return expressionPerformance.optimizate(exprssionName, value);
+ }
+
+ public String getRelation() {
+ return relation;
+ }
+
+ public void setRelation(String relation) {
+ this.relation = relation;
+ }
+
+ /**
+ * 合法性验证
+ *
+ * @return
+ */
+ @Override
+ public boolean volidate() {
+ return true;
+ }
+
+ public String getExpressions() {
+ return expressions;
+ }
+
+ public void setExpressions(String expressions) {
+ if (expressions == null) {
+ return;
+ }
+ this.expressions = expressions.trim();
+
+ String[] exps = expressions.split("@");
+ if (exps.length > 0) {
+ for (String express : exps) {
+ // 防止表达式组里依赖自己，造成死循环
+ if (this.getConfigureName().equals(express)) {
+ continue;
+ }
+ this.addExpression(express.trim());
+ }
+ }
+
+ }
+
+ @Override
+ public Set<String> getDependentFields(Map<String, Expression> expressionMap) {
+ Set<String> set = new HashSet<>();
+ if (getValue() == null) {
+ return set;
+ }
+ for (String expressionName : value) {
+ Expression expression = expressionMap.get(expressionName);
+ if (expression == null) {
+ if (StringUtil.isNotEmpty(expressionName) && !FunctionUtils.isBoolean(expressionName)) {
+ set.add(expressionName);
+ }
+
+ } else {
+ set.addAll(expression.getDependentFields(expressionMap));
+ }
+ }
+ return set;
+ }
+
+ /**
+ * 前端展示用，不设置value值
+ *
+ * @param expressions
+ */
+ public void setExpressionsNotValue(String expressions) {
+ if (expressions == null) {
+ return;
+ }
+ this.expressions = expressions.trim();
+ }
+
+ @Override
+ public String toExpressionString(Map<String, Expression> name2Expression,
+ String... expressionNamePrefixs) {
+ String sign = "and".equals(relation) ? "&" : "|";
+ StringBuilder sb = new StringBuilder();
+ sb.append("(");
+ boolean isFirst = true;
+ for (String expressionName : this.getValue()) {
+ String expressKey = expressionName;
+ Expression expression = null;
+ if (expressionNamePrefixs != null && expressionNamePrefixs.length > 0) {
+ for (String prefix : expressionNamePrefixs) {
+ expressKey = prefix + expressionName;
+ expression = name2Expression.get(expressKey);
+ if (expression != null) {
+ break;
+ }
+ }
+ } else {
+ expression = name2Expression.get(expressKey);
+ }
+
+ if (expression == null) {
+ continue;
+ }
+ if (isFirst) {
+ isFirst = false;
+ } else {
+ sb.append(sign);
+ }
+ sb.append(
+ expression.toExpressionString(name2Expression, expressionNamePrefixs));
+ }
+ sb.append(")");
+ return sb.toString();
+ }
+
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/expression/SimpleExpression.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/expression/SimpleExpression.java
new file mode 100644
index 0000000..df34a3b
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/expression/SimpleExpression.java
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.operator.expression;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import com.alibaba.fastjson.JSONObject;
+import org.apache.rocketmq.streams.filter.contants.RuleElementType;
+import org.apache.rocketmq.streams.filter.builder.ExpressionBuilder;
+import org.apache.rocketmq.streams.filter.context.RuleContext;
+import org.apache.rocketmq.streams.filter.optimization.EqualsExpressionOptimization;
+import org.apache.rocketmq.streams.filter.optimization.IExpressionOptimization;
+import org.apache.rocketmq.streams.filter.optimization.LikeExpressionOptimization;
+import org.apache.rocketmq.streams.filter.optimization.OptimizationExpression;
+import org.apache.rocketmq.streams.filter.optimization.RegexExpressionOptimization;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+import org.apache.rocketmq.streams.common.datatype.DataType;
+import org.apache.rocketmq.streams.common.optimization.CalculationResultCache;
+import org.apache.rocketmq.streams.common.utils.DataTypeUtil;
+
+/**
+ * 变量名就是字段名，不需要声明meta
+ */
+public class SimpleExpression extends Expression {
+ protected static CalculationResultCache calculationResultCache = CalculationResultCache.getInstance();
+
+ public SimpleExpression() {
+ }
+
+ public SimpleExpression(String msgFieldName, String functionName, String value) {
+ this(msgFieldName, functionName, DataTypeUtil.getDataTypeFromClass(String.class), value);
+ }
+
+ public SimpleExpression(String msgFieldName, String functionName, DataType dataType, String value) {
+ setType(RuleElementType.EXPRESSION.getType());
+ setFunctionName(functionName);
+ setVarName(msgFieldName);
+ if (dataType == null) {
+ dataType = DataTypeUtil.getDataTypeFromClass(String.class);
+ }
+ setValue(dataType.getData(value));
+ setDataType(dataType);
+ }
+
+ @Override
+ public Boolean getExpressionValue(RuleContext context, Rule rule) {
+ IExpressionOptimization expressionOptimization = getOptimize();
+ Boolean isHitCache = null;
+ String regex = null;
+ String value = null;
+ if (expressionOptimization != null) {
+ OptimizationExpression expression = expressionOptimization.optimize(this);
+ regex = expression.getRegex();
+ String varName = expression.getVarName();
+ value = context.getMessage().getMessageBody().getString(varName);
+ isHitCache = calculationResultCache.match(regex, value);
+ if (isHitCache != null) {
+ return isHitCache;
+ }
+ }
+ boolean isMatch = super.getExpressionValue(context, rule);
+ if (isHitCache == null && expressionOptimization != null) {
+ calculationResultCache.registeRegex(regex, value, isMatch);
+ }
+ return isMatch;
+ }
+
+ /**
+ * 找到函数对应的优化
+ *
+ * @return
+ */
+ private IExpressionOptimization getOptimize() {
+ for (IExpressionOptimization expressionOptimization : expressionOptimizations) {
+ if (expressionOptimization.support(this)) {
+ return expressionOptimization;
+ }
+ }
+ return null;
+ }
+
+ private static List<IExpressionOptimization> expressionOptimizations = new ArrayList<>();
+
+ static {
+ expressionOptimizations.add(new EqualsExpressionOptimization());
+ expressionOptimizations.add(new RegexExpressionOptimization());
+ expressionOptimizations.add(new LikeExpressionOptimization());
+ }
+
+ public boolean doExecute(JSONObject msg) {
+ return ExpressionBuilder.executeExecute(this, msg);
+ }
+
+ public void setMsgFieldName(String msgFieldName) {
+ setVarName(msgFieldName);
+ }
+
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/var/ConstantVar.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/var/ConstantVar.java
new file mode 100644
index 0000000..8a418bc
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/var/ConstantVar.java
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.operator.var;
+
+import com.alibaba.fastjson.JSONObject;
+import org.apache.rocketmq.streams.filter.context.RuleContext;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+import org.apache.rocketmq.streams.common.datatype.DataType;
+import org.apache.rocketmq.streams.common.datatype.StringDataType;
+import org.apache.rocketmq.streams.common.metadata.MetaDataField;
+import org.apache.rocketmq.streams.common.utils.DataTypeUtil;
+
+public class ConstantVar<T> extends Var<T> {
+
+ private static final long serialVersionUID = 8490929155047073802L;
+ protected T value;
+ protected DataType<T> dataType;
+
+ // 前端处理使用
+ private String dataTypestr;
+
+ @Override
+ public T doAction(RuleContext context, Rule rule) {
+ return value;
+ }
+
+ @Override
+ public boolean volidate(RuleContext context, Rule rule) {
+ if (value == null || dataType == null) {
+ return false;
+ }
+ return true;
+ }
+
+ @SuppressWarnings("unchecked")
+ @Override
+ protected void getJsonObject(JSONObject jsonObject) {
+ // value很多时候会是String类型，这样dataType.toDataJson会报错，所以先转为dataType类型
+ if (dataType == null) {
+ dataType = (DataType<T>)new StringDataType(String.class);
+ }
+ try {
+ this.value = (T)dataType.getData(String.valueOf(value));
+ } catch (Exception e) {
+ e.printStackTrace();
+ }
+
+ jsonObject.put("value", dataType.toDataJson(value));
+ jsonObject.put("dataType", dataType.toJson());
+ }
+
+ @SuppressWarnings("unchecked")
+ @Override
+ protected void setJsonObject(JSONObject jsonObject) {
+ String dataTypeJson = jsonObject.getString("dataType");
+ dataType = (DataType<T>)DataTypeUtil.createDataType(dataTypeJson);
+ String valueString = jsonObject.getString("value");
+ this.value = (T)dataType.getData(valueString);
+
+ // 前端显示用
+ String dataTypestr = "";
+ try {
+ if (dataType != null) {
+ dataTypestr = MetaDataField.getDataTypeStrByType(dataType);
+ }
+ } catch (Exception e) {
+ dataTypestr = "String";
+ }
+ this.dataTypestr = dataTypestr;
+ }
+
+ @Override
+ public boolean canLazyLoad() {
+ return false;
+ }
+
+ public T getValue() {
+ return value;
+ }
+
+ public void setValue(T value) {
+ this.value = value;
+ }
+
+ public DataType<T> getDataType() {
+ return dataType;
+ }
+
+ public void setDataType(DataType<T> dataType) {
+ this.dataType = dataType;
+ }
+
+ public String getDataTypestr() {
+ return dataTypestr;
+ }
+
+ @SuppressWarnings({"rawtypes", "unchecked"})
+ public void setDataTypestr(String dataTypestr) {
+ this.dataTypestr = dataTypestr;
+ DataType dt = MetaDataField.getDataTypeByStr(dataTypestr);
+ this.dataType = dt;
+ }
+
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/var/ContextVar.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/var/ContextVar.java
new file mode 100644
index 0000000..be90dca
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/var/ContextVar.java
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.operator.var;
+
+import com.alibaba.fastjson.JSONObject;
+import org.apache.rocketmq.streams.filter.context.RuleContext;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+import org.apache.rocketmq.streams.common.metadata.MetaData;
+import org.apache.rocketmq.streams.common.metadata.MetaDataField;
+import org.apache.rocketmq.streams.common.utils.ReflectUtil;
+
+public class ContextVar<T> extends Var<T> {
+
+ private static final long serialVersionUID = -7025012350292140132L;
+ private String fieldName;
+ private String metaDataName; // 消息队列对应的meta信息
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public T doAction(RuleContext context, Rule rule) {
+ if (!volidate(context, rule)) {
+ return null;
+ }
+ JSONObject message = context.getMessage().getMessageBody();
+ Object object = null;
+ MetaData metaData = context.getMetaData(metaDataName);
+ if (metaData == null) {
+ object = message.get(fieldName);
+ if (object == null) {
+ object = ReflectUtil.getBeanFieldOrJsonValue(message, String.class, fieldName);
+ }
+ return (T)object;
+ }
+ Class dataClass = String.class;
+ MetaDataField field = metaData.getMetaDataField(fieldName);
+ if (field != null) {
+ dataClass = field.getDataType().getDataClass();
+ }
+ object = ReflectUtil.getBeanFieldOrJsonValue(message, dataClass, fieldName);
+ if (object == null) {
+ return null;
+ }
+ return (T)object;
+ }
+
+ @Override
+ public boolean volidate(RuleContext context, Rule rule) {
+ return true;
+ }
+
+ public String getFieldName() {
+ return fieldName;
+ }
+
+ public void setFieldName(String fieldName) {
+ this.fieldName = fieldName;
+ }
+
+ public String getMetaDataName() {
+ return metaDataName;
+ }
+
+ public void setMetaDataName(String metaDataName) {
+ this.metaDataName = metaDataName;
+ }
+
+ @Override
+ public boolean canLazyLoad() {
+ return false;
+ }
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/var/InnerVar.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/var/InnerVar.java
new file mode 100644
index 0000000..514917b
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/var/InnerVar.java
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.operator.var;
+
+import com.alibaba.fastjson.JSONObject;
+import org.apache.rocketmq.streams.filter.context.RuleContext;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+
+@SuppressWarnings("rawtypes")
+public class InnerVar extends Var {
+
+ private static final long serialVersionUID = -166963014761276615L;
+ public static final String ORIG_MESSAGE = "inner_message";
+ public static final String RULE = "inner_rule";
+ public static final String INNER_RULE_CODE = "inner_rule_code";
+ public static final String INNER_RULE_DESC = "inner_rule_desc";
+ public static final String INNER_RULE_TITEL = "inner_rule_title";
+ public static final String INNER_NAMESPACE = "inner_namespace";
+
+ @Override
+ public Object doAction(RuleContext context, Rule rule) {
+ String varName = getVarName();
+ if (varName.equals(ORIG_MESSAGE)) {
+ return context.getMessage().getMessageBody().toJSONString();
+ } else if (varName.equals(RULE)) {
+ return rule.toJson();
+ } else if (varName.equals(INNER_RULE_CODE)) {
+ return rule.getRuleCode();
+ } else if (varName.equals(INNER_NAMESPACE)) {
+ return context.getNameSpace();
+ } else if (varName.equals(INNER_RULE_TITEL)) {
+ return rule.getRuleTitle();
+ } else if (varName.equals(INNER_RULE_DESC)) {
+ return rule.getRuleDesc();
+ }
+ return null;
+ }
+
+ @Override
+ public boolean canLazyLoad() {
+ return false;
+ }
+
+ @Override
+ public boolean volidate(RuleContext context, Rule rule) {
+ return true;
+ }
+
+ public static void main(String args[]) {
+
+ // String cmdline="\"C:\\Windows\\SysWOW64\\cmd.exe\" /c REG add
+ // HKLM\\Software\\Microsoft\\Windows\\CurrentVersion\\Run /v svchost /t REG_SZ /d
+ // \"C:\\Windows\\svchost.exe\"";
+
+ String message =
+ "{\"uid\":\"N/A\",\"filepath\":\"N/A\",\"sid\":\"-1\",\"ppid\":\"4080\",\"cmdline\":\"REG add "
+ + "HKLM\\\\Software\\\\Microsoft\\\\Windows\\\\CurrentVersion\\\\Run /v svchost /t REG_SZ /d "
+ + "\\\"C:\\\\Windows\\\\svchost.exe\\\" /f\",\"egroupid\":\"N/A\",\"groupname\":\"N/A\","
+ + "\"cwd\":\"N/A\",\"pid\":\"2144\",\"pcmdline\":\"\\\"C:\\\\Windows\\\\SysWOW64\\\\cmd.exe\\\" /c "
+ + "REG add HKLM\\\\Software\\\\Microsoft\\\\Windows\\\\CurrentVersion\\\\Run /v svchost /t REG_SZ /d "
+ + "\\\"C:\\\\Windows\\\\svchost.exe\\\" /f\",\"safe_mode\":\"N/A\",\"username\":\"N/A\","
+ + "\"time\":\"2018-02-26 04:18:20\",\"seq\":\"8136606\",\"tty\":\"N/A\",\"filename\":\"\","
+ + "\"groupid\":\"N/A\",\"uuid\":\"4c8a102a-52ad-480d-92f3-a9a2cce98773\",\"pfilename\":\"N/A\","
+ + "\"messageId\":\"310347\",\"euid\":\"N/A\"}";
+
+ JSONObject obj = JSONObject.parseObject(message);
+ // System.out.println(obj.getString("cmdline"));
+ System.out.println(obj.toJSONString());
+
+ }
+
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/var/Var.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/var/Var.java
new file mode 100644
index 0000000..e6e6480
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/operator/var/Var.java
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.operator.var;
+
+import java.io.Serializable;
+
+import org.apache.rocketmq.streams.filter.operator.action.IConfigurableAction;
+import org.apache.rocketmq.streams.filter.context.RuleContext;
+import org.apache.rocketmq.streams.filter.operator.Rule;
+import org.apache.rocketmq.streams.common.configurable.BasedConfigurable;
+import org.apache.rocketmq.streams.common.configurable.IConfigurable;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+public abstract class Var<T> extends BasedConfigurable implements IConfigurable, IConfigurableAction<T>, Serializable {
+
+ private static final Log LOG = LogFactory.getLog(Var.class);
+
+ public static final String TYPE = "var";
+
+ protected String varName;
+
+ public Var() {
+ setType(TYPE);
+ }
+
+ public String getVarName() {
+ return varName;
+ }
+
+ public void setVarName(String varName) {
+ this.varName = varName;
+ }
+
+ public Object getVarValue(RuleContext context, Rule rule) {
+ if (context.containsVarName(getVarName())) {
+ return context.getVarCacheValue(getVarName());
+ }
+ Object value = doMonitorAction(context, rule);
+ context.putVarValue(getNameSpace(), getVarName(), value);
+ return value;
+ }
+
+ protected Object doMonitorAction(RuleContext context, Rule rule) {
+ Object value = null;
+
+ try {
+ value = doAction(context, rule);
+ } catch (Exception e) {
+ LOG.error(
+ "Var doMonitorAction doAction error, var is: " + getVarName() + " ,rule is : " + rule.getRuleCode(), e);
+ return null;
+ }
+ return value;
+
+ }
+
+ @Override
+ public boolean init() {
+ return true;
+ }
+
+ public abstract boolean canLazyLoad();
+
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/optimization/EqualsExpressionOptimization.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/optimization/EqualsExpressionOptimization.java
new file mode 100644
index 0000000..d0e3301
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/optimization/EqualsExpressionOptimization.java
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.optimization;
+
+import org.apache.rocketmq.streams.filter.function.expression.Equals;
+import org.apache.rocketmq.streams.filter.operator.expression.Expression;
+import org.apache.rocketmq.streams.common.utils.DataTypeUtil;
+
+public class EqualsExpressionOptimization implements IExpressionOptimization {
+ @Override
+ public boolean support(Expression expression) {
+ return Equals.isEqualFunction(expression.getFunctionName()) && DataTypeUtil.isString(expression.getDataType()) && expression.getValue() != null;
+ }
+
+ @Override
+ public OptimizationExpression optimize(Expression expression) {
+ Object value = expression.getValue();
+ String regex = null;
+ if (value != null) {
+ regex = (String)value;
+ }
+ OptimizationExpression optimizationExpression = new OptimizationExpression(expression, regex);
+ return optimizationExpression;
+ }
+}
diff --git a/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/optimization/ExpressionOptimization.java b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/optimization/ExpressionOptimization.java
new file mode 100644
index 0000000..d75e66c
--- /dev/null
+++ b/rocketmq-streams-filter/src/main/java/org/apache/rocketmq/streams/filter/optimization/ExpressionOptimization.java
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.filter.optimization;
+
+import org.apache.rocketmq.streams.filter.builder.ExpressionBuilder;
+import org.apache.rocketmq.streams.filter.operator.expression.Expression;
+import org.apache.rocketmq.streams.filter.operator.expression.RelationExpression;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * 对于表达式解析，为了方便，会产生很多附加的关系，如（a,==,b)&((c,>,0)&&(d,>,1))，每一层括号都会产生一层关系。可以通过优化，降低关系个数 优化思路，是如果顶层关系是&，则子表达式也是&，则可以拉平，变成如下这种（a,==,b)&(c,>,0)&(d,>,1)
+ */
+public class ExpressionOptimization {
+ protected Expression rootExpression;//最外层的expression
+ //protected List<Expression> simpleExpressions;//不是关系的表达式
+ //protected List<RelationExpression> relationExpressions;;//关系表达式
+ protected Map<String, Expression> expressionMap = new HashMap<>();
+
... 2339 lines suppressed ...
[rocketmq-streams] 31/35: Remove .iml file
Posted by du...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
duhengforever pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/rocketmq-streams.git
commit d14febad4a1581b09cb0e546a8d469ed7161984e
Author: RongtongJin <ji...@mails.ucas.ac.cn>
AuthorDate: Mon Aug 2 20:07:28 2021 +0800
Remove .iml file
---
.../rocketmq-streams-channel-http.iml | 62 -------------------
.../rocketmq-streams-filter.iml | 72 ----------------------
.../rocketmq-streams-schedule.iml | 65 -------------------
.../rocketmq-streams-transport-minio.iml | 17 -----
4 files changed, 216 deletions(-)
diff --git a/rocketmq-streams-channel-http/rocketmq-streams-channel-http.iml b/rocketmq-streams-channel-http/rocketmq-streams-channel-http.iml
deleted file mode 100644
index 51a5150..0000000
--- a/rocketmq-streams-channel-http/rocketmq-streams-channel-http.iml
+++ /dev/null
@@ -1,62 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<module org.jetbrains.idea.maven.project.MavenProjectsManager.isMavenModule="true" type="JAVA_MODULE" version="4">
- <component name="NewModuleRootManager" LANGUAGE_LEVEL="JDK_1_8">
- <output url="file://$MODULE_DIR$/target/classes" />
- <output-test url="file://$MODULE_DIR$/target/test-classes" />
- <content url="file://$MODULE_DIR$">
- <sourceFolder url="file://$MODULE_DIR$/src/main/java" isTestSource="false" />
- <sourceFolder url="file://$MODULE_DIR$/src/test/java" isTestSource="true" />
- <excludeFolder url="file://$MODULE_DIR$/target" />
- </content>
- <orderEntry type="inheritedJdk" />
- <orderEntry type="sourceFolder" forTests="false" />
- <orderEntry type="module" module-name="rocketmq-streams-commons" />
- <orderEntry type="library" name="Maven: org.apache.commons:commons-lang3:3.11" level="project" />
- <orderEntry type="library" name="Maven: junit:junit:4.12" level="project" />
- <orderEntry type="library" name="Maven: org.hamcrest:hamcrest-core:1.3" level="project" />
- <orderEntry type="library" name="Maven: com.alibaba:fastjson:1.2.27" level="project" />
- <orderEntry type="library" name="Maven: commons-logging:commons-logging:1.1" level="project" />
- <orderEntry type="library" name="Maven: logkit:logkit:1.0.1" level="project" />
- <orderEntry type="library" name="Maven: avalon-framework:avalon-framework:4.1.3" level="project" />
- <orderEntry type="library" name="Maven: javax.servlet:servlet-api:2.3" level="project" />
- <orderEntry type="library" name="Maven: log4j:log4j:1.2.17" level="project" />
- <orderEntry type="library" name="Maven: com.google.code.gson:gson:2.8.5" level="project" />
- <orderEntry type="library" name="Maven: com.google.auto.service:auto-service:1.0-rc5" level="project" />
- <orderEntry type="library" name="Maven: com.google.auto.service:auto-service-annotations:1.0-rc5" level="project" />
- <orderEntry type="library" name="Maven: com.google.auto:auto-common:0.10" level="project" />
- <orderEntry type="library" name="Maven: com.google.guava:guava:25.1-jre" level="project" />
- <orderEntry type="library" name="Maven: com.google.code.findbugs:jsr305:3.0.2" level="project" />
- <orderEntry type="library" name="Maven: org.checkerframework:checker-qual:2.0.0" level="project" />
- <orderEntry type="library" name="Maven: com.google.errorprone:error_prone_annotations:2.1.3" level="project" />
- <orderEntry type="library" name="Maven: com.google.j2objc:j2objc-annotations:1.1" level="project" />
- <orderEntry type="library" name="Maven: org.codehaus.mojo:animal-sniffer-annotations:1.14" level="project" />
- <orderEntry type="library" name="Maven: com.lmax:disruptor:3.2.0" level="project" />
- <orderEntry type="library" name="Maven: com.gliwka.hyperscan:hyperscan:5.4.0-2.0.0" level="project" />
- <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:5.4.0-1.0.0" level="project" />
- <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:linux-x86_64:5.4.0-1.0.0" level="project" />
- <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:windows-x86_64:5.4.0-1.0.0" level="project" />
- <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:macosx-x86_64:5.4.0-1.0.0" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp-platform:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-arm:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-arm64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-x86:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-x86_64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:ios-arm64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:ios-x86_64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-armhf:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-arm64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-ppc64le:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-x86:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-x86_64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:macosx-x86_64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:windows-x86:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:windows-x86_64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: net.java.dev.jna:platform:3.5.2" level="project" />
- <orderEntry type="library" name="Maven: net.java.dev.jna:jna:3.5.2" level="project" />
- <orderEntry type="library" name="Maven: commons-io:commons-io:2.5" level="project" />
- <orderEntry type="library" name="Maven: org.apache.httpcomponents:httpclient:4.5.2" level="project" />
- <orderEntry type="library" name="Maven: org.apache.httpcomponents:httpcore:4.4.4" level="project" />
- <orderEntry type="library" name="Maven: commons-codec:commons-codec:1.9" level="project" />
- </component>
-</module>
\ No newline at end of file
diff --git a/rocketmq-streams-filter/rocketmq-streams-filter.iml b/rocketmq-streams-filter/rocketmq-streams-filter.iml
deleted file mode 100644
index 70f1781..0000000
--- a/rocketmq-streams-filter/rocketmq-streams-filter.iml
+++ /dev/null
@@ -1,72 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<module org.jetbrains.idea.maven.project.MavenProjectsManager.isMavenModule="true" type="JAVA_MODULE" version="4">
- <component name="NewModuleRootManager" LANGUAGE_LEVEL="JDK_1_8">
- <output url="file://$MODULE_DIR$/target/classes" />
- <output-test url="file://$MODULE_DIR$/target/test-classes" />
- <content url="file://$MODULE_DIR$">
- <sourceFolder url="file://$MODULE_DIR$/src/main/java" isTestSource="false" />
- <sourceFolder url="file://$MODULE_DIR$/src/test/java" isTestSource="true" />
- <sourceFolder url="file://$MODULE_DIR$/src/test/resources" type="java-test-resource" />
- <excludeFolder url="file://$MODULE_DIR$/target" />
- </content>
- <orderEntry type="inheritedJdk" />
- <orderEntry type="sourceFolder" forTests="false" />
- <orderEntry type="module" module-name="rocketmq-streams-script" />
- <orderEntry type="module" module-name="rocketmq-streams-commons" />
- <orderEntry type="library" name="Maven: org.apache.commons:commons-lang3:3.11" level="project" />
- <orderEntry type="library" name="Maven: junit:junit:4.12" level="project" />
- <orderEntry type="library" name="Maven: org.hamcrest:hamcrest-core:1.3" level="project" />
- <orderEntry type="library" name="Maven: com.alibaba:fastjson:1.2.27" level="project" />
- <orderEntry type="library" name="Maven: commons-logging:commons-logging:1.1" level="project" />
- <orderEntry type="library" name="Maven: logkit:logkit:1.0.1" level="project" />
- <orderEntry type="library" name="Maven: avalon-framework:avalon-framework:4.1.3" level="project" />
- <orderEntry type="library" name="Maven: javax.servlet:servlet-api:2.3" level="project" />
- <orderEntry type="library" name="Maven: commons-io:commons-io:2.5" level="project" />
- <orderEntry type="library" name="Maven: log4j:log4j:1.2.17" level="project" />
- <orderEntry type="library" name="Maven: com.google.code.gson:gson:2.8.5" level="project" />
- <orderEntry type="library" name="Maven: com.google.auto.service:auto-service:1.0-rc5" level="project" />
- <orderEntry type="library" name="Maven: com.google.auto.service:auto-service-annotations:1.0-rc5" level="project" />
- <orderEntry type="library" name="Maven: com.google.auto:auto-common:0.10" level="project" />
- <orderEntry type="library" name="Maven: com.google.guava:guava:25.1-jre" level="project" />
- <orderEntry type="library" name="Maven: com.google.code.findbugs:jsr305:3.0.2" level="project" />
- <orderEntry type="library" name="Maven: org.checkerframework:checker-qual:2.0.0" level="project" />
- <orderEntry type="library" name="Maven: com.google.errorprone:error_prone_annotations:2.1.3" level="project" />
- <orderEntry type="library" name="Maven: com.google.j2objc:j2objc-annotations:1.1" level="project" />
- <orderEntry type="library" name="Maven: org.codehaus.mojo:animal-sniffer-annotations:1.14" level="project" />
- <orderEntry type="library" name="Maven: com.lmax:disruptor:3.2.0" level="project" />
- <orderEntry type="library" name="Maven: com.gliwka.hyperscan:hyperscan:5.4.0-2.0.0" level="project" />
- <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:5.4.0-1.0.0" level="project" />
- <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:linux-x86_64:5.4.0-1.0.0" level="project" />
- <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:windows-x86_64:5.4.0-1.0.0" level="project" />
- <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:macosx-x86_64:5.4.0-1.0.0" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp-platform:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-arm:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-arm64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-x86:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-x86_64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:ios-arm64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:ios-x86_64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-armhf:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-arm64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-ppc64le:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-x86:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-x86_64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:macosx-x86_64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:windows-x86:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:windows-x86_64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: net.java.dev.jna:platform:3.5.2" level="project" />
- <orderEntry type="library" name="Maven: net.java.dev.jna:jna:3.5.2" level="project" />
- <orderEntry type="library" name="Maven: io.krakens:java-grok:0.1.9" level="project" />
- <orderEntry type="library" name="Maven: org.codehaus.groovy:groovy-all:2.1.8" level="project" />
- <orderEntry type="library" name="Maven: org.python:jython-standalone:2.7.0" level="project" />
- <orderEntry type="module" module-name="rocketmq-streams-db-operator" />
- <orderEntry type="module" module-name="rocketmq-streams-configurable" />
- <orderEntry type="module" module-name="rocketmq-streams-serviceloader" />
- <orderEntry type="library" name="Maven: org.springframework:spring-jdbc:3.2.13.RELEASE" level="project" />
- <orderEntry type="library" name="Maven: org.springframework:spring-beans:3.2.13.RELEASE" level="project" />
- <orderEntry type="library" name="Maven: org.springframework:spring-core:3.2.13.RELEASE" level="project" />
- <orderEntry type="library" name="Maven: org.springframework:spring-tx:3.2.13.RELEASE" level="project" />
- <orderEntry type="library" name="Maven: mysql:mysql-connector-java:5.1.40" level="project" />
- </component>
-</module>
\ No newline at end of file
diff --git a/rocketmq-streams-schedule/rocketmq-streams-schedule.iml b/rocketmq-streams-schedule/rocketmq-streams-schedule.iml
deleted file mode 100644
index b51d7e1..0000000
--- a/rocketmq-streams-schedule/rocketmq-streams-schedule.iml
+++ /dev/null
@@ -1,65 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<module org.jetbrains.idea.maven.project.MavenProjectsManager.isMavenModule="true" type="JAVA_MODULE" version="4">
- <component name="NewModuleRootManager" LANGUAGE_LEVEL="JDK_1_8">
- <output url="file://$MODULE_DIR$/target/classes" />
- <output-test url="file://$MODULE_DIR$/target/test-classes" />
- <content url="file://$MODULE_DIR$">
- <sourceFolder url="file://$MODULE_DIR$/src/main/java" isTestSource="false" />
- <sourceFolder url="file://$MODULE_DIR$/src/test/java" isTestSource="true" />
- <sourceFolder url="file://$MODULE_DIR$/src/test/resources" type="java-test-resource" />
- <excludeFolder url="file://$MODULE_DIR$/target" />
- </content>
- <orderEntry type="inheritedJdk" />
- <orderEntry type="sourceFolder" forTests="false" />
- <orderEntry type="library" name="Maven: org.quartz-scheduler:quartz:2.2.1" level="project" />
- <orderEntry type="library" name="Maven: c3p0:c3p0:0.9.1.1" level="project" />
- <orderEntry type="library" name="Maven: org.slf4j:slf4j-api:1.6.6" level="project" />
- <orderEntry type="library" name="Maven: org.quartz-scheduler:quartz-jobs:2.2.1" level="project" />
- <orderEntry type="module" module-name="rocketmq-streams-serviceloader" />
- <orderEntry type="module" module-name="rocketmq-streams-commons" />
- <orderEntry type="library" name="Maven: org.apache.commons:commons-lang3:3.11" level="project" />
- <orderEntry type="library" name="Maven: junit:junit:4.12" level="project" />
- <orderEntry type="library" name="Maven: org.hamcrest:hamcrest-core:1.3" level="project" />
- <orderEntry type="library" name="Maven: com.alibaba:fastjson:1.2.27" level="project" />
- <orderEntry type="library" name="Maven: commons-logging:commons-logging:1.1" level="project" />
- <orderEntry type="library" name="Maven: logkit:logkit:1.0.1" level="project" />
- <orderEntry type="library" name="Maven: avalon-framework:avalon-framework:4.1.3" level="project" />
- <orderEntry type="library" name="Maven: javax.servlet:servlet-api:2.3" level="project" />
- <orderEntry type="library" name="Maven: commons-io:commons-io:2.5" level="project" />
- <orderEntry type="library" name="Maven: log4j:log4j:1.2.17" level="project" />
- <orderEntry type="library" name="Maven: com.google.code.gson:gson:2.8.5" level="project" />
- <orderEntry type="library" name="Maven: com.google.auto.service:auto-service:1.0-rc5" level="project" />
- <orderEntry type="library" name="Maven: com.google.auto.service:auto-service-annotations:1.0-rc5" level="project" />
- <orderEntry type="library" name="Maven: com.google.auto:auto-common:0.10" level="project" />
- <orderEntry type="library" name="Maven: com.google.guava:guava:25.1-jre" level="project" />
- <orderEntry type="library" name="Maven: com.google.code.findbugs:jsr305:3.0.2" level="project" />
- <orderEntry type="library" name="Maven: org.checkerframework:checker-qual:2.0.0" level="project" />
- <orderEntry type="library" name="Maven: com.google.errorprone:error_prone_annotations:2.1.3" level="project" />
- <orderEntry type="library" name="Maven: com.google.j2objc:j2objc-annotations:1.1" level="project" />
- <orderEntry type="library" name="Maven: org.codehaus.mojo:animal-sniffer-annotations:1.14" level="project" />
- <orderEntry type="library" name="Maven: com.lmax:disruptor:3.2.0" level="project" />
- <orderEntry type="library" name="Maven: com.gliwka.hyperscan:hyperscan:5.4.0-2.0.0" level="project" />
- <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:5.4.0-1.0.0" level="project" />
- <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:linux-x86_64:5.4.0-1.0.0" level="project" />
- <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:windows-x86_64:5.4.0-1.0.0" level="project" />
- <orderEntry type="library" name="Maven: com.gliwka.hyperscan:native:macosx-x86_64:5.4.0-1.0.0" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp-platform:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-arm:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-arm64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-x86:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:android-x86_64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:ios-arm64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:ios-x86_64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-armhf:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-arm64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-ppc64le:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-x86:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:linux-x86_64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:macosx-x86_64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:windows-x86:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: org.bytedeco:javacpp:windows-x86_64:1.5.4" level="project" />
- <orderEntry type="library" name="Maven: net.java.dev.jna:platform:3.5.2" level="project" />
- <orderEntry type="library" name="Maven: net.java.dev.jna:jna:3.5.2" level="project" />
- </component>
-</module>
\ No newline at end of file
diff --git a/rocketmq-streams-transport-minio/rocketmq-streams-transport-minio.iml b/rocketmq-streams-transport-minio/rocketmq-streams-transport-minio.iml
deleted file mode 100644
index af11f77..0000000
--- a/rocketmq-streams-transport-minio/rocketmq-streams-transport-minio.iml
+++ /dev/null
@@ -1,17 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<module org.jetbrains.idea.maven.project.MavenProjectsManager.isMavenModule="true" type="JAVA_MODULE" version="4">
- <component name="NewModuleRootManager" LANGUAGE_LEVEL="JDK_1_5">
- <output url="file://$MODULE_DIR$/${project.build.directory}/classes" />
- <output-test url="file://$MODULE_DIR$/${project.build.directory}/test-classes" />
- <content url="file://$MODULE_DIR$">
- <sourceFolder url="file://$MODULE_DIR$/src/main/java" isTestSource="false" />
- <sourceFolder url="file://$MODULE_DIR$/src/test/java" isTestSource="true" />
- <sourceFolder url="file://$MODULE_DIR$/src/test/resources" type="java-test-resource" />
- <excludeFolder url="file://$MODULE_DIR$/${project.build.directory}/classes" />
- <excludeFolder url="file://$MODULE_DIR$/${project.build.directory}/test-classes" />
- <excludeFolder url="file://$MODULE_DIR$/target" />
- </content>
- <orderEntry type="inheritedJdk" />
- <orderEntry type="sourceFolder" forTests="false" />
- </component>
-</module>
\ No newline at end of file
[rocketmq-streams] 32/35: Update README.md
Posted by du...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
duhengforever pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/rocketmq-streams.git
commit 7981c2349689954fb3531dd5e33ec2088a6a3f31
Author: Heng Du <du...@apache.org>
AuthorDate: Mon Aug 2 20:47:11 2021 +0800
Update README.md
---
README.md | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/README.md b/README.md
index 92a9b2f..019ea63 100644
--- a/README.md
+++ b/README.md
@@ -13,7 +13,7 @@ import org.apache.rocketmq.streams.client.transform.DataStream;
DataStreamSource source=StreamBuilder.dataStream("namespace","pipeline");
source
- .fromFile("/Users/junjie.cheng/text.txt",false)
+ .fromFile("～/admin/data/text.txt",false)
.map(message->message)
.toPrint(1)
.start();
@@ -104,4 +104,4 @@ source
.toPrint(1)
.with(CheckpointStrategy.db("jdbc:mysql://XXXXX:3306/XXXXX","","",0L))
.start();
-```
\ No newline at end of file
+```
[rocketmq-streams] 34/35: add dbinit module,
it will create tables into user's database,
the first version can support MySql only
Posted by du...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
duhengforever pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/rocketmq-streams.git
commit 320904471e1b4c975404598dcf045beaa284f6cf
Author: xstorm1 <xs...@live.cn>
AuthorDate: Tue Aug 3 14:42:10 2021 +0800
add dbinit module, it will create tables into user's database, the first version can support MySql only
---
pom.xml | 2 +-
.../streams/common/configure/ConfigureFileKey.java | 1 +
.../rocketmq/streams/db/driver/orm/ORMUtil.java | 23 +++
rocketmq-streams-dbinit/pom.xml | 38 ++++
.../streams/dbinit/mysql/delegate/DBDelegate.java | 9 +
.../dbinit/mysql/delegate/DBDelegateFactory.java | 27 +++
.../streams/dbinit/mysql/delegate/DBType.java | 6 +
.../dbinit/mysql/delegate/MysqlDelegate.java | 48 +++++
.../src/main/resources/tables_mysql_innodb.sql | 199 +++++++++++++++++++++
9 files changed, 352 insertions(+), 1 deletion(-)
diff --git a/pom.xml b/pom.xml
index 3675792..2969816 100644
--- a/pom.xml
+++ b/pom.xml
@@ -31,13 +31,13 @@
<module>rocketmq-streams-dim</module>
<module>rocketmq-streams-transport-minio</module>
<module>rocketmq-streams-script</module>
- <module>rocketmq-streams-script-python</module>
<module>rocketmq-streams-configurable</module>
<module>rocketmq-streams-serviceloader</module>
<module>rocketmq-streams-filter</module>
<module>rocketmq-streams-schedule</module>
<module>rocketmq-streams-lease</module>
<module>rocketmq-streams-db-operator</module>
+ <module>rocketmq-streams-dbinit</module>
<module>rocketmq-streams-window</module>
<module>rocketmq-streams-clients</module>
<module>rocketmq-streams-channel-rocketmq</module>
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/configure/ConfigureFileKey.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/configure/ConfigureFileKey.java
index 316a3b2..e6a796d 100644
--- a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/configure/ConfigureFileKey.java
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/configure/ConfigureFileKey.java
@@ -27,6 +27,7 @@ public interface ConfigureFileKey {
/**
* 数据库url
*/
+ String DB_TYPE = "dipper.rds.jdbc.type";
String JDBC_URL = "dipper.rds.jdbc.url";
String JDBC_USERNAME = "dipper.rds.jdbc.username";
String JDBC_PASSWORD = "dipper.rds.jdbc.password";
diff --git a/rocketmq-streams-db-operator/src/main/java/org/apache/rocketmq/streams/db/driver/orm/ORMUtil.java b/rocketmq-streams-db-operator/src/main/java/org/apache/rocketmq/streams/db/driver/orm/ORMUtil.java
index 20529b0..670cb76 100644
--- a/rocketmq-streams-db-operator/src/main/java/org/apache/rocketmq/streams/db/driver/orm/ORMUtil.java
+++ b/rocketmq-streams-db-operator/src/main/java/org/apache/rocketmq/streams/db/driver/orm/ORMUtil.java
@@ -161,6 +161,29 @@ public class ORMUtil {
}
}
+
+ public static boolean executeSQL(String sql, Object paras, String driver, final String url, final String userName,
+ final String password) {
+ if (paras != null) {
+ sql = SQLUtil.parseIbatisSQL(paras, sql);
+ }
+ JDBCDriver dataSource = null;
+ try {
+ dataSource = DriverBuilder.createDriver(driver, url, userName, password);
+ dataSource.execute(sql);
+ return true;
+ } catch (Exception e) {
+ String errorMsg = ("execute sql error ,the sql is " + sql + ". the error msg is " + e.getMessage());
+ LOG.error(errorMsg);
+ e.printStackTrace();
+ throw new RuntimeException(errorMsg, e);
+ } finally {
+ if (dataSource != null) {
+ dataSource.destroy();
+ }
+ }
+ }
+
/**
* 把一个对象的字段拼接成where条件，如果字段值为null，不拼接
*
diff --git a/rocketmq-streams-dbinit/pom.xml b/rocketmq-streams-dbinit/pom.xml
new file mode 100644
index 0000000..3434616
--- /dev/null
+++ b/rocketmq-streams-dbinit/pom.xml
@@ -0,0 +1,38 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>rocketmq-streams</artifactId>
+ <groupId>org.apache.rocketmq</groupId>
+ <version>2.0.0-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+ <artifactId>rocketmq-streams-dbinit</artifactId>
+ <name>ROCKETMQ STREAMS :: dbinit</name>
+
+ <build>
+ <resources>
+ <resource>
+ <directory>src/main/resources</directory>
+
+ <includes>
+ <include>**/*.sql</include>
+ <include>**/*.properties</include>
+ </includes>
+
+
+ <filtering>true</filtering>
+ </resource>
+ </resources>
+ </build>
+
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams-db-operator</artifactId>
+ </dependency>
+ </dependencies>
+
+</project>
\ No newline at end of file
diff --git a/rocketmq-streams-dbinit/src/main/java/org/apache/rocketmq/streams/dbinit/mysql/delegate/DBDelegate.java b/rocketmq-streams-dbinit/src/main/java/org/apache/rocketmq/streams/dbinit/mysql/delegate/DBDelegate.java
new file mode 100644
index 0000000..2737fb5
--- /dev/null
+++ b/rocketmq-streams-dbinit/src/main/java/org/apache/rocketmq/streams/dbinit/mysql/delegate/DBDelegate.java
@@ -0,0 +1,9 @@
+package org.apache.rocketmq.streams.dbinit.mysql.delegate;
+
+public interface DBDelegate {
+
+ public void init(String driver, String url, String userName,
+ String password);
+
+ public void init();
+}
diff --git a/rocketmq-streams-dbinit/src/main/java/org/apache/rocketmq/streams/dbinit/mysql/delegate/DBDelegateFactory.java b/rocketmq-streams-dbinit/src/main/java/org/apache/rocketmq/streams/dbinit/mysql/delegate/DBDelegateFactory.java
new file mode 100644
index 0000000..3b02516
--- /dev/null
+++ b/rocketmq-streams-dbinit/src/main/java/org/apache/rocketmq/streams/dbinit/mysql/delegate/DBDelegateFactory.java
@@ -0,0 +1,27 @@
+package org.apache.rocketmq.streams.dbinit.mysql.delegate;
+
+import org.apache.rocketmq.streams.common.component.ComponentCreator;
+import org.apache.rocketmq.streams.common.configure.ConfigureFileKey;
+
+public class DBDelegateFactory {
+
+ public static DBDelegate getDelegate() {
+ String dbType = ComponentCreator.getProperties().getProperty(ConfigureFileKey.DB_TYPE);
+ if (dbType == null || "".equalsIgnoreCase(dbType)) {
+ dbType = DBType.DB_MYSQL;
+ }
+ if (DBType.DB_MYSQL.equalsIgnoreCase(dbType)) {
+ return new MysqlDelegate();
+ }
+
+ return new MysqlDelegate();
+ }
+
+ public static DBDelegate getDelegate(String dbType) {
+ if (DBType.DB_MYSQL.equalsIgnoreCase(dbType)) {
+ return new MysqlDelegate();
+ }
+
+ return new MysqlDelegate();
+ }
+}
diff --git a/rocketmq-streams-dbinit/src/main/java/org/apache/rocketmq/streams/dbinit/mysql/delegate/DBType.java b/rocketmq-streams-dbinit/src/main/java/org/apache/rocketmq/streams/dbinit/mysql/delegate/DBType.java
new file mode 100644
index 0000000..d8908f8
--- /dev/null
+++ b/rocketmq-streams-dbinit/src/main/java/org/apache/rocketmq/streams/dbinit/mysql/delegate/DBType.java
@@ -0,0 +1,6 @@
+package org.apache.rocketmq.streams.dbinit.mysql.delegate;
+
+public class DBType {
+
+ public static final String DB_MYSQL = "MYSQL";
+}
diff --git a/rocketmq-streams-dbinit/src/main/java/org/apache/rocketmq/streams/dbinit/mysql/delegate/MysqlDelegate.java b/rocketmq-streams-dbinit/src/main/java/org/apache/rocketmq/streams/dbinit/mysql/delegate/MysqlDelegate.java
new file mode 100644
index 0000000..1295368
--- /dev/null
+++ b/rocketmq-streams-dbinit/src/main/java/org/apache/rocketmq/streams/dbinit/mysql/delegate/MysqlDelegate.java
@@ -0,0 +1,48 @@
+package org.apache.rocketmq.streams.dbinit.mysql.delegate;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.rocketmq.streams.common.utils.FileUtil;
+import org.apache.rocketmq.streams.db.driver.orm.ORMUtil;
+
+import java.io.IOException;
+import java.net.URL;
+
+public class MysqlDelegate implements DBDelegate {
+
+ public static final Log LOG = LogFactory.getLog(MysqlDelegate.class);
+
+
+ @Override
+ public void init(String driver, final String url, final String userName,
+ final String password) {
+ String[] sqls = loadSqls();
+ for (String sql : sqls) {
+ ORMUtil.executeSQL(sql, null, driver, url, userName, password);
+ }
+ }
+
+ @Override
+ public void init() {
+ String[] sqls = loadSqls();
+ for (String sql : sqls) {
+ ORMUtil.executeSQL(sql, null);
+ }
+ }
+
+ private String[] loadSqls() {
+ String[] sqls = null;
+ URL url = this.getClass().getClassLoader().getResource("tables_mysql_innodb.sql");
+ try {
+ String tables = FileUtil.loadFileContent(url.openStream());
+ sqls = tables.split(";");
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Init db sqls : " + tables);
+ }
+ } catch (IOException e) {
+ e.printStackTrace();
+ }
+ return sqls;
+ }
+
+}
diff --git a/rocketmq-streams-dbinit/src/main/resources/tables_mysql_innodb.sql b/rocketmq-streams-dbinit/src/main/resources/tables_mysql_innodb.sql
new file mode 100644
index 0000000..dc5771f
--- /dev/null
+++ b/rocketmq-streams-dbinit/src/main/resources/tables_mysql_innodb.sql
@@ -0,0 +1,199 @@
+CREATE TABLE IF NOT EXISTS `window_max_value` (
+ `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
+ `gmt_create` datetime NOT NULL,
+ `gmt_modified` datetime NOT NULL,
+ `max_value` bigint(20) unsigned NOT NULL,
+ `max_event_time` bigint(20) unsigned NOT NULL,
+ `msg_key` varchar(256) NOT NULL,
+ PRIMARY KEY (`id`),
+ UNIQUE KEY `uk__ket` (`msg_key`(250)),
+ KEY `idx_modifytime` (`gmt_modified`)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+
+CREATE TABLE IF NOT EXISTS `window_value` (
+ `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
+ `gmt_create` datetime NOT NULL,
+ `gmt_modified` datetime NOT NULL,
+ `start_time` varchar(20) NOT NULL,
+ `end_time` varchar(20) NOT NULL,
+ `max_offset` longtext,
+ `group_by` text,
+ `agg_column_result` longtext,
+ `computed_column_result` longtext,
+ `version` varchar(64) DEFAULT NULL,
+ `name_space` varchar(256) DEFAULT NULL,
+ `configure_name` varchar(256) DEFAULT NULL,
+ `msg_key` varchar(64) NOT NULL,
+ `window_instance_id` varchar(64) NOT NULL,
+ `partition` varchar(512) DEFAULT NULL,
+ `partition_num` bigint(20) DEFAULT NULL,
+ `fire_time` varchar(20) DEFAULT NULL,
+ `update_version` bigint(20) unsigned DEFAULT NULL,
+ `update_flag` bigint(20) DEFAULT NULL,
+ `window_instance_partition_id` varchar(64) DEFAULT NULL,
+ `type` varchar(64) DEFAULT NULL,
+ PRIMARY KEY (`id`),
+ UNIQUE KEY `uk_window_state` (`msg_key`),
+ KEY `idx_window_instance_shuffle` (`window_instance_partition_id`,`partition_num`),
+ KEY `idx_window_instance_firetime` (`window_instance_partition_id`,`fire_time`),
+ KEY `idx_window` (`name_space`(128),`configure_name`(128),`partition`(128))
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+
+CREATE TABLE IF NOT EXISTS `window_task` (
+ `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
+ `gmt_create` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
+ `gmt_modified` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
+ `task_id` varchar(64) NOT NULL,
+ `untreated_flag` int(11) NOT NULL DEFAULT '0',
+ `group_by_value` varchar(1024) NOT NULL,
+ `task_owner` varchar(256) NOT NULL,
+ `task_send_time` datetime DEFAULT NULL,
+ `send_task_msg` text NOT NULL,
+ `msg_send_time` bigint(20) DEFAULT NULL,
+ `name` varchar(128) NOT NULL,
+ `start_time` varchar(20) NOT NULL,
+ `end_time` varchar(20) NOT NULL,
+ PRIMARY KEY (`id`),
+ UNIQUE KEY `uk_taskid` (`task_id`),
+ KEY `idx_flag_modifytime` (`name`,`untreated_flag`,`gmt_modified`)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+
+CREATE TABLE IF NOT EXISTS `window_instance` (
+ `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
+ `gmt_create` datetime NOT NULL,
+ `gmt_modified` datetime NOT NULL,
+ `start_time` varchar(20) NOT NULL,
+ `end_time` varchar(20) NOT NULL,
+ `fire_time` varchar(20) NOT NULL,
+ `window_name` varchar(128) NOT NULL,
+ `window_name_space` varchar(128) NOT NULL,
+ `status` tinyint(4) NOT NULL DEFAULT '0',
+ `version` int(11) DEFAULT '0',
+ `window_instance_key` varchar(128) DEFAULT NULL,
+ `window_instance_name` varchar(128) DEFAULT NULL,
+ `window_Instance_split_name` varchar(128) DEFAULT NULL,
+ `split_id` varchar(128) DEFAULT NULL,
+ PRIMARY KEY (`id`),
+ UNIQUE KEY `uk_window_instance_uniq_index` (`window_instance_key`),
+ KEY `idx_gmt_modified` (`fire_time`,`window_name`,`window_name_space`,`status`),
+ KEY `idx_windowinstance_name` (`window_instance_name`),
+ KEY `idx_windowinstance_split_name` (`window_Instance_split_name`),
+ KEY `idx_windowinstance_split_name_firetime` (`window_Instance_split_name`,`fire_time`)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+
+CREATE TABLE IF NOT EXISTS `lease_info` (
+ `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
+ `gmt_create` datetime NOT NULL,
+ `gmt_modified` datetime NOT NULL,
+ `lease_name` varchar(255) NOT NULL,
+ `lease_user_ip` varchar(255) NOT NULL,
+ `lease_end_time` varchar(255) NOT NULL,
+ `status` int(11) NOT NULL DEFAULT '1',
+ `version` bigint(20) NOT NULL,
+ `candidate_lease_ip` varchar(255) DEFAULT NULL,
+ PRIMARY KEY (`id`),
+ UNIQUE KEY `uk_name` (`lease_name`)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+
+CREATE TABLE IF NOT EXISTS `dipper_sql_configure` (
+ `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
+ `gmt_create` datetime NOT NULL,
+ `gmt_modified` datetime NOT NULL,
+ `namespace` varchar(32) NOT NULL,
+ `type` varchar(32) NOT NULL,
+ `name` varchar(128) NOT NULL,
+ `json_value` longtext NOT NULL,
+ `request_id` varchar(128) NOT NULL,
+ `account_id` varchar(32) NOT NULL,
+ `account_name` varchar(32) NOT NULL,
+ `account_nickname` varchar(32) NOT NULL,
+ `client_ip` varchar(64) NOT NULL,
+ `status` tinyint(3) unsigned NOT NULL DEFAULT '0',
+ `is_publish` int(11) NOT NULL DEFAULT '0',
+ PRIMARY KEY (`id`),
+ UNIQUE KEY `uk_namespace_type_name` (`namespace`,`type`,`name`),
+ KEY `idx_namespace` (`namespace`)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+
+CREATE TABLE IF NOT EXISTS `dipper_configure` (
+ `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
+ `gmt_create` datetime NOT NULL,
+ `gmt_modified` datetime NOT NULL,
+ `namespace` varchar(32) NOT NULL,
+ `type` varchar(32) NOT NULL,
+ `name` varchar(128) NOT NULL,
+ `json_value` text NOT NULL,
+ `request_id` varchar(128) NOT NULL,
+ `account_id` varchar(32) NOT NULL,
+ `account_name` varchar(32) NOT NULL,
+ `account_nickname` varchar(32) NOT NULL,
+ `client_ip` varchar(64) NOT NULL,
+ `status` tinyint(3) unsigned NOT NULL DEFAULT '0',
+ `isPublish` int(1) NOT NULL DEFAULT '0',
+ PRIMARY KEY (`id`),
+ UNIQUE KEY `uk_namespace_type_name` (`namespace`,`type`,`name`),
+ KEY `idx_namespace` (`namespace`)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+
+CREATE TABLE IF NOT EXISTS `join_right_state` (
+ `id` bigint(20) NOT NULL AUTO_INCREMENT,
+ `gmt_create` datetime DEFAULT NULL,
+ `gmt_modified` datetime DEFAULT NULL,
+ `window_id` bigint(20) DEFAULT NULL,
+ `window_name` varchar(200) DEFAULT NULL,
+ `window_name_space` varchar(45) DEFAULT NULL,
+ `message_id` varchar(200) DEFAULT NULL,
+ `message_key` varchar(32) DEFAULT NULL,
+ `message_time` datetime DEFAULT NULL,
+ `message_body` longtext,
+ `msg_key` varchar(400) DEFAULT NULL,
+ `window_instance_id` varchar(200) DEFAULT NULL,
+ `partition` varchar(200) DEFAULT NULL,
+ `partition_num` bigint(20) DEFAULT NULL,
+ `window_instance_partition_id` varchar(200) DEFAULT NULL,
+ `version` varchar(64) DEFAULT NULL,
+ `update_flag` bigint(20) DEFAULT NULL,
+ `name_space` varchar(256) DEFAULT NULL,
+ `configure_name` varchar(256) DEFAULT NULL,
+ `type` varchar(64) DEFAULT NULL,
+ `name` varchar(64) DEFAULT NULL,
+ `update_version` bigint(20) unsigned DEFAULT NULL,
+ PRIMARY KEY (`id`),
+ UNIQUE KEY `uk_message_id_unique` (`message_id`),
+ KEY `idx_message_key_index` (`message_key`),
+ KEY `idx_gmt_create_index` (`gmt_create`),
+ KEY `idx_window_name_index` (`window_name`(70)),
+ KEY `idx_message_key_gmt_create_index` (`message_key`,`gmt_create`)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+
+CREATE TABLE IF NOT EXISTS `join_left_state` (
+ `id` bigint(20) NOT NULL AUTO_INCREMENT,
+ `gmt_create` datetime DEFAULT NULL,
+ `gmt_modified` datetime DEFAULT NULL,
+ `window_id` bigint(20) DEFAULT NULL,
+ `window_name` varchar(200) DEFAULT NULL,
+ `window_name_space` varchar(45) DEFAULT NULL,
+ `message_id` varchar(200) DEFAULT NULL,
+ `message_key` varchar(32) DEFAULT NULL,
+ `message_time` datetime DEFAULT NULL,
+ `message_body` longtext,
+ `msg_key` varchar(400) DEFAULT NULL,
+ `window_instance_id` varchar(200) DEFAULT NULL,
+ `partition` varchar(200) DEFAULT NULL,
+ `partition_num` bigint(20) DEFAULT NULL,
+ `window_instance_partition_id` varchar(200) DEFAULT NULL,
+ `version` varchar(64) DEFAULT NULL,
+ `update_flag` bigint(20) DEFAULT NULL,
+ `name_space` varchar(256) DEFAULT NULL,
+ `configure_name` varchar(256) DEFAULT NULL,
+ `type` varchar(64) DEFAULT NULL,
+ `name` varchar(64) DEFAULT NULL,
+ `update_version` bigint(20) unsigned DEFAULT NULL,
+ PRIMARY KEY (`id`),
+ UNIQUE KEY `uk_message_id_unique` (`message_id`),
+ KEY `idx_message_key_index` (`message_key`),
+ KEY `idx_gmt_create_index` (`gmt_create`),
+ KEY `idx_window_name_index` (`window_name`(70)),
+ KEY `idx_message_key_gmt_create_index` (`message_key`,`gmt_create`)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+
[rocketmq-streams] 11/35: modify README.md
Posted by du...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
duhengforever pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/rocketmq-streams.git
commit b013eb2b5c9d2c26d53769192f34488a2d372544
Author: xstorm1 <xs...@live.cn>
AuthorDate: Mon Aug 2 12:15:15 2021 +0800
modify README.md
---
README.md | 1 +
1 file changed, 1 insertion(+)
diff --git a/README.md b/README.md
index f138e6e..678bd5a 100644
--- a/README.md
+++ b/README.md
@@ -1,5 +1,6 @@
# Rocketmq Streams
+
## Features
* 轻量级部署：可以单独部署，也支持集群部署
[rocketmq-streams] 33/35: Merge branch 'apache:main' into main
Posted by du...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
duhengforever pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/rocketmq-streams.git
commit 480b1cddc45be6a91cda5d943aa3e4ecc9dd7f68
Merge: 4ce3b3b 7981c23
Author: xstorm1 <xs...@live.cn>
AuthorDate: Tue Aug 3 10:57:52 2021 +0800
Merge branch 'apache:main' into main
README.md | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
[rocketmq-streams] 07/35: modify README.md
Posted by du...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
duhengforever pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/rocketmq-streams.git
commit 2fb7d91c110e1f58ffd28c50049efd1b746653de
Author: xstorm1 <yu...@alibaba-inc.com>
AuthorDate: Mon Aug 2 12:04:25 2021 +0800
modify README.md
---
README.md | 1 -
1 file changed, 1 deletion(-)
diff --git a/README.md b/README.md
index 678bd5a..f138e6e 100644
--- a/README.md
+++ b/README.md
@@ -1,6 +1,5 @@
# Rocketmq Streams
-
## Features
* 轻量级部署：可以单独部署，也支持集群部署
[rocketmq-streams] 22/35: modify README.md
Posted by du...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
duhengforever pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/rocketmq-streams.git
commit d8216f0a8ee72e34a2a6ecd1d67c36c4bf5696dd
Merge: b013eb2 607418a
Author: xstorm1 <xs...@live.cn>
AuthorDate: Mon Aug 2 15:43:16 2021 +0800
modify README.md
.gitignore | 25 ++
README.md | 55 ++-
pom.xml | 410 +++++++++++++++++
rocketmq-streams-clients/pom.xml | 46 ++
.../rocketmq/streams/client/DataStreamAction.java | 101 +++++
.../rocketmq/streams/client/StreamBuilder.java | 28 ++
.../streams/client/source/DataStreamSource.java | 76 ++++
.../client/strategy/CheckpointStrategy.java | 69 +++
.../streams/client/strategy/StateStrategy.java | 37 ++
.../rocketmq/streams/client/strategy/Strategy.java | 25 ++
.../streams/client/transform/DataStream.java | 437 ++++++++++++++++++
.../streams/client/transform/JoinStream.java | 212 +++++++++
.../streams/client/transform/SplitStream.java | 61 +++
.../streams/client/transform/WindowStream.java | 210 +++++++++
.../client/transform/window/HoppingWindow.java | 32 ++
.../client/transform/window/SessionWindow.java | 32 ++
.../streams/client/transform/window/Time.java | 45 ++
.../client/transform/window/TumblingWindow.java | 33 ++
.../client/transform/window/WindowInfo.java | 83 ++++
.../rocketmq/streams/client/DBDriverTest.java | 75 ++++
.../rocketmq/streams/client/DataStreamTest.java | 107 +++++
.../apache/rocketmq/streams/client/FilterTest.java | 49 +++
.../apache/rocketmq/streams/client/JoinTest.java | 89 ++++
.../apache/rocketmq/streams/client/LeaseTest.java | 98 +++++
.../rocketmq/streams/client/ORMUtilTest.java | 172 ++++++++
.../apache/rocketmq/streams/client/SplitTest.java | 86 ++++
.../apache/rocketmq/streams/client/UnionTest.java | 82 ++++
.../apache/rocketmq/streams/client/WindowTest.java | 86 ++++
.../client/windows/AbstractWindowFireModeTest.java | 189 ++++++++
.../streams/client/windows/WindowFromFileTest.java | 158 +++++++
.../streams/client/windows/WindowFromMetaq.java | 47 ++
.../client/windows/WindowHighAvailabilityTest.java | 131 ++++++
.../src/test/resources/log4j.xml | 36 ++
rocketmq-streams-db-operator/pom.xml | 34 ++
.../rocketmq-streams-db-operator.iml | 16 +
.../streams/db/configuable/DBConfigureService.java | 282 ++++++++++++
.../DBSupportParentConfigureService.java | 37 ++
.../rocketmq/streams/db/driver/DriverBuilder.java | 111 +++++
.../rocketmq/streams/db/driver/IDriverBudiler.java | 36 ++
.../rocketmq/streams/db/driver/JDBCDriver.java | 277 ++++++++++++
.../db/driver/batchloader/BatchRowLoader.java | 179 ++++++++
.../db/driver/batchloader/IRowOperator.java | 33 ++
.../rocketmq/streams/db/driver/orm/ORMUtil.java | 490 +++++++++++++++++++++
.../rocketmq/streams/db/operator/SQLOperator.java | 178 ++++++++
.../org/apache/rocketmq/streams/db/Person.java | 110 +++++
.../DBSupportParentConfigureServiceTest.java | 74 ++++
.../streams/db/driver/orm/ORMUtilTest.java | 86 ++++
rocketmq-streams-dim/pom.xml | 47 ++
.../apache/rocketmq/streams/dim/DimComponent.java | 63 +++
.../rocketmq/streams/dim/builder/DimBuilder.java | 94 ++++
.../function/expression/InExpressionResource.java | 80 ++++
.../expression/NotInExpressionResource.java | 45 ++
.../dim/function/script/IntelligenceFunction.java | 81 ++++
.../script/IntelligenceNameListFunction.java | 24 +
.../dim/function/script/NameListFunction.java | 203 +++++++++
.../rocketmq/streams/dim/index/DimIndex.java | 319 ++++++++++++++
.../rocketmq/streams/dim/index/IndexExecutor.java | 258 +++++++++++
.../intelligence/AbstractIntelligenceCache.java | 395 +++++++++++++++++
.../dim/intelligence/AccountIntelligenceCache.java | 77 ++++
.../dim/intelligence/DomainIntelligenceCache.java | 83 ++++
.../dim/intelligence/IPIntelligenceCache.java | 108 +++++
.../dim/intelligence/URLIntelligenceCache.java | 80 ++++
.../rocketmq/streams/dim/model/AbstractDim.java | 312 +++++++++++++
.../streams/dim/model/BooleanFieldDBDim.java | 55 +++
.../apache/rocketmq/streams/dim/model/DBDim.java | 140 ++++++
.../rocketmq/streams/dim/service/IDimService.java | 65 +++
.../streams/dim/service/impl/DimServiceImpl.java | 92 ++++
.../com/aliyun/service/ConfigureLoaderTest.java | 37 ++
.../com/aliyun/service/ExpressionExecutorTest.java | 80 ++++
.../java/com/aliyun/service/JsonParserTest.java | 40 ++
.../com/aliyun/service/NameListFunctionTest.java | 90 ++++
.../java/com/aliyun/service/TableCompressTest.java | 26 ++
rocketmq-streams-lease/pom.xml | 25 ++
.../rocketmq/streams/lease/LeaseComponent.java | 103 +++++
.../rocketmq/streams/lease/model/LeaseInfo.java | 127 ++++++
.../streams/lease/service/ILeaseGetCallback.java | 30 ++
.../streams/lease/service/ILeaseService.java | 136 ++++++
.../streams/lease/service/ILeaseStorage.java | 73 +++
.../streams/lease/service/ILeaseStorasge.java | 63 +++
.../lease/service/impl/BasedLesaseImpl.java | 404 +++++++++++++++++
.../lease/service/impl/LeaseServiceImpl.java | 275 ++++++++++++
.../streams/lease/service/impl/MockLeaseImpl.java | 95 ++++
.../lease/service/storages/DBLeaseStorage.java | 229 ++++++++++
.../rocketmq/streams/lease/LeaseComponentTest.java | 119 +++++
.../src/test/resources/log4j.xml | 20 +
rocketmq-streams-transport-minio/pom.xml | 25 ++
.../rocketmq-streams-transport-minio.iml | 17 +
.../transport/minio/MinioFileTransport.java | 141 ++++++
.../yundun/dipper/configurable/DataTpyeTest.java | 70 +++
.../streams/configuable/model/DataTpyeTest.java | 68 +++
.../rocketmq/streams/configuable/model/Person.java | 97 ++++
.../streams/configurable/model/Person.java | 97 ++++
.../component/ConfigurableComponent.properties | 7 +
.../src/test/resources/log4j.xml | 20 +
.../src/test/resources/pro-function.txt | 11 +
.../src/test/resources/python_script.py | 22 +
96 files changed, 10532 insertions(+), 1 deletion(-)
diff --cc README.md
index 678bd5a,92a9b2f..d2ff23e
--- a/README.md
+++ b/README.md
@@@ -9,17 -8,15 +8,29 @@@
## DataStream Example
```java
++<<<<<<< HEAD
+ import org.apache.rocketmq.streams.client.transform.DataStream;
+
+
+
+ DataStreamSource source=StreamBuilder.dataStream("namespace","pipeline");
+
+ source
+ .fromFile("/Users/junjie.cheng/text.txt",false)
+ .map(message->message)
+ .toPrint(1)
+ .start();
++=======
+ import org.apache.rocketmq.streams.client.transform.DataStream;
+
+ DataStreamSource source=StreamBuilder.dataStream("namespace","pipeline");
+
+ source
+ .fromFile("/Users/junjie.cheng/text.txt",false)
+ .map(message->message)
+ .toPrint(1)
+ .start();
++>>>>>>> main
```
## Maven Repository
@@@ -47,6 -44,7 +58,10 @@@ StreamBuilder 用于构建流任务的�
## DataStream API
### Source
++<<<<<<< HEAD
++=======
+
++>>>>>>> main
DataStreamSource 是分段式编程的源头类，用于对接各种数据源， 从各大消息队列中获取数据；
+ ```fromFile``` 从文件中读取数据， 该方法包含俩个参数
@@@ -60,13 -58,14 +75,24 @@@
+ ```isJson``` 是否json格式，非必填参数
+ ```tags``` rocketmq消费的tags值，用于过滤消息，非必填参数
++<<<<<<< HEAD
+
++ ```from``` 自定义的数据源， 通过实现ISource接口实现自己的数据源
+
+### transform
+transform 允许在流计算过程中对输入源的数据进行修改，进行下一步的操作；DataStream API中包括```DataStream```,```JoinStream```, ```SplitStream```,```WindowStream```等多个transform类；
+
+#### DataStream
++=======
+ + ```from``` 自定义的数据源， 通过实现ISource接口实现自己的数据源
+
+ ### transform
+
+ transform 允许在流计算过程中对输入源的数据进行修改，进行下一步的操作；DataStream API中包括```DataStream```,```JoinStream```, ```SplitStream```,```WindowStream```等多个transform类；
+
+ #### DataStream
+
++>>>>>>> main
DataStream实现了一系列常见的流计算算子
+ ```map``` 通过将源的每个记录传递给函数func来返回一个新的DataStream
@@@ -80,30 -79,29 +106,57 @@@
+ ```toFile``` 将结果保存为文件，生成一个新的DataStreamAction实例
+ ```toDB``` 将结果保存到数据库
+ ```toRocketmq``` 将结果输出到rocketmq
++<<<<<<< HEAD
++ ```toSls``` 将结果输出到sls
++ ```to``` 将结果经过自定义的ISink接口输出到指定的存储
++ ```window``` 在窗口内进行相关的统计分析，一般会与```groupBy```连用， ```window()```用来定义窗口的大小， ```groupBy()```用来定义统计分析的主key，可以指定多个
+ + ```count``` 在窗口内计数
+ + ```min``` 获取窗口内统计值的最小值
+ + ```max``` 获取窗口内统计值得最大值
+ + ```avg``` 获取窗口内统计值的平均值
+ + ```sum``` 获取窗口内统计值的加和值
+ + ```reduce``` 在窗口内进行自定义的汇总运算
++=======
+ + ```to``` 将结果经过自定义的ISink接口输出到指定的存储
+ + ```window``` 在窗口内进行相关的统计分析，一般会与```groupBy```连用， ```window()```用来定义窗口的大小， ```groupBy()```用来定义统计分析的主key，可以指定多个
+ + ```count``` 在窗口内计数
+ + ```min``` 获取窗口内统计值的最小值
+ + ```max``` 获取窗口内统计值得最大值
+ + ```avg``` 获取窗口内统计值的平均值
+ + ```sum``` 获取窗口内统计值的加和值
+ + ```reduce``` 在窗口内进行自定义的汇总运算
++>>>>>>> main
+ ```join``` 根据条件将将俩个流进行关联， 合并为一个大流进行相关的运算
+ ```union``` 将俩个流进行合并
+ ```split``` 将一个数据流按照标签进行拆分，分为不同的数据流供下游进行分析计算
+ ```with``` with算子用来指定计算过程中的相关策略，包括checkpoint的存储策略，state的存储策略等
++<<<<<<< HEAD
+
+# Strategy
+策略机制主要用来控制计算引擎运行过程中的底层逻辑，如checkpoint，state的存储方式等，后续还会增加对窗口、双流join等的控制；所有的控制策略通过```with```算子传入，可以同时传入多个策略类型；
+
+```java
+ //指定checkpoint的存储策略
+ source
+ .fromRocketmq("TSG_META_INFO", "")
+ .map(message -> message + "--")
+ .toPrint(1)
+ .with(CheckpointStrategy.db("jdbc:mysql://XXXXX:3306/XXXXX", "", "", 0L))
+ .start();
+```
++=======
+ # Strategy
+
+ 策略机制主要用来控制计算引擎运行过程中的底层逻辑，如checkpoint，state的存储方式等，后续还会增加对窗口、双流join等的控制；所有的控制策略通过```with```算子传入，可以同时传入多个策略类型；
+
+ ```java
+ //指定checkpoint的存储策略
+ source
+ .fromRocketmq("TSG_META_INFO","")
+ .map(message->message+"--")
+ .toPrint(1)
+ .with(CheckpointStrategy.db("jdbc:mysql://XXXXX:3306/XXXXX","","",0L))
+ .start();
-```
++```
++>>>>>>> main
[rocketmq-streams] 06/35: add channel-db module
Posted by du...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
duhengforever pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/rocketmq-streams.git
commit ed97ca3daee8bd3e71c4c5150a54381867a07756
Author: vv <ze...@alibaba-inc.com>
AuthorDate: Mon Aug 2 12:03:59 2021 +0800
add channel-db module
---
rocketmq-streams-channel-db/pom.xml | 21 ++
.../streams/db/sink/AbstractMultiTableSink.java | 150 +++++++++++++
.../apache/rocketmq/streams/db/sink/DBSink.java | 239 +++++++++++++++++++++
.../rocketmq/streams/db/sink/DBSinkBuilder.java | 76 +++++++
.../streams/db/sink/SelfMultiTableSink.java | 53 +++++
.../streams/db/sink/SplitBySerialNumber.java | 36 ++++
.../streams/db/sink/SplitByTimeMultiTableSink.java | 36 ++++
.../streams/db/sink/db/DBWriteOnlyChannelTest.java | 84 ++++++++
8 files changed, 695 insertions(+)
diff --git a/rocketmq-streams-channel-db/pom.xml b/rocketmq-streams-channel-db/pom.xml
new file mode 100755
index 0000000..10d760e
--- /dev/null
+++ b/rocketmq-streams-channel-db/pom.xml
@@ -0,0 +1,21 @@
+<?xml version="1.0" encoding="utf-8"?>
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams</artifactId>
+ <version>2.0.0-SNAPSHOT</version>
+ </parent>
+ <artifactId>rocketmq-streams-channel-db</artifactId>
+ <name>ROCKETMQ STREAMS :: channel-db</name>
+ <packaging>jar</packaging>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams-db-operator</artifactId>
+ </dependency>
+ </dependencies>
+</project>
diff --git a/rocketmq-streams-channel-db/src/main/java/org/apache/rocketmq/streams/db/sink/AbstractMultiTableSink.java b/rocketmq-streams-channel-db/src/main/java/org/apache/rocketmq/streams/db/sink/AbstractMultiTableSink.java
new file mode 100644
index 0000000..6547615
--- /dev/null
+++ b/rocketmq-streams-channel-db/src/main/java/org/apache/rocketmq/streams/db/sink/AbstractMultiTableSink.java
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.db.sink;
+
+import org.apache.rocketmq.streams.common.channel.sinkcache.IMessageFlushCallBack;
+import org.apache.rocketmq.streams.common.channel.sinkcache.impl.MessageCache;
+import org.apache.rocketmq.streams.common.channel.split.ISplit;
+import org.apache.rocketmq.streams.common.context.IMessage;
+import org.apache.rocketmq.streams.common.functions.MultiTableSplitFunction;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicLong;
+
+public abstract class AbstractMultiTableSink extends DBSink {
+ protected transient ConcurrentHashMap<String, DBSink> tableSinks = new ConcurrentHashMap();
+ protected transient AtomicLong messageCount = new AtomicLong(0);
+ protected transient MultiTableSplitFunction<IMessage> multiTableSplitFunction;
+
+ public AbstractMultiTableSink(String url, String userName, String password) {
+ this.url = url;
+ this.userName = userName;
+ this.password = password;
+ }
+
+ @Override
+ public boolean batchAdd(IMessage message, ISplit split) {
+
+ DBSink sink = getOrCreateDBSink(split.getQueueId());
+ boolean success = sink.batchAdd(message, split);
+ long count = messageCount.incrementAndGet();
+ if (count >= getBatchSize()) {
+ Set<String> queueIds = new HashSet<>();
+ queueIds.add(split.getQueueId());
+ flush(queueIds);
+ }
+ return success;
+ }
+
+ @Override
+ public boolean batchAdd(IMessage message) {
+ ISplit split = getSplitFromMessage(message);
+ return batchAdd(message, split);
+ }
+
+ @Override
+ public boolean batchSave(List<IMessage> messages) {
+ throw new RuntimeException("can not support this method");
+ }
+
+ @Override
+ public boolean flush(Set<String> splitIds) {
+ if (splitIds == null) {
+ return true;
+ }
+ for (String splitId : splitIds) {
+ DBSink sink = getOrCreateDBSink(splitId);
+ sink.flush();
+ }
+ return true;
+ }
+
+ @Override
+ public boolean flush() {
+ for (DBSink dbSink : tableSinks.values()) {
+ dbSink.flush();
+ }
+ return true;
+ }
+
+ @Override
+ public void openAutoFlush() {
+ for (DBSink dbSink : tableSinks.values()) {
+ dbSink.openAutoFlush();
+ }
+ }
+
+ @Override
+ public void closeAutoFlush() {
+ for (DBSink dbSink : tableSinks.values()) {
+ dbSink.closeAutoFlush();
+ }
+ }
+
+ protected DBSink getOrCreateDBSink(String splitId) {
+ DBSink sink = this.tableSinks.get(splitId);
+ if (sink != null) {
+ return sink;
+ }
+ sink = new DBSink();
+ sink.setUrl(url);
+ sink.setPassword(password);
+ sink.setUserName(userName);
+ sink.setTableName(createTableName(splitId));
+ sink.openAutoFlush();
+ sink.setBatchSize(batchSize);
+ sink.setJdbcDriver(this.jdbcDriver);
+ sink.setMessageCache(new SingleDBSinkCache(sink));
+ sink.init();
+ DBSink existDBSink = this.tableSinks.putIfAbsent(splitId, sink);
+ if (existDBSink != null) {
+ return existDBSink;
+ }
+
+ return sink;
+ }
+
+ protected abstract String createTableName(String splitId);
+
+ protected abstract ISplit getSplitFromMessage(IMessage message);
+
+ protected class SingleDBSinkCache extends MessageCache<IMessage> {
+
+ public SingleDBSinkCache(
+ IMessageFlushCallBack<IMessage> flushCallBack) {
+ super(flushCallBack);
+ }
+
+ @Override
+ public int flush(Set<String> splitIds) {
+ int size = super.flush(splitIds);
+ messageCount.addAndGet(-size);
+ return size;
+ }
+
+ @Override
+ public int flush() {
+ int size = super.flush();
+ messageCount.addAndGet(-size);
+ return size;
+ }
+ }
+
+}
diff --git a/rocketmq-streams-channel-db/src/main/java/org/apache/rocketmq/streams/db/sink/DBSink.java b/rocketmq-streams-channel-db/src/main/java/org/apache/rocketmq/streams/db/sink/DBSink.java
new file mode 100644
index 0000000..5732424
--- /dev/null
+++ b/rocketmq-streams-channel-db/src/main/java/org/apache/rocketmq/streams/db/sink/DBSink.java
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.db.sink;
+
+import com.alibaba.fastjson.JSONObject;
+
+import org.apache.rocketmq.streams.common.channel.IChannel;
+import org.apache.rocketmq.streams.common.channel.sink.AbstractSink;
+import org.apache.rocketmq.streams.common.component.AbstractComponent;
+import org.apache.rocketmq.streams.common.configurable.annotation.ENVDependence;
+import org.apache.rocketmq.streams.common.context.IMessage;
+import org.apache.rocketmq.streams.common.metadata.MetaData;
+import org.apache.rocketmq.streams.common.utils.SQLUtil;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+import org.apache.rocketmq.streams.db.driver.DriverBuilder;
+import org.apache.rocketmq.streams.db.driver.JDBCDriver;
+
+import java.sql.*;
+import java.util.List;
+
+/**
+ * 主要用于写db，输入可以是一个insert/replace 模版，也可以是metadata对象，二者选一即可。都支持批量插入，提高吞吐 sql 模版：insert into table(column1,column2,column3)values('#{var1}',#{var2},'#{var3}') MetaData:主要是描述每个字段的类型，是否必须 二者选一个即可。sql模式，系统会把一批（batchSize）数据拼成一个大sql。metadata模式，基于字段描述，最终也是拼成一个大sql
+ */
+public class DBSink extends AbstractSink {
+
+ protected String insertSQLTemplate;//完成插入部分的工作，和metadata二选一。insert into table(column1,column2,column3)values('#{var1}',#{var2},'#{var3}')
+
+ protected MetaData metaData;//可以指定meta data，和insertSQL二选一
+
+ protected String tableName; //指定要插入的数据表
+
+ @ENVDependence
+ protected String jdbcDriver = AbstractComponent.DEFAULT_JDBC_DRIVER;
+ @ENVDependence
+ protected String url;
+ @ENVDependence
+ protected String userName;
+ @ENVDependence
+ protected String password;
+
+ /**
+ * db串多数是名字，可以取个名字前缀，如果值为空，默认为此类的name，name为空，默认为简单类名
+ *
+ * @param insertSQL sql模版
+ * @param dbInfoNamePrefix 参数可以是名字，这个是名字前缀.真实值可以配置在配置文件中
+ */
+ public DBSink(String insertSQL, String dbInfoNamePrefix) {
+ setType(IChannel.TYPE);
+ if (StringUtil.isEmpty(dbInfoNamePrefix)) {
+ dbInfoNamePrefix = getConfigureName();
+ }
+ if (StringUtil.isEmpty(dbInfoNamePrefix)) {
+ dbInfoNamePrefix = this.getClass().getSimpleName();
+ }
+ this.insertSQLTemplate = insertSQL;
+ this.url = dbInfoNamePrefix + ".url";
+ this.password = dbInfoNamePrefix + ".password";
+ this.userName = dbInfoNamePrefix + ".userName";
+ }
+
+ public DBSink() {
+ setType(IChannel.TYPE);
+ }
+
+ public DBSink(String url, String userName, String password) {
+ setType(IChannel.TYPE);
+ this.url = url;
+ this.userName = userName;
+ this.password = password;
+ }
+
+ public DBSink(String insertSQL, String url, String userName, String password) {
+ setType(IChannel.TYPE);
+ this.url = url;
+ this.userName = userName;
+ this.password = password;
+ this.insertSQLTemplate = insertSQL;
+ }
+
+ @Override
+ protected boolean initConfigurable() {
+ try {
+ Class.forName("com.mysql.jdbc.Driver");
+ if (StringUtil.isNotEmpty(this.tableName)) {
+ Connection connection = DriverManager.getConnection(url, userName, password);
+ DatabaseMetaData metaData = connection.getMetaData();
+ ResultSet metaResult = metaData.getColumns(connection.getCatalog(), "%", this.tableName, null);
+ this.metaData = MetaData.createMetaData(metaResult);
+ this.metaData.setTableName(this.tableName);
+ }
+ return super.initConfigurable();
+ } catch (ClassNotFoundException | SQLException e) {
+ e.printStackTrace();
+ }
+ return false;
+ }
+
+ @Override
+ protected boolean batchInsert(List<IMessage> messageList) {
+ JDBCDriver dbDataSource = DriverBuilder.createDriver(jdbcDriver, url, userName, password);
+ try {
+ if (messageList == null || messageList.size() == 0) {
+ return true;
+ }
+ List<JSONObject> messages = convertJsonObjectFromMessage(messageList);
+ if (StringUtil.isEmpty(insertSQLTemplate) && metaData != null) {
+ String sql = SQLUtil.createInsertSql(metaData, messages.get(0));
+ sql = sql + SQLUtil.createInsertValuesSQL(metaData, messages.subList(1, messages.size()));
+ executeSQL(dbDataSource, sql);
+ return true;
+ }
+ String insertValueSQL = parseInsertValues(insertSQLTemplate);
+ if (StringUtil.isEmpty(insertValueSQL) || insertSQLTemplate.replace(insertValueSQL, "").contains("#{")) {
+ for (JSONObject message : messages) {
+ String sql = parseSQL(message, insertSQLTemplate);
+ executeSQL(dbDataSource, sql);
+ }
+ return true;
+ } else {
+ StringBuilder sb = new StringBuilder();
+ String insertSQL;
+ boolean isFirst = true;
+ int i = 0;
+ for (JSONObject message : messages) {
+ insertSQL = parseSQL(message, insertValueSQL);
+ if (isFirst) {
+ isFirst = false;
+ } else {
+ sb.append(",");
+ }
+ i++;
+
+ sb.append(insertSQL);
+ }
+ insertSQL = this.insertSQLTemplate.replace(insertValueSQL, sb.toString());
+ executeSQL(dbDataSource, insertSQL);
+ return true;
+ }
+ } finally {
+ dbDataSource.destroy();
+ }
+ }
+
+ protected void executeSQL(JDBCDriver dbDataSource, String sql) {
+ dbDataSource.execute(sql);
+ }
+
+ /**
+ * 解析出insert value数据部分，对于批量的插入，效果会更佳
+ */
+ private static final String VALUES_NAME = "values";
+
+ protected String parseInsertValues(String insertSQL) {
+ int start = insertSQL.toLowerCase().indexOf(VALUES_NAME);
+ if (start == -1) {
+ return null;
+ }
+ String valuesSQL = insertSQL.substring(start + VALUES_NAME.length());
+ int end = valuesSQL.toLowerCase().lastIndexOf(")");
+ if (end == -1) {
+ return null;
+ }
+ return valuesSQL.substring(0, end + 1);
+ }
+
+ protected String parseSQL(JSONObject message, String sql) {
+ return SQLUtil.parseIbatisSQL(message, sql);
+ }
+
+ public String getInsertSQLTemplate() {
+ return insertSQLTemplate;
+ }
+
+ public void setInsertSQLTemplate(String insertSQLTemplate) {
+ this.insertSQLTemplate = insertSQLTemplate;
+ }
+
+ public String getJdbcDriver() {
+ return jdbcDriver;
+ }
+
+ public void setJdbcDriver(String jdbcDriver) {
+ this.jdbcDriver = jdbcDriver;
+ }
+
+ public String getUrl() {
+ return url;
+ }
+
+ public void setUrl(String url) {
+ this.url = url;
+ }
+
+ public String getUserName() {
+ return userName;
+ }
+
+ public void setUserName(String userName) {
+ this.userName = userName;
+ }
+
+ public String getPassword() {
+ return password;
+ }
+
+ public void setPassword(String password) {
+ this.password = password;
+ }
+
+ public MetaData getMetaData() {
+ return metaData;
+ }
+
+ public void setMetaData(MetaData metaData) {
+ this.metaData = metaData;
+ }
+
+ public String getTableName() {
+ return tableName;
+ }
+
+ public void setTableName(String tableName) {
+ this.tableName = tableName;
+ }
+}
diff --git a/rocketmq-streams-channel-db/src/main/java/org/apache/rocketmq/streams/db/sink/DBSinkBuilder.java b/rocketmq-streams-channel-db/src/main/java/org/apache/rocketmq/streams/db/sink/DBSinkBuilder.java
new file mode 100644
index 0000000..ef7ae28
--- /dev/null
+++ b/rocketmq-streams-channel-db/src/main/java/org/apache/rocketmq/streams/db/sink/DBSinkBuilder.java
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.db.sink;
+
+import java.util.List;
+import java.util.Properties;
+
+import org.apache.rocketmq.streams.common.channel.builder.IChannelBuilder;
+import org.apache.rocketmq.streams.common.channel.sink.ISink;
+import org.apache.rocketmq.streams.common.channel.source.ISource;
+import org.apache.rocketmq.streams.common.model.ServiceName;
+import org.apache.rocketmq.streams.common.metadata.MetaData;
+import org.apache.rocketmq.streams.common.metadata.MetaDataField;
+import org.apache.rocketmq.streams.common.utils.DataTypeUtil;
+import com.google.auto.service.AutoService;
+
+@AutoService(IChannelBuilder.class)
+@ServiceName(DBSinkBuilder.TYPE)
+public class DBSinkBuilder implements IChannelBuilder {
+ public static final String TYPE = "rds";
+
+ @Override
+ public ISink createSink(String namespace, String name, Properties properties, MetaData metaData) {
+ DBSink sink = new DBSink();
+ sink.setUrl(properties.getProperty("url"));
+ sink.setUserName("userName");
+ sink.setPassword("password");
+ List<MetaDataField> fieldList = metaData.getMetaDataFields();
+ StringBuilder insertSQL = new StringBuilder();
+ StringBuilder insertValueSQL = new StringBuilder();
+ boolean isFirst = true;
+ for (MetaDataField field : fieldList) {
+ String fieldName = field.getFieldName();
+ if (isFirst) {
+ isFirst = false;
+ } else {
+ insertSQL.append(",");
+ insertValueSQL.append(",");
+ }
+ insertSQL.append(fieldName);
+ if (DataTypeUtil.isNumber(field.getDataType())) {
+ insertValueSQL.append(fieldName);
+ } else {
+ insertValueSQL.append("'#{" + fieldName + "}'");
+ }
+ }
+ String sql = "insert into " + properties.getProperty("tableName") + "(" + insertSQL.toString() + ")values(" + insertValueSQL.toString() + ")";
+ sink.setInsertSQLTemplate(sql);
+ return sink;
+ }
+
+ @Override
+ public ISource createSource(String namespace, String name, Properties properties, MetaData metaData) {
+ throw new RuntimeException("can not support this method");
+ }
+
+ @Override
+ public String getType() {
+ return TYPE;
+ }
+
+}
diff --git a/rocketmq-streams-channel-db/src/main/java/org/apache/rocketmq/streams/db/sink/SelfMultiTableSink.java b/rocketmq-streams-channel-db/src/main/java/org/apache/rocketmq/streams/db/sink/SelfMultiTableSink.java
new file mode 100644
index 0000000..96922e6
--- /dev/null
+++ b/rocketmq-streams-channel-db/src/main/java/org/apache/rocketmq/streams/db/sink/SelfMultiTableSink.java
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.db.sink;
+
+import org.apache.rocketmq.streams.common.channel.split.ISplit;
+import org.apache.rocketmq.streams.common.configurable.IAfterConfiguableRefreshListerner;
+import org.apache.rocketmq.streams.common.configurable.IConfigurableService;
+import org.apache.rocketmq.streams.common.context.IMessage;
+import org.apache.rocketmq.streams.common.functions.MultiTableSplitFunction;
+import org.apache.rocketmq.streams.common.utils.Base64Utils;
+import org.apache.rocketmq.streams.common.utils.InstantiationUtil;
+
+public class SelfMultiTableSink extends AbstractMultiTableSink implements IAfterConfiguableRefreshListerner {
+ protected String multiTableSplitFunctionSerializeValue;//用户自定义的operator的序列化字节数组，做了base64解码
+ protected transient MultiTableSplitFunction<IMessage> multiTableSplitFunction;
+
+ public SelfMultiTableSink(String url, String userName, String password, MultiTableSplitFunction<IMessage> multiTableSplitFunction) {
+ super(url, userName, password);
+ this.multiTableSplitFunction = multiTableSplitFunction;
+ byte[] bytes = InstantiationUtil.serializeObject(multiTableSplitFunction);
+ multiTableSplitFunctionSerializeValue = Base64Utils.encode(bytes);
+ }
+
+ @Override
+ protected String createTableName(String splitId) {
+ return multiTableSplitFunction.createTableFromSplitId(splitId);
+ }
+
+ @Override
+ protected ISplit getSplitFromMessage(IMessage message) {
+ return multiTableSplitFunction.createSplit(message);
+ }
+
+ @Override
+ public void doProcessAfterRefreshConfigurable(IConfigurableService configurableService) {
+ byte[] bytes = Base64Utils.decode(multiTableSplitFunctionSerializeValue);
+ this.multiTableSplitFunction = InstantiationUtil.deserializeObject(bytes);
+ }
+}
diff --git a/rocketmq-streams-channel-db/src/main/java/org/apache/rocketmq/streams/db/sink/SplitBySerialNumber.java b/rocketmq-streams-channel-db/src/main/java/org/apache/rocketmq/streams/db/sink/SplitBySerialNumber.java
new file mode 100644
index 0000000..c2a49b7
--- /dev/null
+++ b/rocketmq-streams-channel-db/src/main/java/org/apache/rocketmq/streams/db/sink/SplitBySerialNumber.java
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.db.sink;
+
+import org.apache.rocketmq.streams.common.channel.split.ISplit;
+import org.apache.rocketmq.streams.common.context.IMessage;
+
+public class SplitBySerialNumber extends AbstractMultiTableSink {
+ public SplitBySerialNumber(String url, String userName, String password) {
+ super(url, userName, password);
+ }
+
+ @Override
+ protected String createTableName(String splitId) {
+ return null;
+ }
+
+ @Override
+ protected ISplit getSplitFromMessage(IMessage message) {
+ return null;
+ }
+}
diff --git a/rocketmq-streams-channel-db/src/main/java/org/apache/rocketmq/streams/db/sink/SplitByTimeMultiTableSink.java b/rocketmq-streams-channel-db/src/main/java/org/apache/rocketmq/streams/db/sink/SplitByTimeMultiTableSink.java
new file mode 100644
index 0000000..87a2b3e
--- /dev/null
+++ b/rocketmq-streams-channel-db/src/main/java/org/apache/rocketmq/streams/db/sink/SplitByTimeMultiTableSink.java
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.db.sink;
+
+import org.apache.rocketmq.streams.common.channel.split.ISplit;
+import org.apache.rocketmq.streams.common.context.IMessage;
+
+public class SplitByTimeMultiTableSink extends AbstractMultiTableSink {
+ public SplitByTimeMultiTableSink(String url, String userName, String password) {
+ super(url, userName, password);
+ }
+
+ @Override
+ protected String createTableName(String splitId) {
+ return null;
+ }
+
+ @Override
+ protected ISplit getSplitFromMessage(IMessage message) {
+ return null;
+ }
+}
diff --git a/rocketmq-streams-channel-db/src/test/java/org/apache/rocketmq/streams/db/sink/db/DBWriteOnlyChannelTest.java b/rocketmq-streams-channel-db/src/test/java/org/apache/rocketmq/streams/db/sink/db/DBWriteOnlyChannelTest.java
new file mode 100644
index 0000000..c14bbc3
--- /dev/null
+++ b/rocketmq-streams-channel-db/src/test/java/org/apache/rocketmq/streams/db/sink/db/DBWriteOnlyChannelTest.java
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.db.sink.db;
+
+import com.alibaba.fastjson.JSONObject;
+
+import org.apache.rocketmq.streams.common.context.IMessage;
+import org.apache.rocketmq.streams.common.context.Message;
+import org.apache.rocketmq.streams.common.metadata.MetaData;
+import org.apache.rocketmq.streams.db.driver.JDBCDriver;
+import org.apache.rocketmq.streams.db.sink.DBSink;
+import org.junit.Test;
+
+public class DBWriteOnlyChannelTest {
+
+ private String URL = "jdbc:mysql://XXXXX:3306/yundun_soc?useUnicode=true&characterEncoding=utf8&autoReconnect=true";
+ protected String USER_NAME = "XXXX";
+ protected String PASSWORD = "XXXX";
+
+ @Test
+ public void testOutputBySQL() {
+ String sql = "insert into table(name,age) values('#{name}',#{age})";
+ DBSink sink = new DBSink(sql, URL, USER_NAME, PASSWORD) {
+
+ /**
+ * 因为不是真实表，会报错，把执行sql，改成打印sql
+ */
+ @Override
+ protected void executeSQL(JDBCDriver dbDataSource, String sql) {
+ System.out.println(sql);
+ }
+ };
+ for (int i = 0; i < 10; i++) {
+ JSONObject msg = new JSONObject();
+ msg.put("name", "chris" + i);
+ msg.put("age", i);
+ IMessage message = new Message(msg);
+ sink.batchAdd(message);
+ }
+ sink.flush();
+ }
+
+ @Test
+ public void testOutputByMetaData() {
+ DBSink sink = new DBSink() {
+ /**
+ * 因为不是真实表，会报错，把执行sql，改成打印sql
+ */
+ @Override
+ protected void executeSQL(JDBCDriver dbDataSource, String sql) {
+ System.out.println(sql);
+ }
+ };
+ JSONObject msg = new JSONObject();
+ msg.put("name", "chris");
+ msg.put("age", 18);
+ MetaData metaData = MetaData.createMetaData(msg);
+ metaData.setTableName("tableName");
+ sink.setMetaData(metaData);
+ for (int i = 0; i < 10; i++) {
+ msg = new JSONObject();
+ msg.put("name", "chris" + i);
+ msg.put("age", i);
+ IMessage message = new Message(msg);
+ sink.batchAdd(message);
+ }
+ sink.flush();
+ }
+
+}
[rocketmq-streams] 27/35: Merge pull request #4 from speak2me/main
Posted by du...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
duhengforever pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/rocketmq-streams.git
commit 177cc756b595932dcbe17ca14bfb09f7381c903d
Merge: 5ef7b1d c41c825
Author: rongtong <ji...@163.com>
AuthorDate: Mon Aug 2 19:13:41 2021 +0800
Merge pull request #4 from speak2me/main
add filter channel-http schedule modules
rocketmq-streams-channel-http/pom.xml | 30 ++
.../rocketmq-streams-channel-http.iml | 62 +++
.../rocketmq/streams/http/source/HttpSource.java | 157 ++++++
.../streams/http/source/server/HttpServer.java | 273 ++++++++++
.../http/source/server/HttpServerManager.java | 60 +++
.../streams/http/source/util/HttpUtil.java | 251 +++++++++
.../streams/http/source/http/HttpChannelTest.java | 45 ++
rocketmq-streams-filter/pom.xml | 31 ++
.../rocketmq-streams-filter.iml | 72 +++
.../rocketmq/streams/filter/FilterComponent.java | 115 ++++
.../streams/filter/builder/ExpressionBuilder.java | 409 ++++++++++++++
.../streams/filter/builder/RuleBuilder.java | 592 +++++++++++++++++++++
.../streams/filter/builder/RuleElementBuilder.java | 137 +++++
.../streams/filter/contants/RuleElementType.java | 43 ++
.../streams/filter/contants/RuleStatus.java | 98 ++++
.../streams/filter/context/ContextConfigure.java | 272 ++++++++++
.../streams/filter/context/RuleContext.java | 399 ++++++++++++++
.../streams/filter/context/RuleMessage.java | 48 ++
.../streams/filter/engine/IRuleEngine.java | 51 ++
.../rocketmq/streams/filter/engine/SplitFlow.java | 159 ++++++
.../filter/engine/impl/DefaultRuleEngine.java | 280 ++++++++++
.../filter/exception/RegexTimeoutException.java | 48 ++
.../filter/function/etl/RenameFunction.java | 37 ++
.../expression/AbstractExpressionFunction.java | 67 +++
.../function/expression/CompareFunction.java | 74 +++
.../ContainsCaseInsensitiveFunction.java | 74 +++
.../function/expression/ContainsFunction.java | 62 +++
.../streams/filter/function/expression/Equals.java | 166 ++++++
.../function/expression/ExpressionFunction.java | 33 ++
.../filter/function/expression/GreaterEquals.java | 163 ++++++
.../filter/function/expression/GreaterThan.java | 132 +++++
.../filter/function/expression/InFunction.java | 85 +++
.../filter/function/expression/InMetaData.java | 65 +++
.../function/expression/IpContainsFunction.java | 180 +++++++
.../filter/function/expression/IsNotNull.java | 50 ++
.../streams/filter/function/expression/IsNull.java | 49 ++
.../filter/function/expression/LessEquals.java | 118 ++++
.../filter/function/expression/LessThan.java | 119 +++++
.../filter/function/expression/LikeFunction.java | 167 ++++++
.../NotContainsCaseInsensitiveFunction.java | 76 +++
.../function/expression/NotContainsFunction.java | 63 +++
.../filter/function/expression/NotEquals.java | 169 ++++++
.../filter/function/expression/NotInFunction.java | 35 ++
.../function/expression/NotLikeFunction.java | 36 ++
.../NotRegexCaseInsensitiveFunction.java | 36 ++
.../function/expression/NotRegexFunction.java | 36 ++
.../expression/RegexCaseInsensitiveFunction.java | 41 ++
.../filter/function/expression/RegexFunction.java | 123 +++++
.../filter/function/expression/ScriptFunction.java | 145 +++++
.../filter/function/script/CaseFunction.java | 86 +++
.../rocketmq/streams/filter/monitor/Monitor.java | 160 ++++++
.../filter/monitor/contants/MonitorType.java | 40 ++
.../streams/filter/monitor/rule/ActionMonitor.java | 27 +
.../filter/monitor/rule/ExpressionMonitor.java | 27 +
.../filter/monitor/rule/MessageMonitor.java | 119 +++++
.../streams/filter/monitor/rule/NullMonitor.java | 55 ++
.../streams/filter/monitor/rule/RuleMonitor.java | 62 +++
.../streams/filter/monitor/rule/VarMonitor.java | 27 +
.../streams/filter/operator/FilterOperator.java | 31 ++
.../rocketmq/streams/filter/operator/Rule.java | 538 +++++++++++++++++++
.../streams/filter/operator/action/Action.java | 28 +
.../operator/action/IConfigurableAction.java | 41 ++
.../filter/operator/action/impl/ChannelAction.java | 69 +++
.../operator/action/impl/MetaDataAction.java | 185 +++++++
.../filter/operator/expression/Expression.java | 405 ++++++++++++++
.../operator/expression/ExpressionPerformance.java | 99 ++++
.../expression/ExpressionRelationParser.java | 106 ++++
.../expression/ExpressionRelationPaser.java | 107 ++++
.../operator/expression/GroupExpression.java | 146 +++++
.../expression/GroupExpressionManager.java | 86 +++
.../operator/expression/OptimizationRule.java | 28 +
.../operator/expression/RelationExpression.java | 347 ++++++++++++
.../operator/expression/SimpleExpression.java | 113 ++++
.../streams/filter/operator/var/ConstantVar.java | 118 ++++
.../streams/filter/operator/var/ContextVar.java | 85 +++
.../streams/filter/operator/var/InnerVar.java | 86 +++
.../rocketmq/streams/filter/operator/var/Var.java | 79 +++
.../optimization/EqualsExpressionOptimization.java | 39 ++
.../optimization/ExpressionOptimization.java | 144 +++++
.../optimization/IExpressionOptimization.java | 38 ++
.../optimization/LikeExpressionOptimization.java | 39 ++
.../optimization/OptimizationExpression.java | 71 +++
.../PiplineLogFingerprintAnalysis.java | 433 +++++++++++++++
.../optimization/RegexExpressionOptimization.java | 39 ++
.../streams/filter/service/IRuleEngineService.java | 50 ++
.../filter/service/impl/RuleEngineServiceImpl.java | 104 ++++
.../rocketmq/streams/filter/utils/IPUtil.java | 209 ++++++++
.../rocketmq/streams/filter/utils/RegexUtil.java | 211 ++++++++
.../streams/filter/FilterComponentTest.java | 36 ++
.../streams/filter/operator/ExpressionTest.java | 49 ++
.../rocketmq/streams/filter/operator/RuleTest.java | 40 ++
.../test/resources/credible/ChannelComponent.xml | 70 +++
.../resources/credible/credible-channel.properties | 12 +
.../test/resources/credible/credible.properties | 8 +
.../src/test/resources/log4j.xml | 20 +
rocketmq-streams-schedule/pom.xml | 33 ++
.../rocketmq-streams-schedule.iml | 65 +++
.../streams/schedule/ScheduleComponent.java | 65 +++
.../schedule/job/ConfigurableExecutorJob.java | 55 ++
.../streams/schedule/service/IScheduleService.java | 90 ++++
.../schedule/service/impl/ScheduleServiceImpl.java | 264 +++++++++
.../stream/schedule/ScheduleComponentTest.java | 62 +++
.../src/test/resources/log4j.xml | 20 +
103 files changed, 11599 insertions(+)
[rocketmq-streams] 25/35: add script module add common module
Posted by du...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
duhengforever pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/rocketmq-streams.git
commit 60409d597e503e7fb2d382ef07c367efb46200f0
Author: yuanxiaodong <ch...@alibaba-inc.com>
AuthorDate: Mon Aug 2 17:14:46 2021 +0800
add script module
add common module
---
rocketmq-streams-commons/pom.xml | 76 ++
.../streams/common/cache/CompressTable.java | 305 +++++++
.../common/cache/compress/AdditionStore.java | 161 ++++
.../streams/common/cache/compress/ByteArray.java | 103 +++
.../common/cache/compress/ByteArrayValueKV.java | 106 +++
.../streams/common/cache/compress/CacheKV.java | 371 ++++++++
.../streams/common/cache/compress/ICacheKV.java | 63 ++
.../streams/common/cache/compress/KVElement.java | 123 +++
.../cache/compress/impl/FixedLenRowCacheKV.java | 148 ++++
.../common/cache/compress/impl/IntValueKV.java | 195 +++++
.../streams/common/cache/compress/impl/KeySet.java | 109 +++
.../common/cache/compress/impl/ListValueKV.java | 66 ++
.../common/cache/compress/impl/MultiValueKV.java | 114 +++
.../common/cache/compress/impl/MutilValueKV.java | 116 +++
.../common/cache/compress/impl/SplitCache.java | 86 ++
.../common/cache/compress/impl/StringValueKV.java | 137 +++
.../streams/common/cache/softreference/ICache.java | 58 ++
.../cache/softreference/RebuildCacheElement.java | 26 +
.../softreference/impl/SoftReferenceCache.java | 127 +++
.../streams/common/calssscaner/AbstractScan.java | 257 ++++++
.../streams/common/channel/AbstractChannel.java | 220 +++++
.../rocketmq/streams/common/channel/IChannel.java | 34 +
.../AbstractSupportShuffleChannelBuilder.java | 31 +
.../common/channel/builder/IChannelBuilder.java | 70 ++
.../channel/builder/IShuffleChannelBuilder.java | 39 +
.../common/channel/impl/OutputPrintChannel.java | 47 +
.../common/channel/impl/PrintChannelBuilder.java | 47 +
.../channel/impl/file/FileChannelBuilder.java | 96 ++
.../streams/common/channel/impl/file/FileSink.java | 164 ++++
.../common/channel/impl/file/FileSource.java | 232 +++++
.../common/channel/impl/file/FileSplit.java | 64 ++
.../common/channel/impl/memory/MemoryCache.java | 83 ++
.../common/channel/impl/memory/MemoryChannel.java | 79 ++
.../channel/impl/memory/MemoryChannelBuilder.java | 71 ++
.../common/channel/impl/memory/MemorySink.java | 107 +++
.../common/channel/impl/memory/MemorySource.java | 92 ++
.../common/channel/impl/memory/MemorySplit.java | 52 ++
.../streams/common/channel/sink/AbstractSink.java | 243 +++++
.../channel/sink/AbstractSupportShuffleSink.java | 80 ++
.../streams/common/channel/sink/ISink.java | 104 +++
.../channel/sinkcache/DataSourceAutoFlushTask.java | 66 ++
.../common/channel/sinkcache/IMessageCache.java | 79 ++
.../channel/sinkcache/IMessageFlushCallBack.java | 31 +
.../impl/AbstractMutilSplitMessageCache.java | 137 +++
.../channel/sinkcache/impl/MessageCache.java | 142 +++
.../sinkcache/impl/MultiSplitMessageCache.java | 33 +
.../sinkcache/impl/MutilSplitMessageCache.java | 33 +
.../common/channel/source/AbstractBatchSource.java | 202 +++++
.../common/channel/source/AbstractSource.java | 536 ++++++++++++
.../source/AbstractSupportOffsetResetSource.java | 21 +
.../channel/source/AbstractUnreliableSource.java | 164 ++++
.../streams/common/channel/source/ISource.java | 90 ++
.../channel/source/ISplitChangedListener.java | 41 +
.../channel/source/systemmsg/NewSplitMessage.java | 26 +
.../source/systemmsg/RemoveSplitMessage.java | 26 +
.../source/systemmsg/SplitChangedMessage.java | 73 ++
.../streams/common/channel/split/ISplit.java | 49 ++
.../streams/common/checkpoint/CheckPoint.java | 78 ++
.../common/checkpoint/CheckPointManager.java | 279 ++++++
.../common/checkpoint/CheckPointMessage.java | 77 ++
.../streams/common/checkpoint/CheckPointState.java | 65 ++
.../streams/common/checkpoint/ICheckPoint.java | 31 +
.../common/classloader/ByteClassLoader.java | 30 +
.../common/classloader/FileClassLoader.java | 83 ++
.../common/classloader/IsolationClassLoader.java | 78 ++
.../common/compiler/CustomJavaCompiler.java | 248 ++++++
.../common/component/AbstractComponent.java | 208 +++++
.../streams/common/component/ComponentCreator.java | 293 +++++++
.../common/component/ConfigureDescriptor.java | 98 +++
.../streams/common/component/IComponent.java | 73 ++
.../streams/common/component/IgnoreNameSpace.java | 24 +
.../PropertyConfigureDescriptorManager.java | 104 +++
.../PropertyConfigureDiscriptorManager.java | 110 +++
.../common/configurable/AbstractConfigurable.java | 216 +++++
.../common/configurable/BasedConfigurable.java | 249 ++++++
.../IAfterConfiguableRefreshListerner.java | 25 +
.../streams/common/configurable/IConfigurable.java | 84 ++
.../configurable/IConfigurableIdentification.java | 40 +
.../configurable/IConfigurableListerner.java | 20 +
.../common/configurable/IConfigurableService.java | 136 +++
.../common/configurable/IFieldProcessor.java | 24 +
.../common/configurable/annotation/Changeable.java | 31 +
.../configurable/annotation/ENVDependence.java | 30 +
.../configurable/annotation/NoSerialized.java | 29 +
.../streams/common/configure/ConfigureFileKey.java | 142 +++
.../streams/common/context/AbstractContext.java | 364 ++++++++
.../streams/common/context/BatchMessageOffset.java | 123 +++
.../rocketmq/streams/common/context/Context.java | 33 +
.../rocketmq/streams/common/context/IMessage.java | 85 ++
.../streams/common/context/IgnoreMessage.java | 23 +
.../rocketmq/streams/common/context/Message.java | 129 +++
.../streams/common/context/MessageHeader.java | 360 ++++++++
.../streams/common/context/MessageOffset.java | 208 +++++
.../streams/common/context/UserDefinedMessage.java | 502 +++++++++++
.../streams/common/datatype/ArrayDataType.java | 146 ++++
.../streams/common/datatype/BaseDataType.java | 229 +++++
.../streams/common/datatype/BooleanDataType.java | 142 +++
.../streams/common/datatype/ByteDataType.java | 107 +++
.../common/datatype/ConfigurableDataType.java | 127 +++
.../streams/common/datatype/DataJsonable.java | 36 +
.../rocketmq/streams/common/datatype/DataType.java | 109 +++
.../streams/common/datatype/DateDataType.java | 153 ++++
.../streams/common/datatype/DoubleDataType.java | 156 ++++
.../streams/common/datatype/FloatDataType.java | 121 +++
.../common/datatype/GenericParameterDataType.java | 122 +++
.../common/datatype/GenericParamterDataType.java | 123 +++
.../streams/common/datatype/IJsonable.java | 34 +
.../streams/common/datatype/IntDataType.java | 122 +++
.../streams/common/datatype/JavaBeanDataType.java | 92 ++
.../streams/common/datatype/JsonableDataType.java | 104 +++
.../streams/common/datatype/ListDataType.java | 223 +++++
.../streams/common/datatype/LongDataType.java | 173 ++++
.../streams/common/datatype/MapDataType.java | 286 ++++++
.../common/datatype/NotSupportDataType.java | 74 ++
.../streams/common/datatype/NumberDataType.java | 84 ++
.../streams/common/datatype/SetDataType.java | 257 ++++++
.../streams/common/datatype/ShortDataType.java | 118 +++
.../streams/common/datatype/StringDataType.java | 83 ++
.../streams/common/dboperator/IDBDriver.java | 79 ++
.../common/disruptor/BufferFullFunction.java | 27 +
.../streams/common/disruptor/DisruptorEvent.java | 34 +
.../common/disruptor/DisruptorEventFactory.java | 26 +
.../common/disruptor/DisruptorProducer.java | 51 ++
.../streams/common/functions/FilterFunction.java | 24 +
.../streams/common/functions/ForEachFunction.java | 31 +
.../common/functions/ForEachMessageFunction.java | 30 +
.../streams/common/functions/Function.java | 19 +
.../streams/common/functions/MapFunction.java | 24 +
.../common/functions/MultiTableSplitFunction.java | 29 +
.../streams/common/functions/ReduceFunction.java | 24 +
.../streams/common/functions/SplitFunction.java | 30 +
.../common/functions/TableSplitFunction.java | 46 +
.../common/interfaces/IBaseStreamOperator.java | 33 +
.../streams/common/interfaces/IFilterService.java | 57 ++
.../streams/common/interfaces/IJDBCOperator.java | 77 ++
.../common/interfaces/IJsonobjectProcessor.java | 25 +
.../common/interfaces/ILineMessageProcessor.java | 32 +
.../streams/common/interfaces/IPropertyEnable.java | 39 +
.../common/interfaces/IScheduleExecutor.java | 27 +
.../streams/common/interfaces/IStreamOperator.java | 29 +
.../streams/common/interfaces/ISystemMessage.java | 20 +
.../common/interfaces/ISystemMessageProcessor.java | 23 +
.../streams/common/interfaces/ITransport.java | 32 +
.../streams/common/logger/LoggerCreator.java | 57 ++
.../streams/common/logger/LoggerOutputChannel.java | 79 ++
.../common/logger/MyDailyRollingFileAppender.java | 545 ++++++++++++
.../streams/common/metadata/AbstractMetaData.java | 247 ++++++
.../rocketmq/streams/common/metadata/MetaData.java | 227 +++++
.../streams/common/metadata/MetaDataAdapter.java | 48 +
.../streams/common/metadata/MetaDataField.java | 167 ++++
.../rocketmq/streams/common/model/BeanHolder.java | 36 +
.../rocketmq/streams/common/model/Entity.java | 49 ++
.../rocketmq/streams/common/model/NameCreator.java | 63 ++
.../rocketmq/streams/common/model/ServiceName.java | 28 +
.../streams/common/model/ThreadContext.java | 31 +
.../rocketmq/streams/common/monitor/IMonitor.java | 171 ++++
.../streams/common/monitor/MonitorFactory.java | 133 +++
.../common/monitor/TopologyFilterMonitor.java | 66 ++
.../common/monitor/group/GroupedMonitorInfo.java | 119 +++
.../common/monitor/group/MonitorCommander.java | 200 +++++
.../streams/common/monitor/impl/DipperMonitor.java | 510 +++++++++++
.../streams/common/monitor/impl/MonitorItem.java | 177 ++++
.../common/monitor/impl/NothingMonitorItem.java | 59 ++
.../common/monitor/impl/NothingMontior.java | 112 +++
.../optimization/CalculationResultCache.java | 64 ++
.../common/optimization/HyperscanRegex.java | 104 +++
.../streams/common/optimization/LikeRegex.java | 175 ++++
.../common/optimization/LogFingerprintFilter.java | 152 ++++
.../common/optimization/MessageGloableTrace.java | 80 ++
.../common/optimization/OptimizationRegex.java | 308 +++++++
.../optimization/SQLLogFingerprintFilter.java | 53 ++
.../AbstractMutilPipelineChainPipline.java | 193 ++++
.../streams/common/topology/ChainPipeline.java | 447 ++++++++++
.../streams/common/topology/ChainStage.java | 189 ++++
.../common/topology/builder/IStageBuilder.java | 39 +
.../common/topology/builder/PipelineBuilder.java | 370 ++++++++
.../common/topology/model/AbstractRule.java | 181 ++++
.../common/topology/model/AbstractScript.java | 110 +++
.../common/topology/model/AbstractStage.java | 298 +++++++
.../streams/common/topology/model/ILifeCycle.java | 33 +
.../common/topology/model/IStageHandle.java | 44 +
.../streams/common/topology/model/IWindow.java | 92 ++
.../streams/common/topology/model/Pipeline.java | 291 ++++++
.../topology/model/PipelineSourceJoiner.java | 48 +
.../streams/common/topology/model/Union.java | 63 ++
.../stages/AbstractStatelessChainStage.java | 41 +
.../topology/stages/AbstractWindowStage.java | 90 ++
.../common/topology/stages/FilterChainStage.java | 357 ++++++++
.../common/topology/stages/JoinChainStage.java | 160 ++++
.../common/topology/stages/NewSQLChainStage.java | 82 ++
.../common/topology/stages/OpenAPIChainStage.java | 80 ++
.../common/topology/stages/OutputChainStage.java | 231 +++++
.../common/topology/stages/PythonChainStage.java | 80 ++
.../topology/stages/RightJoinChainStage.java | 92 ++
.../common/topology/stages/SQLChainStage.java | 81 ++
.../common/topology/stages/ScriptChainStage.java | 124 +++
.../topology/stages/SubPiplineChainStage.java | 410 +++++++++
.../common/topology/stages/UnionChainStage.java | 51 ++
.../common/topology/stages/WindowChainStage.java | 83 ++
.../common/topology/stages/udf/FilterOperator.java | 37 +
.../topology/stages/udf/FlatMapOperator.java | 61 ++
.../common/topology/stages/udf/IReducer.java | 35 +
.../common/topology/stages/udf/IRedurce.java | 36 +
.../common/topology/stages/udf/MapOperator.java | 35 +
.../common/topology/stages/udf/StageBuilder.java | 92 ++
.../common/topology/stages/udf/UDFChainStage.java | 74 ++
.../topology/stages/udf/UDFUnionChainStage.java | 92 ++
.../common/transport/AbstractFileTransport.java | 41 +
.../streams/common/transport/IFileTransport.java | 68 ++
.../rocketmq/streams/common/utils/AESUtil.java | 108 +++
.../rocketmq/streams/common/utils/Base64Utils.java | 100 +++
.../rocketmq/streams/common/utils/ClassUtil.java | 157 ++++
.../streams/common/utils/CollectionUtil.java | 65 ++
.../rocketmq/streams/common/utils/CommandUtil.java | 184 ++++
.../streams/common/utils/CompressUtil.java | 88 ++
.../streams/common/utils/ConfigurableUtil.java | 248 ++++++
.../streams/common/utils/ContantsUtil.java | 355 ++++++++
.../rocketmq/streams/common/utils/DESUtils.java | 142 +++
.../streams/common/utils/DataTypeUtil.java | 504 +++++++++++
.../rocketmq/streams/common/utils/DateUtil.java | 603 +++++++++++++
.../common/utils/DipperThreadLocalUtil.java | 35 +
.../rocketmq/streams/common/utils/ENVUtile.java | 55 ++
.../rocketmq/streams/common/utils/FileUtil.java | 973 +++++++++++++++++++++
.../rocketmq/streams/common/utils/IOUtil.java | 99 +++
.../rocketmq/streams/common/utils/IPUtil.java | 201 +++++
.../streams/common/utils/InstantiationUtil.java | 118 +++
.../streams/common/utils/Ip2LongUtils.java | 296 +++++++
.../rocketmq/streams/common/utils/JarUtil.java | 147 ++++
.../streams/common/utils/JsonableUtil.java | 61 ++
.../streams/common/utils/LogParserUtil.java | 241 +++++
.../rocketmq/streams/common/utils/MapKeyUtil.java | 155 ++++
.../rocketmq/streams/common/utils/MessageUtil.java | 102 +++
.../streams/common/utils/NameCreatorUtil.java | 61 ++
.../rocketmq/streams/common/utils/NumberUtils.java | 179 ++++
.../rocketmq/streams/common/utils/PrintUtil.java | 205 +++++
.../streams/common/utils/PropertiesUtils.java | 144 +++
.../streams/common/utils/RandomStrUtil.java | 38 +
.../rocketmq/streams/common/utils/ReflectUtil.java | 882 +++++++++++++++++++
.../rocketmq/streams/common/utils/RegexUtil.java | 21 +
.../rocketmq/streams/common/utils/RuntimeUtil.java | 63 ++
.../streams/common/utils/SQLFormatterUtil.java | 362 ++++++++
.../rocketmq/streams/common/utils/SQLUtil.java | 413 +++++++++
.../streams/common/utils/ScheduleUtil.java | 52 ++
.../rocketmq/streams/common/utils/ShellUtil.java | 189 ++++
.../rocketmq/streams/common/utils/StringUtil.java | 369 ++++++++
.../streams/common/utils/SyslogParserUtil.java | 25 +
.../rocketmq/streams/common/utils/ThreadUtil.java | 29 +
.../rocketmq/streams/common/utils/TraceUtil.java | 118 +++
.../common/cache/compress/IntValueKVTest.java | 66 ++
.../softreference/SoftReferenceCacheTest.java | 55 ++
.../rocketmq/streams/common/channel/SinkTest.java | 73 ++
.../streams/common/channel/SourceTest.java | 75 ++
.../streams/common/classscan/ClassScanTest.java | 34 +
.../common/compiler/CustomJavaCompilerTest.java | 35 +
.../streams/common/datatype/DataTypeTest.java | 39 +
.../streams/common/stages/SelfChainStageTest.java | 36 +
.../src/test/resources/log4j.xml | 16 +
rocketmq-streams-script/pom.xml | 33 +
.../rocketmq/streams/script/ScriptComponent.java | 76 ++
.../streams/script/annotation/Function.java | 31 +
.../streams/script/annotation/FunctionMethod.java | 51 ++
.../script/annotation/FunctionMethodAilas.java | 37 +
.../script/annotation/FunctionParamter.java | 44 +
.../streams/script/annotation/UDAFFunction.java | 34 +
.../streams/script/context/FunctionContext.java | 87 ++
.../function/aggregation/AverageAccumulator.java | 98 +++
.../function/aggregation/ConcatAccumulator.java | 96 ++
.../aggregation/ConcatDistinctAccumulator.java | 98 +++
.../function/aggregation/CountAccumulator.java | 79 ++
.../function/aggregation/DistinctAccumulator.java | 75 ++
.../function/aggregation/MaxAccumulator.java | 111 +++
.../function/aggregation/MinAccumulator.java | 111 +++
.../function/aggregation/SumAccumulator.java | 82 ++
.../script/function/impl/bool/BooleanFunction.java | 39 +
.../script/function/impl/common/EchoFunction.java | 47 +
.../function/impl/condition/EqualsFunction.java | 155 ++++
.../impl/condition/ExecuteByConditionFuction.java | 36 +
.../impl/condition/GreateEqualsFunction.java | 74 ++
.../function/impl/condition/GreaterFunction.java | 69 ++
.../script/function/impl/condition/IFFunction.java | 39 +
.../impl/condition/InConditionFunction.java | 99 +++
.../impl/condition/LessEqualsFunction.java | 71 ++
.../function/impl/condition/LessFunction.java | 69 ++
.../function/impl/context/ContextFunction.java | 44 +
.../function/impl/date/AddMonthFunction.java | 73 ++
.../impl/date/CurrentTimestampFunction.java | 63 ++
.../script/function/impl/date/DateAddFunction.java | 146 ++++
.../function/impl/date/DateDiffFunction.java | 124 +++
.../function/impl/date/DateFormatFunction.java | 92 ++
.../function/impl/date/DatePartFunction.java | 218 +++++
.../function/impl/date/DateToCharFunction.java | 93 ++
.../function/impl/date/DateTruncFunction.java | 143 +++
.../function/impl/date/DateUtilFunction.java | 130 +++
.../function/impl/date/DateUtileFunction.java | 134 +++
.../script/function/impl/date/DayFunction.java | 62 ++
.../function/impl/date/FromUnixTimeFunction.java | 72 ++
.../script/function/impl/date/GetDateFunction.java | 43 +
.../script/function/impl/date/IsDateFunction.java | 54 ++
.../script/function/impl/date/LastDayFunction.java | 69 ++
.../script/function/impl/date/NextDayFunction.java | 100 +++
.../script/function/impl/date/NowFuntion.java | 46 +
.../script/function/impl/date/QuarterFuction.java | 75 ++
.../script/function/impl/date/QuarterFunction.java | 76 ++
.../function/impl/date/UnixTimeStampFunction.java | 77 ++
.../function/impl/date/WeekOfYearFunction.java | 59 ++
.../script/function/impl/date/WeekdayFunction.java | 60 ++
.../script/function/impl/env/ENVFunction.java | 47 +
.../impl/field/AdditionalFiledFunction.java | 318 +++++++
.../function/impl/field/CoalesceFunction.java | 51 ++
.../function/impl/field/ExtraFieldFunction.java | 67 ++
.../script/function/impl/field/FieldFunction.java | 155 ++++
.../function/impl/field/RemoveFieldFunction.java | 59 ++
.../function/impl/field/RetainFieldFunction.java | 51 ++
.../script/function/impl/filter/BreakFunction.java | 41 +
.../function/impl/filter/FilterFunction.java | 74 ++
.../function/impl/flatmap/SplitArrayFunction.java | 164 ++++
.../function/impl/header/HeaderFunction.java | 48 +
.../function/impl/json/JsonArrayFunction.java | 230 +++++
.../function/impl/json/JsonCreatorFunction.java | 218 +++++
.../function/impl/json/JsonValueFunction.java | 108 +++
.../script/function/impl/math/AbsFunction.java | 118 +++
.../script/function/impl/math/AcosFunction.java | 109 +++
.../function/impl/math/AdditionFunction.java | 62 ++
.../script/function/impl/math/AsinFunction.java | 109 +++
.../script/function/impl/math/AtanFunction.java | 88 ++
.../script/function/impl/math/BinFunction.java | 54 ++
.../script/function/impl/math/BitFunction.java | 201 +++++
.../script/function/impl/math/BroundFunction.java | 113 +++
.../function/impl/math/CardinalityFunction.java | 49 ++
.../script/function/impl/math/CbrtFunction.java | 67 ++
.../script/function/impl/math/CeilFunction.java | 109 +++
.../script/function/impl/math/ConvFunction.java | 244 ++++++
.../script/function/impl/math/CosFunction.java | 109 +++
.../script/function/impl/math/CosHFunction.java | 109 +++
.../script/function/impl/math/CotFunction.java | 109 +++
.../script/function/impl/math/DegreesFunction.java | 90 ++
.../function/impl/math/DivisionFunction.java | 62 ++
.../script/function/impl/math/EFunction.java | 42 +
.../script/function/impl/math/ExpFunction.java | 109 +++
.../function/impl/math/FactorialFunction.java | 79 ++
.../script/function/impl/math/FloorFunction.java | 115 +++
.../script/function/impl/math/HexFunction.java | 69 ++
.../script/function/impl/math/LnFunction.java | 109 +++
.../script/function/impl/math/Log10Function.java | 105 +++
.../script/function/impl/math/Log2Function.java | 105 +++
.../script/function/impl/math/LogFunction.java | 114 +++
.../script/function/impl/math/MathFunction.java | 132 +++
.../function/impl/math/MultiplicationFunction.java | 63 ++
.../function/impl/math/OperationFunction.java | 123 +++
.../script/function/impl/math/PIFunction.java | 42 +
.../script/function/impl/math/PowerFunction.java | 114 +++
.../script/function/impl/math/RadiansFunction.java | 48 +
.../script/function/impl/math/RandFunction.java | 62 ++
.../script/function/impl/math/RoundFunction.java | 117 +++
.../function/impl/math/ShiftLeftFunction.java | 52 ++
.../function/impl/math/ShiftRightFunction.java | 52 ++
.../impl/math/ShiftRightUnSignedFunction.java | 52 ++
.../script/function/impl/math/SignFunction.java | 50 ++
.../script/function/impl/math/SinFunction.java | 109 +++
.../script/function/impl/math/SinHFunction.java | 109 +++
.../script/function/impl/math/SqrtFunction.java | 109 +++
.../function/impl/math/SubtractionFunction.java | 62 ++
.../script/function/impl/math/TanFunction.java | 108 +++
.../script/function/impl/math/TanHFunction.java | 109 +++
.../script/function/impl/math/TruncFunction.java | 118 +++
.../script/function/impl/math/UnHexFunction.java | 64 ++
.../script/function/impl/mock/MockFunction.java | 39 +
.../function/impl/offset/OffsetFunction.java | 67 ++
.../script/function/impl/parser/GrokFunction.java | 91 ++
.../function/impl/parser/LogParserFunction.java | 85 ++
.../function/impl/parser/Paser2JsonFunction.java | 169 ++++
.../function/impl/parser/PaserBySplitFunction.java | 172 ++++
.../function/impl/parser/RegexParserFunction.java | 85 ++
.../function/impl/platform/BlinkUDFFunction.java | 33 +
.../function/impl/python/GroovyFunction.java | 52 ++
.../function/impl/python/PythonFunction.java | 49 ++
.../script/function/impl/router/RouteFunction.java | 37 +
.../script/function/impl/sql/SQLFunction.java | 56 ++
.../script/function/impl/string/AsciiFunction.java | 52 ++
.../function/impl/string/Base64Function.java | 86 ++
.../function/impl/string/BlinkStringFunction.java | 53 ++
.../impl/string/CharMatchCountFunction.java | 61 ++
.../script/function/impl/string/ChrFunction.java | 49 ++
.../function/impl/string/HashCodeFunction.java | 50 ++
.../function/impl/string/InitCapFunction.java | 53 ++
.../script/function/impl/string/InstrFunction.java | 136 +++
.../function/impl/string/InstrtFunction.java | 134 +++
.../function/impl/string/Ip2RegionFunction.java | 43 +
.../function/impl/string/IsEncodingFunction.java | 62 ++
.../function/impl/string/KeyValueFunction.java | 120 +++
.../script/function/impl/string/LTrimFunction.java | 49 ++
.../function/impl/string/LengthBFunction.java | 50 ++
.../function/impl/string/LengthFunction.java | 50 ++
.../script/function/impl/string/Md5Function.java | 84 ++
.../script/function/impl/string/NullFunction.java | 43 +
.../function/impl/string/ParseUrlFunction.java | 102 +++
.../script/function/impl/string/RTrimFunction.java | 50 ++
.../function/impl/string/RegexCountFunction.java | 113 +++
.../script/function/impl/string/RegexFunction.java | 159 ++++
.../function/impl/string/RegexInstrFunction.java | 180 ++++
.../function/impl/string/RegexReplaceFunction.java | 92 ++
.../function/impl/string/RegexSubStrFunction.java | 127 +++
.../function/impl/string/RepeatFunction.java | 60 ++
.../function/impl/string/ReplaceFunction.java | 56 ++
.../function/impl/string/ReverseFunction.java | 50 ++
.../function/impl/string/SoundxFunction.java | 112 +++
.../script/function/impl/string/SpaceFunction.java | 60 ++
.../function/impl/string/SplitPartFunction.java | 133 +++
.../impl/string/SubStringIndexFunction.java | 128 +++
.../function/impl/string/ToCharFunction.java | 119 +++
.../function/impl/string/ToLowerFunction.java | 49 ++
.../function/impl/string/ToUpperFunction.java | 49 ++
.../function/impl/string/TransLateFunction.java | 57 ++
.../script/function/impl/string/TrimFunction.java | 50 ++
.../script/function/impl/string/UUIDFunction.java | 39 +
.../function/impl/string/UrlDecodeFunction.java | 85 ++
.../function/impl/string/UrlEncodeFunction.java | 125 +++
.../script/function/impl/type/CastFunction.java | 73 ++
.../script/function/impl/udtf/UDTFFunction.java | 61 ++
.../function/impl/window/WindowFunction.java | 64 ++
.../script/function/model/FunctionConfigure.java | 573 ++++++++++++
.../function/model/FunctionConfigureMap.java | 109 +++
.../script/function/model/FunctionInfo.java | 101 +++
.../script/function/model/FunctionInfoMap.java | 84 ++
.../script/function/model/FunctionType.java | 45 +
.../function/service/IDipperInterfaceAdpater.java | 46 +
.../script/function/service/IFunctionService.java | 130 +++
.../service/impl/DefaultFunctionServiceImpl.java | 396 +++++++++
.../function/service/impl/ScanFunctionService.java | 96 ++
.../script/operator/AbstractScriptOperator.java | 82 ++
.../operator/expression/GroupScriptExpression.java | 239 +++++
.../operator/expression/ScriptExpression.java | 327 +++++++
.../operator/expression/ScriptParameter.java | 135 +++
.../script/operator/impl/AggregationScript.java | 211 +++++
.../operator/impl/FileGroovyScriptOperator.java | 33 +
.../operator/impl/FileJPythonScriptOperatore.java | 32 +
.../script/operator/impl/FunctionScript.java | 247 ++++++
.../script/operator/impl/GroovyScriptOperator.java | 108 +++
.../operator/impl/JPythonScriptOperator.java | 95 ++
.../script/operator/impl/JavaScriptOperator.java | 63 ++
.../script/operator/impl/ScriptOperator.java | 29 +
.../optimization/AbstractFunctionOptimization.java | 47 +
.../script/optimization/CompileParameter.java | 132 +++
.../optimization/CompileScriptExpression.java | 121 +++
.../script/optimization/EqualsOptimization.java | 44 +
.../script/optimization/IFunctionOptimization.java | 38 +
.../optimization/OptimizationScriptExpression.java | 67 ++
.../script/optimization/RegexOptimization.java | 43 +
.../script/optimization/ScriptOptimization.java | 407 +++++++++
.../script/parser/IScriptExpressionParser.java | 41 +
.../parser/ScriptExpressionParserFactory.java | 70 ++
.../parser/imp/ConditionExpressionParser.java | 227 +++++
.../parser/imp/FunctionExpressionParser.java | 361 ++++++++
.../streams/script/parser/imp/FunctionParser.java | 166 ++++
.../streams/script/service/IAccumulator.java | 67 ++
.../script/service/IFunctionFieldDependent.java | 40 +
.../streams/script/service/IScriptExpression.java | 59 ++
.../streams/script/service/IScriptParamter.java | 43 +
.../streams/script/service/IScriptService.java | 84 ++
.../streams/script/service/IScriptUDFInit.java | 37 +
.../script/service/impl/ScriptServiceImpl.java | 121 +++
.../streams/script/service/udf/UDAFScript.java | 230 +++++
.../streams/script/service/udf/UDFScript.java | 296 +++++++
.../streams/script/service/udf/UDTFScript.java | 113 +++
.../streams/script/utils/FunctionUtils.java | 254 ++++++
.../rocketmq/streams/script/utils/MatchUtil.java | 82 ++
.../streams/script/utils/RandomStrUtil.java | 38 +
.../streams/script/utils/ScriptParserUtil.java | 23 +
.../streams/script/utils/StringMatchUtil.java | 87 ++
.../rocketmq/streams/script/utils/UniqID.java | 66 ++
.../src/main/resources/log4j.xml | 20 +
.../streams/script/function/FunctionTest.java | 89 ++
.../streams/script/function/GrokFunctionTest.java | 44 +
.../script/function/GroovyFunctionTest.java | 52 ++
.../script/function/JPythonFunctionTest.java | 48 +
.../function/aggregation/AccumulatorTest.java | 235 +++++
.../script/operator/ScriptOperatorTest.java | 105 +++
.../streams/script/parser/ScriptParserTest.java | 43 +
.../src/test/resources/log4j.xml | 20 +
.../src/test/resources/python_script.py | 22 +
480 files changed, 54314 insertions(+)
diff --git a/rocketmq-streams-commons/pom.xml b/rocketmq-streams-commons/pom.xml
new file mode 100755
index 0000000..e152477
--- /dev/null
+++ b/rocketmq-streams-commons/pom.xml
@@ -0,0 +1,76 @@
+<?xml version="1.0" encoding="utf-8"?>
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-streams</artifactId>
+ <version>2.0.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>rocketmq-streams-commons</artifactId>
+ <name>ROCKETMQ STREAMS :: commons</name>
+ <packaging>jar</packaging>
+ <dependencies>
+ <!-- http -->
+ <dependency>
+ <groupId>org.apache.commons</groupId>
+ <artifactId>commons-lang3</artifactId>
+ <version>3.11</version>
+ </dependency>
+
+ <!-- 测试依赖 -->
+ <dependency>
+ <groupId>junit</groupId>
+ <artifactId>junit</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>com.alibaba</groupId>
+ <artifactId>fastjson</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>commons-logging</groupId>
+ <artifactId>commons-logging</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>commons-io</groupId>
+ <artifactId>commons-io</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>log4j</groupId>
+ <artifactId>log4j</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>com.google.code.gson</groupId>
+ <artifactId>gson</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>com.google.auto.service</groupId>
+ <artifactId>auto-service</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>com.lmax</groupId>
+ <artifactId>disruptor</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>com.gliwka.hyperscan</groupId>
+ <artifactId>hyperscan</artifactId>
+ <version>5.4.0-2.0.0</version>
+ </dependency>
+
+ <dependency>
+ <groupId>net.java.dev.jna</groupId>
+ <artifactId>platform</artifactId>
+ <version>3.5.2</version>
+ </dependency>
+
+ </dependencies>
+</project>
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/CompressTable.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/CompressTable.java
new file mode 100644
index 0000000..bdc1d41
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/CompressTable.java
@@ -0,0 +1,305 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.cache;
+
+import org.apache.rocketmq.streams.common.datatype.DataType;
+import org.apache.rocketmq.streams.common.datatype.NotSupportDataType;
+import org.apache.rocketmq.streams.common.datatype.StringDataType;
+import org.apache.rocketmq.streams.common.utils.DataTypeUtil;
+
+import java.util.*;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * 压缩表，行数据以byte[][]存放
+ */
+public class CompressTable {
+
+ /**
+ * 列名和位置，根据列名生成一个位置
+ */
+ protected Map<String, Integer> cloumnName2Index = new HashMap<>();
+
+ /**
+ * 位置和列名，根据位置获取列名
+ */
+ protected Map<Integer, String> index2ColumnName = new HashMap<>();
+
+ /**
+ * 列名和类型的映射关系
+ */
+ protected Map<String, DataType> cloumnName2DatatType = new HashMap<>();
+
+ /**
+ * 列名当前索引值。根据获取的列的先后顺序，建立序号
+ */
+ protected AtomicInteger index = new AtomicInteger(0);
+
+ /**
+ * 表的全部数据，一行是一个byte[][]
+ */
+ protected List<byte[][]> rows = new ArrayList<>();
+
+ /**
+ * 总字节数
+ */
+ protected AtomicInteger byteCount = new AtomicInteger(0);
+
+ /**
+ * 创建迭代器，可以循环获取全部数据，一行数据是Map<String, Object>
+ *
+ * @return
+ */
+ public Iterator<Map<String, Object>> newIterator() {
+ return new Iterator<Map<String, Object>>() {
+
+ protected int rowIndex = 0;
+
+ @Override
+ public boolean hasNext() {
+ return rowIndex < rows.size();
+ }
+
+ @Override
+ public Map<String, Object> next() {
+ Map<String, Object> row = byte2Row(rows.get(rowIndex));
+ rowIndex++;
+ return row;
+ }
+ };
+ }
+
+ /**
+ * 增加一行数据，会进行序列化成二进制数组存储。数字类型会有较好的压缩效果，字符串未做压缩
+ *
+ * @param row
+ * @return
+ */
+ public Integer addRow(Map<String, Object> row) {
+ CountHolder countHolder = new CountHolder();
+ byte[][] values = row2Byte(row, countHolder);
+ byteCount.addAndGet(countHolder.count);
+ return saveRowByte(values, countHolder.count);
+ }
+
+ /**
+ * 获取一行数据，会反序列化为Map<String, Object>
+ *
+ * @param index 行号， List<byte[][]> 的下标
+ * @return
+ */
+ public Map<String, Object> getRow(Integer index) {
+ byte[][] bytes = loadRowByte(index);
+ if (bytes == null) {
+ return null;
+ }
+ return byte2Row(bytes);
+ }
+
+ /**
+ * 保存row到list
+ *
+ * @param values
+ * @return
+ */
+ protected Integer saveRowByte(byte[][] values, int byteSize) {
+ this.rows.add(values);
+ return rows.size() - 1;
+ }
+
+ /**
+ * 从list中加载行
+ *
+ * @param index
+ * @return
+ */
+ protected byte[][] loadRowByte(Integer index) {
+ return this.rows.get(index);
+ }
+
+ /**
+ * 把一个row行转换成byte[][]数组
+ *
+ * @param row
+ * @return
+ */
+ public byte[][] row2Byte(Map<String, Object> row) {
+ return row2Byte(row, null);
+ }
+
+ protected class CountHolder {
+ int count = 0;
+ }
+
+ /**
+ * 把一个row行转换成byte[][]数组
+ *
+ * @param row
+ * @return
+ */
+ protected byte[][] row2Byte(Map<String, Object> row, CountHolder countHolder) {
+
+ // byte[][] byteRows=new byte[row.size()][];
+ List<byte[]> list = new ArrayList<>();
+ Iterator<Map.Entry<String, Object>> it = row.entrySet().iterator();
+ while (it.hasNext()) {
+ Map.Entry<String, Object> entry = it.next();
+ int index = getColumnIndex(entry.getKey());
+ byte[] byteValue = createBytes(entry.getKey(), entry.getValue());
+ if (byteValue == null) {
+ list.add(new byte[0]);
+ // byteRows[value]=new byte[0];
+ } else {
+ list.add(byteValue);
+ // byteRows[value]=byteValue;
+ }
+ if (countHolder != null) {
+ int length = 0;
+ if (byteValue != null) {
+ length = byteValue.length;
+ }
+ countHolder.count = countHolder.count + length;
+ }
+ }
+ byte[][] byteRows = new byte[list.size()][];
+ for (int i = 0; i < list.size(); i++) {
+ byte[] temp = list.get(i);
+ byteRows[i] = temp;
+ }
+ return byteRows;
+ }
+
+ /**
+ * 把序列化的字节数组转换成map
+ *
+ * @param bytes 一行的字节数组
+ * @return
+ */
+ public Map<String, Object> byte2Row(byte[][] bytes) {
+ Map<String, Object> row = new HashMap<>();
+ for (int i = 0; i < bytes.length; i++) {
+ byte[] columnValue = bytes[i];
+ String columnName = index2ColumnName.get(i);
+ DataType dataType = cloumnName2DatatType.get(columnName);
+ Object object = null;
+ if (dataType != null) {
+ object = dataType.byteToValue(columnValue);
+ }
+ row.put(columnName, object);
+ }
+ return row;
+ }
+
+ /**
+ * 把数据转换成字节
+ *
+ * @param key
+ * @param value
+ * @return
+ */
+ private byte[] createBytes(String key, Object value) {
+ if (value == null) {
+ return null;
+ }
+ Object tmp = value;
+ DataType dataType = cloumnName2DatatType.get(key);
+ if (dataType == null) {
+ dataType = DataTypeUtil.getDataTypeFromClass(value.getClass());
+ if (dataType == null || dataType.getClass().getName().equals(NotSupportDataType.class.getName())) {
+ dataType = new StringDataType();
+ tmp = value.toString();
+ }
+ cloumnName2DatatType.put(key, dataType);
+ }
+ Object object = dataType.convert(tmp);
+ return dataType.toBytes(object, true);
+ }
+
+ /**
+ * 给每个列一个index，方便把数据放入数组中
+ *
+ * @param key
+ * @return
+ */
+ public int getColumnIndex(String key) {
+ Integer columnIndex = cloumnName2Index.get(key);
+ if (columnIndex == null) {
+ columnIndex = index.incrementAndGet() - 1;
+ cloumnName2Index.put(key, columnIndex);
+ index2ColumnName.put(columnIndex, key);
+ }
+ return columnIndex;
+ }
+
+ public static void main(String[] args) {
+ Map<String, Object> row = new HashMap<>();
+ row.put("name", "chris");
+ row.put("age", 18);
+ row.put("address", "huilongguafdffdsfdfdsn");
+ row.put("country", "china");
+ row.put("abc", "fdfd");
+ row.put("dxxx", "fdfd");
+ row.put("date", new Date());
+ row.put("ip", "192.168.0.1");
+ CompressTable tableCompress = new CompressTable();
+ byte[][] value = tableCompress.row2Byte(row);
+ int count = 0;
+ for (int i = 0; i < value.length; i++) {
+ byte[] tmp = value[i];
+ count += tmp.length;
+ }
+ System.out.println(count);
+ Map<String, Object> values = tableCompress.byte2Row(value);
+ Iterator<Map.Entry<String, Object>> it = values.entrySet().iterator();
+ while (it.hasNext()) {
+ Map.Entry<String, Object> entry = it.next();
+ System.out.println(entry.getKey() + ":" + entry.getValue());
+ }
+ }
+
+ public Map<String, Integer> getCloumnName2Index() {
+ return cloumnName2Index;
+ }
+
+ public Map<Integer, String> getIndex2ColumnName() {
+ return index2ColumnName;
+ }
+
+ public Map<String, DataType> getCloumnName2DatatType() {
+ return cloumnName2DatatType;
+ }
+
+ public void setCloumnName2Index(Map<String, Integer> cloumnName2Index) {
+ this.cloumnName2Index = cloumnName2Index;
+ }
+
+ public void setIndex2ColumnName(Map<Integer, String> index2ColumnName) {
+ this.index2ColumnName = index2ColumnName;
+ }
+
+ public void setCloumnName2DatatType(Map<String, DataType> cloumnName2DatatType) {
+ this.cloumnName2DatatType = cloumnName2DatatType;
+ }
+
+ public int getByteCount() {
+ return byteCount.get();
+ }
+
+ public int getRowCount() {
+ return rows.size();
+ }
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/compress/AdditionStore.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/compress/AdditionStore.java
new file mode 100644
index 0000000..0b441bf
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/compress/AdditionStore.java
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.cache.compress;
+
+import java.util.ArrayList;
+import java.util.List;
+
+
+public class AdditionStore {
+
+ /**
+ * 每个冲突域列表，每个列表的最大值
+ */
+ public static final int CONFLICT_UNIT_SIZE = 1024 * 1024;
+
+ /**
+ * 如果value是非int值，可以通过这个值存储。原来value部分存储地址
+ */
+ private List<byte[]> values = new ArrayList<>();
+
+ /**
+ * 当前冲突元素存放在list的哪个index中
+ */
+ private int conflictIndex = -1;
+
+ /**
+ * 当前冲突的元素在byte中的可用位置
+ */
+ private int conflictOffset = 0;
+
+ /**
+ * 如果元素是固定大小，则这个值表示元素字节个数；
+ */
+ private int elementSize = -1;
+
+ /**
+ * 值是否是变长的，如果是变长的，需要额外两个字段存储长度
+ */
+ private boolean isVarLen = true;
+
+ /**
+ * 每个存储单元的大小
+ */
+ protected int blockSize = CONFLICT_UNIT_SIZE;
+
+ public AdditionStore() {
+
+ }
+
+ public AdditionStore(int elementSize, int blockSize) {
+ this.elementSize = elementSize;
+ if (elementSize > 0) {
+ isVarLen = false;
+ }
+ if (blockSize > 0) {
+ this.blockSize = blockSize;
+ }
+ }
+
+ public AdditionStore(int elementSize) {
+ this(elementSize, CONFLICT_UNIT_SIZE);
+ }
+
+ /**
+ * 把一个字节数组插入到存储中，并返回当前地址
+ *
+ * @param value
+ */
+ public CacheKV.MapAddress add2Store(byte[] value) {
+ if (conflictIndex == -1 || values.size() <= conflictIndex) {
+ byte[] bytes = new byte[blockSize];
+ values.add(bytes);
+ conflictOffset = 0;
+ }
+ if (conflictIndex == -1) {
+ conflictIndex = 0;
+ }
+ int length = value.length;
+ if (isVarLen) {
+ length = length + 2;
+ }
+ if (conflictOffset + length > blockSize) {
+ byte[] bytes = new byte[blockSize];
+ values.add(bytes);
+ conflictOffset = 0;
+ conflictIndex++;
+ }
+ byte[] bytes = values.get(conflictIndex);
+
+ CacheKV.MapAddress address = new CacheKV.MapAddress(conflictIndex, conflictOffset);
+ if (isVarLen) {
+ int size = value.length;
+ bytes[conflictOffset] = (byte)(size & 0xff);
+ bytes[conflictOffset + 1] = (byte)(size >> 8 & 0xff);
+ conflictOffset = conflictOffset + 2;
+ }
+ for (int i = 0; i < value.length; i++) {
+ try {
+ bytes[i + conflictOffset] = value[i];
+ } catch (Exception e) {
+ e.printStackTrace();
+ }
+
+ }
+ conflictOffset += value.length;
+ return address;
+ }
+
+ /**
+ * 根据当前地址获取对应的byte值
+ *
+ * @param mapAddress
+ * @return
+ */
+ public ByteArray getValue(CacheKV.MapAddress mapAddress) {
+ byte[] bytes = values.get(mapAddress.conflictIndex);
+ if (bytes == null) {
+ return null;
+ }
+ if (isVarLen == false) {
+ return new ByteArray(bytes, mapAddress.offset, elementSize);
+ } else {
+ int len = new ByteArray(bytes, mapAddress.offset, 2).castInt(0, 2);
+ return new ByteArray(bytes, mapAddress.offset + 2, len);
+ }
+ }
+
+ public int getConflictIndex() {
+ return conflictIndex;
+ }
+
+ public int getConflictOffset() {
+ return conflictOffset;
+ }
+
+ public int getBlockSize() {
+ return blockSize;
+ }
+
+ public void setConflictIndex(int conflictIndex) {
+ this.conflictIndex = conflictIndex;
+ }
+
+ public void setConflictOffset(int conflictOffset) {
+ this.conflictOffset = conflictOffset;
+ }
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/compress/ByteArray.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/compress/ByteArray.java
new file mode 100644
index 0000000..969d299
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/compress/ByteArray.java
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.cache.compress;
+
+import java.util.Arrays;
+
+public class ByteArray {
+
+ private byte[] bytes;
+ private int startIndex;
+ private int size;
+
+ public ByteArray(ByteArray byteArray, int startIndex, int size) {
+ this.bytes = byteArray.bytes;
+ this.startIndex = byteArray.startIndex + startIndex;
+ this.size = size;
+ }
+
+ public ByteArray(byte[] bytes) {
+ this.bytes = bytes;
+ this.startIndex = 0;
+ this.size = bytes.length;
+ }
+
+ public ByteArray(byte[] bytes, int startIndex, int size) {
+ this.bytes = bytes;
+ this.startIndex = startIndex;
+ this.size = size;
+ }
+
+ public ByteArray subByteArray(int offset, int size) {
+ return new ByteArray(bytes, startIndex + offset, size);
+ }
+
+ public byte[] getByteArray() {
+ return Arrays.copyOfRange(bytes, startIndex, startIndex + size);
+ }
+
+ public int castInt(int offset, int size) {
+ int index = startIndex + offset;
+ int res = 0;
+ for (int i = 0; i < size; i++) {
+ res += (bytes[i + index] & 0xff) << (i * 8);
+ }
+ return res;
+ }
+
+ public byte getByte(int offset) {
+ int index = startIndex + offset;
+ return bytes[index];
+ }
+
+ public int getSize() {
+ return size;
+ }
+
+ public void flush(CacheKV.MapAddress address) {
+ if (address == null) {
+ return;
+ }
+ if (address.isConflict == false) {
+ flush(address.offset);
+ } else {
+ flush(address.createBytes());
+ }
+ }
+
+ protected void flush(ByteArray bytes) {
+ for (int i = 0; i < bytes.size; i++) {
+ this.bytes[i + this.startIndex] = bytes.getByte(i);
+ }
+ }
+
+ protected void flush(byte[] bytes) {
+ for (int i = 0; i < bytes.length; i++) {
+ this.bytes[i + this.startIndex] = bytes[i];
+ }
+ }
+
+ protected void flush(int value) {
+ for (int i = 0; i < 4; i++) {
+ if (i == 0) {
+ this.bytes[i + this.startIndex] = (byte)(value & 0xff);
+ } else {
+ this.bytes[i + this.startIndex] = (byte)(value >> (i * 8) & 0xff);
+ }
+ }
+ }
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/compress/ByteArrayValueKV.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/compress/ByteArrayValueKV.java
new file mode 100644
index 0000000..0e3d789
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/compress/ByteArrayValueKV.java
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.cache.compress;
+
+import org.apache.rocketmq.streams.common.utils.NumberUtils;
+
+/**
+ * 支持key是string，value是int的场景，支持size不大于10000000.只支持int，long，boolean，string类型 只能一次行load，不能进行更新
+ */
+public class ByteArrayValueKV extends CacheKV<byte[]> {
+
+ protected final static String CODE = "UTF-8";
+ protected AdditionStore values;
+ protected boolean isFixedLength = false;//只有是定长时，才允许更新
+
+ public ByteArrayValueKV(int capacity, boolean isFixedLength) {
+ super(capacity);
+ this.isFixedLength = isFixedLength;
+ if (isFixedLength) {
+ values = new AdditionStore(capacity);
+ }
+ }
+
+ /**
+ * 直接存取byte数组
+ *
+ * @param key
+ * @return
+ */
+ @Override
+ public byte[] get(String key) {
+ ByteArray value = super.getInner(key);
+ if (value == null) {
+ return null;
+ }
+ MapAddress mapAddress = new MapAddress(value);
+ ByteArray byteArray = values.getValue(mapAddress);
+ return byteArray.getByteArray();
+ }
+
+ /**
+ * 如果是定长的字节，则判断已经有的value字节数和当前字节数相同，否则不允许插入
+ *
+ * @param key
+ * @param value
+ */
+ @Override
+ public void put(String key, byte[] value) {
+ if (key == null || value == null) {
+ return;
+ }
+
+ if (isFixedLength) {
+ byte[] oriValue = get(key);
+ if (oriValue != null) {
+ if (oriValue.length != value.length) {
+ throw new RuntimeException("the string must length equals ,but not。 the key is " + key + ", the ori value is " + oriValue + ", the put value is " + value);
+ }
+ }
+ }
+ MapAddress address = null;
+
+ int index = values.getConflictIndex();
+ int offset = values.getConflictOffset();
+ address = values.add2Store(value);
+ byte[] bytes = address.createBytes();
+ boolean success = super.putInner(key, NumberUtils.toInt(bytes), isFixedLength);
+ if (!success) {//不支持更新，如果存在已经有的key，则不插入，并回退刚插入的数据
+ values.setConflictOffset(offset);
+ values.setConflictIndex(index);
+ throw new RuntimeException("can not update value, the key has exist");
+ }
+
+ }
+
+ @Override
+ public int calMemory() {
+ int value = super.calMemory() + (this.conflicts.getConflictIndex() + 1) * this.conflicts
+ .getBlockSize();
+ return value;
+ }
+
+ @Override
+ public boolean contains(String key) {
+ byte[] bytes = get(key);
+ if (bytes == null) {
+ return false;
+ }
+ return true;
+ }
+
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/compress/CacheKV.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/compress/CacheKV.java
new file mode 100644
index 0000000..08bb056
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/compress/CacheKV.java
@@ -0,0 +1,371 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.cache.compress;
+
+import org.apache.rocketmq.streams.common.utils.AESUtil;
+import org.apache.rocketmq.streams.common.utils.NumberUtils;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+
+/**
+ * 支持key是string，value是int的场景，支持size不大于10000000.只支持int，long，boolean，string类型
+ */
+public abstract class CacheKV<T> implements ICacheKV<T> {
+ /**
+ * 如果存储int作为值，最大值不能超过这个值，如果超过，会改成冲突链的模式
+ */
+ protected static final int MAX_INT_VALUE = 2147483647;
+
+ /**
+ * 发生冲突的次数
+ */
+ protected int conflictCount;
+
+ protected int elementSize = 24;
+
+ /**
+ * 建议设置，如果不设置，默认容量为最大容量
+ */
+ protected int capacity = 8000000;
+
+ /**
+ * 元素个数
+ */
+ protected int size = 0;
+
+ /**
+ * 每一个元素分24个字节部分(下一个地址4个字节，key的md5值16个字节，int的值4个字节)
+ */
+ protected byte[] map;
+
+ /**
+ * 当发生hash冲突时，map值存储在这里
+ */
+ protected AdditionStore conflicts;
+
+ public CacheKV(int capacity, int elementSize) {
+ this.capacity = capacity;
+ this.elementSize = elementSize;
+ map = new byte[this.capacity * elementSize];
+ conflicts = new AdditionStore(elementSize);
+
+ }
+
+ public CacheKV(int capacity) {
+ this(capacity, 24);
+ }
+
+ @Override
+ public abstract T get(String key);
+
+ @Override
+ public abstract void put(String key, T value);
+
+ public T remove(String key) {
+ if (StringUtil.isEmpty(key)) { return null; }
+ MapElementContext context = queryMapElementByHashCode(key);
+ /**
+ * TODO:
+ * 1 清空MD5
+ * 2 前驱的后继=后继
+ * 3 当前链接到冲突域的最后一个节点后
+ */
+ return null;
+ }
+
+ public ByteArray getInner(String key) {
+ if (key == null) {
+ return null;
+ }
+ MapElementContext context = queryMapElementByHashCode(key);
+ if (context.isMatchKey) {
+ return context.mapElement.value;
+ }
+ return null;
+ }
+
+ /**
+ * address:是否冲突（1位），conflict index（7位），3个字节地址域名
+ *
+ * @param key
+ * @param value
+ */
+ public boolean putInner(String key, int value, boolean supportUpdate) {
+ if (key == null) {
+ return false;
+ }
+ MapElementContext context = null;
+ //符合hash值的最后一个next=null的节点或者是和当前key匹配的节点
+ context = queryMapElementByHashCode(key);
+
+ KVElement mapElement = context.mapElement;
+ //如果没有发生冲突，说明当前节点无被占用，直接写入
+ if (context.isOccurConflict == false) {
+ size++;
+
+ mapElement.keyHashCode.flush(mapElement.getKeyHashCode());
+ if (mapElement.isNoValue() == false) {
+ mapElement.value.flush(value);
+ }
+
+ NumberUtils.putSubByte2ByteArray(map, context.mapAddress.offset, mapElement.getBytes());
+ } else {
+ //如果key已经存在，覆盖value
+ if (context.isMatchKey) {
+ if (mapElement.isNoValue() == false) {
+ if (!supportUpdate) {
+ return false;
+ }
+ mapElement.value.flush(value);
+ }
+ } else {//如果是新key，且有hash冲突，把新值写入冲突域，把冲突域地址更新next地址
+ size++;
+ conflictCount++;
+ byte[] bytes = KVElement.createByteArray(new MapAddress(), context.keyMd5, value, elementSize);
+ MapAddress mapAddress = conflicts.add2Store(bytes);
+ context.mapElement.nextAddressByte.flush(mapAddress);
+
+ }
+ }
+ return true;
+ }
+
+ /**
+ * 查找同hashcode的冲突链，如果key的值等于当前key值，直接返回。如果key值不同，返回冲突链最后一个元素
+ *
+ * @param key
+ * @return
+ */
+ public MapElementContext queryMapElementByHashCode(String key) {
+ int offset = getElementIndex(key);
+ byte[] hashCodes = AESUtil.stringToMD5(key);
+
+ MapAddress address = new MapAddress(offset);//先从map中查找
+
+ MapElementContext context = getMapElementByAddress(hashCodes, address);
+
+ if (context.mapElement.isEmpty()) {//如果map中无值，直接返回
+ context.keyMd5 = hashCodes;
+ context.mapElement.setKeyHashCode(hashCodes);
+ return context;
+ } else {
+ while (true) {//通过链表找到最后一个节点
+ context.keyMd5 = hashCodes;
+ if (context.isMatchKey) {
+ return context;
+ } else {
+ // System.out.println("occure conflict");
+ context.mapAddress.isConflict = true;
+ if (context.mapElement.nextAddress.isEmpty()) {
+ return context;
+ } else {
+ context = getMapElementByAddress(hashCodes, context.mapElement.nextAddress);
+ }
+ }
+ }
+ }
+
+ }
+
+ @Override
+ public int calMemory() {//计算总共分配的内存
+ int totalMemory = this.capacity * elementSize + (this.conflicts.getConflictIndex() + 1) * this.conflicts
+ .getBlockSize();
+
+ return totalMemory / 1024 / 1024;
+ }
+
+ private boolean equalsByte(byte[] hashCodes, ByteArray keyHashCode) {//判读两个hash code是否相等
+ if (hashCodes == null || keyHashCode == null) {
+ return false;
+ }
+ if (hashCodes.length != keyHashCode.getSize()) {
+ return false;
+ }
+ for (int i = 0; i < hashCodes.length; i++) {
+ if (hashCodes[i] != keyHashCode.getByte(i)) {
+ return false;
+ }
+ }
+ return true;
+
+ }
+
+ @Override
+ public int getSize() {
+ return size;
+ }
+
+ /**
+ * 查找某个地址对应的map element
+ *
+ * @param hashCodes
+ * @param address
+ * @return
+ */
+ protected MapElementContext getMapElementByAddress(byte[] hashCodes, MapAddress address) {
+ if (address.isConflict) {
+ ByteArray valueBytes = this.conflicts.getValue(address);
+ KVElement mapElement = new KVElement(valueBytes);
+ mapElement.setElementSize(this.elementSize);
+ return new MapElementContext(mapElement, address, equalsByte(hashCodes, mapElement.keyHashCode));
+ } else {
+ KVElement mapElement = new KVElement(new ByteArray(map, address.offset, elementSize));
+ mapElement.setElementSize(this.elementSize);
+ return new MapElementContext(mapElement, address, equalsByte(hashCodes, mapElement.keyHashCode));
+ }
+ }
+
+ /**
+ * 获取map的索引，通过hashcode获取
+ *
+ * @param key
+ * @return
+ */
+ protected int getElementIndex(String key) {
+ if (key == null) {
+ return 0;
+ }
+ int hashCode;
+ int value = (hashCode = key.hashCode()) ^ (hashCode >>> 16);
+ value = String.valueOf(value).hashCode();
+ if (value < 0) {
+ value = -value;
+ }
+ int index = value % capacity;
+ return index * elementSize;
+ }
+
+ protected class MapElementContext {
+
+ /**
+ * 当前的元素
+ */
+ protected KVElement mapElement;
+
+ /**
+ * 元素所在的地址
+ */
+ protected MapAddress mapAddress;
+
+ /**
+ * 是否和key的hashcode 匹配
+ */
+ protected boolean isMatchKey = false;
+
+ /**
+ * 是否发生冲突
+ */
+ protected boolean isOccurConflict = false;
+
+ /**
+ * 本次查询key对应的hash code
+ */
+ protected byte[] keyMd5;
+
+ public MapElementContext(KVElement mapElement, MapAddress mapAddress, boolean isMatchKey) {
+ this.mapAddress = mapAddress;
+ this.mapElement = mapElement;
+ this.isMatchKey = isMatchKey;
+ if (mapElement.isEmpty() == false) {
+ isOccurConflict = true;
+ }
+ }
+
+ public boolean isMatchKey() {
+ return isMatchKey;
+ }
+ }
+
+ /**
+ * 地址表示，如果放在list<byte[]>中，isConflict=true，7个bit位构成list的index，后面3个字节代表具体byte[]的位置 如果直接放到byte[]中，isConflict=false，后面7个bit+3个字节构成byte数组的offset
+ */
+ protected static class MapAddress {
+
+ /**
+ * 是否在冲突域
+ */
+ protected boolean isConflict = false;
+
+ /**
+ * 冲突域的list index
+ */
+ protected int conflictIndex = 0;
+
+ /**
+ * list byte【】对应的offset
+ */
+ protected int offset = 0;
+
+ public MapAddress(int conflictIndex, int offset) {
+ this.isConflict = true;
+ this.conflictIndex = conflictIndex;
+ this.offset = offset;
+ }
+
+ public MapAddress(int offset) {
+ this.isConflict = false;
+ this.conflictIndex = -1;
+ this.offset = offset;
+ }
+
+ public MapAddress() {
+
+ }
+
+ public boolean isEmpty() {
+ return isConflict == false && conflictIndex == 0 && offset == 0;
+ }
+
+ /**
+ * 高位是0则全部四位代表在map中的偏移量，高位是1则最高位代表冲突域的索引，第三位代表在冲突域链表中的地址
+ *
+ * @param byteArray
+ */
+ public MapAddress(ByteArray byteArray) {
+ byte firstByte = 0;
+ firstByte = byteArray.getByte(byteArray.getSize() - 1);
+ // byte firstByte=byteArray.getByte(byteArray.getSize()-1);
+ int conflictValue = NumberUtils.toInt(firstByte);
+ int conflictFlag = conflictValue >> 7;
+ if (conflictFlag == 1) {
+ isConflict = true;
+ } else {
+ isConflict = false;
+ this.offset = byteArray.castInt(0, 4);
+ return;
+ }
+ //TODO 这个地址是不是每次相同？
+ this.conflictIndex = conflictValue & 127;
+ this.offset = byteArray.castInt(0, 3);
+ }
+
+ public byte[] createBytes() {
+ byte[] bytes = NumberUtils.toByte(offset);
+ int value = 0;
+ byte fisrtByte = (byte)(conflictIndex & 0xff);
+ if (isConflict) {
+ value = (fisrtByte | (1 << 7));//把第一位变成1
+ } else {
+ return bytes;
+ }
+
+ bytes[bytes.length - 1] = (byte)(value & 0xff);
+ return bytes;
+ }
+ }
+
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/compress/ICacheKV.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/compress/ICacheKV.java
new file mode 100644
index 0000000..9a766e0
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/compress/ICacheKV.java
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.cache.compress;
+
+/**
+ * kv提供的对外接口,通过二进制实现存储，减少java对象带来的头部开销。 需要指定初始容量，会在创建对象时分配内存。
+ *
+ */
+public interface ICacheKV<T> {
+
+ /**
+ * 根据key获取value
+ *
+ * @param key
+ * @return
+ */
+ T get(String key);
+
+ /**
+ * 把key，value插入到kv中
+ *
+ * @param key
+ * @param value
+ */
+ void put(String key, T value);
+
+ /**
+ * 是否包含
+ *
+ * @param key
+ * @return
+ */
+ boolean contains(String key);
+
+ /**
+ * 一共有多少行
+ *
+ * @return
+ */
+ int getSize();
+
+ /**
+ * 占用多少内存
+ *
+ * @return
+ */
+ int calMemory();
+
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/compress/KVElement.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/compress/KVElement.java
new file mode 100644
index 0000000..60f2f0b
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/compress/KVElement.java
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.cache.compress;
+
+import org.apache.rocketmq.streams.common.utils.NumberUtils;
+
+public class KVElement {
+
+ protected ByteArray nextAddressByte;
+
+ protected CacheKV.MapAddress nextAddress;
+
+ protected ByteArray keyHashCode;
+
+ /**
+ * value的值
+ */
+ protected ByteArray value;
+
+ /**
+ * 是否没有value
+ */
+ protected boolean noValue = false;
+
+ /**
+ * 元素个数
+ */
+ protected int elementSize = 24;
+
+ public KVElement(ByteArray byteArray) {
+ this.nextAddressByte = byteArray.subByteArray(0, 4);
+ nextAddress = new CacheKV.MapAddress(this.nextAddressByte);
+ this.keyHashCode = byteArray.subByteArray(4, 16);
+ if (!noValue) {
+ value = byteArray.subByteArray(20, 4);
+ }
+ }
+
+ public static byte[] createByteArray(CacheKV.MapAddress nextAddress, byte[] keyHashCode, int value,
+ int elementSize) {
+ KVElement element = new KVElement(nextAddress, keyHashCode, value);
+ element.setElementSize(elementSize);
+ return element.getBytes();
+ }
+
+ private KVElement(CacheKV.MapAddress nextAddress, byte[] keyHashCode, int value) {
+ this.nextAddress = nextAddress;
+ this.keyHashCode = new ByteArray(keyHashCode, 0, keyHashCode.length);
+ if (!noValue) {
+ this.value = new ByteArray(NumberUtils.toByte(value));
+ }
+
+ }
+
+ public boolean isEmpty() {
+ boolean empytHashCode = true;
+ for (int i = 0; i < keyHashCode.getSize(); i++) {
+ if (keyHashCode.getByte(i) != 0) {
+ empytHashCode = false;
+ break;
+ }
+ }
+ return (empytHashCode && nextAddress.isEmpty()) ? true : false;
+ }
+
+ public ByteArray getKeyHashCode() {
+ return keyHashCode;
+ }
+
+ public void setKeyHashCode(ByteArray keyHashCode) {
+ this.keyHashCode = keyHashCode;
+ }
+
+ public void setKeyHashCode(byte[] keyHashCode) {
+ this.keyHashCode = new ByteArray(keyHashCode);
+ }
+
+ public byte[] getBytes() {
+ byte[] bytes = new byte[elementSize];
+ NumberUtils.putSubByte2ByteArray(bytes, 0, nextAddress.createBytes());
+ NumberUtils.putSubByte2ByteArray(bytes, 4, keyHashCode.getByteArray());
+ if (!noValue) {
+ NumberUtils.putSubByte2ByteArray(bytes, 20, value.getByteArray());
+ }
+
+ return bytes;
+ }
+
+ public void setValue(int value) {
+ this.value = new ByteArray(NumberUtils.toByte(value));
+ }
+
+ public boolean isNoValue() {
+ return noValue;
+ }
+
+ public int getElementSize() {
+ return elementSize;
+ }
+
+ public KVElement setElementSize(int elementSize) {
+ this.elementSize = elementSize;
+ if (this.elementSize == 20) {
+ this.noValue = true;
+ }
+ return this;
+ }
+
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/compress/impl/FixedLenRowCacheKV.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/compress/impl/FixedLenRowCacheKV.java
new file mode 100644
index 0000000..71e8062
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/compress/impl/FixedLenRowCacheKV.java
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.cache.compress.impl;
+
+import org.apache.rocketmq.streams.common.cache.compress.ByteArrayValueKV;
+import org.apache.rocketmq.streams.common.cache.compress.ICacheKV;
+import org.apache.rocketmq.streams.common.datatype.*;
+import org.apache.rocketmq.streams.common.utils.NumberUtils;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * 常用类型的存储，目前只支持，float，int，long，double，set<String>，date，byte，short,boolean 存入一行数据，且必须是定长的行
+ */
+public class FixedLenRowCacheKV {
+ protected static final int MAX_ELIMENT = 3000000;//最大的元素个数
+ protected DataType[] datatypes;
+ protected int byteLength = -1;
+ protected MultiValueKV<byte[]> cache;//每块存储固定，超过后，用多块存储
+
+ protected Map<Integer, MultiValueKV> keySetMap = new HashMap<>();//存对应的set值。每个KeySet有最大值，超过最大值需要多个对象存储
+
+ public FixedLenRowCacheKV(DataType... datatypes) {
+ this.datatypes = datatypes;
+ if (datatypes != null) {
+ int index = 0;
+ int byteLength = 0;
+ for (DataType datatype : datatypes) {
+ if (datatype instanceof SetDataType) {
+ keySetMap.put(index, new MultiValueKV(MAX_ELIMENT) {
+ @Override
+ protected ICacheKV create() {
+ return new KeySet(this.capacity);
+ }
+ });
+ byteLength++;
+ } else if (datatype instanceof FloatDataType || datatype instanceof DoubleDataType || datatype instanceof LongDataType || datatype instanceof DateDataType) {
+ byteLength = byteLength + 8;
+ } else if (datatype instanceof IntDataType) {
+ byteLength = byteLength + 4;
+ } else if (datatype instanceof ByteDataType || datatype instanceof BooleanDataType) {
+ byteLength++;
+ } else if (datatype instanceof ShortDataType) {
+ byteLength = byteLength + 2;
+ }
+ index++;
+ }
+ }
+ cache = new MultiValueKV<byte[]>(MAX_ELIMENT) {
+ @Override
+ protected ICacheKV<byte[]> create() {
+ return new ByteArrayValueKV(this.capacity, true);
+ }
+ };
+ }
+
+ /**
+ * 写入一行数据
+ *
+ * @param key
+ * @param values
+ */
+ public void put(String key, Object... values) {
+ byte[] value = createBytes(values);
+ cache.put(key, value);
+ }
+
+ /**
+ * 获取一行数据
+ *
+ * @param key
+ * @return
+ */
+ public Object[] get(String key) {
+ byte[] value = cache.get(key);
+ if (value == null) {
+ return null;
+ }
+ Object[] result = new Object[datatypes.length];
+ AtomicInteger offset = new AtomicInteger(0);
+ for (int i = 0; i < result.length; i++) {
+ DataType dataType = datatypes[i];
+ if (dataType instanceof SetDataType) {
+ MultiValueKV multiValueKV = keySetMap.get(i);
+ result[i] = multiValueKV;
+ } else {
+ result[i] = dataType.byteToValue(value, offset);
+ }
+
+ }
+ return result;
+ }
+
+ /**
+ * 把具体的值，转化成固定长度的byte数组
+ *
+ * @param values
+ * @return
+ */
+ public byte[] createBytes(Object... values) {
+ if (values == null || values.length == 0) {
+ return null;
+ }
+ byte[] bytes = new byte[byteLength];
+ int index = 0;
+ for (int i = 0; i < values.length; i++) {
+ DataType dataType = datatypes[i];
+ Object o = values[i];
+ byte[] byteValue = null;
+ if (dataType instanceof SetDataType) {
+ byteValue = new byte[] {(byte)i};
+ add2Set(i, (Set<String>)o);
+ } else {
+ byteValue = dataType.toBytes(o, false);
+ }
+ NumberUtils.putSubByte2ByteArray(bytes, index, byteValue);
+ index = index + byteValue.length;
+ }
+ return bytes;
+ }
+
+ /**
+ * 插入数据到set中
+ *
+ * @param i 第几个变量
+ * @param setValue
+ */
+ protected void add2Set(int i, Set<String> setValue) {
+ }
+
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/compress/impl/IntValueKV.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/compress/impl/IntValueKV.java
new file mode 100644
index 0000000..f2ce1f7
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/compress/impl/IntValueKV.java
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.cache.compress.impl;
+
+import org.apache.rocketmq.streams.common.cache.compress.AdditionStore;
+import org.apache.rocketmq.streams.common.cache.compress.ByteArray;
+import org.apache.rocketmq.streams.common.cache.compress.CacheKV;
+import org.apache.rocketmq.streams.common.utils.NumberUtils;
+import org.junit.Assert;
+
+import java.io.BufferedReader;
+import java.io.FileInputStream;
+import java.io.InputStreamReader;
+import java.util.*;
+import java.util.Map.Entry;
+
+/**
+ * 支持key是string，value是int的场景，支持size不大于10000000.只支持int，long，boolean，string类型
+ *
+ */
+public class IntValueKV extends CacheKV<Integer> {
+
+ protected AdditionStore conflicts = new AdditionStore(4);
+
+ @Override
+ public Integer get(String key) {
+ ByteArray byteArray = super.getInner(key);
+ if (byteArray == null) {
+ return null;
+ }
+ int value = byteArray.castInt(0, 4);
+ if (value < MAX_INT_VALUE) {
+ return value;
+ }
+ MapAddress mapAddress = new MapAddress(byteArray);
+ ByteArray bytes = conflicts.getValue(mapAddress);
+ return bytes.castInt(0, 4);
+ }
+
+ @Override
+ public void put(String key, Integer value) {
+ if (value < MAX_INT_VALUE) {
+ super.putInner(key, value, true);
+ return;
+ }
+ MapAddress address = conflicts.add2Store(NumberUtils.toByte(value));
+ super.putInner(key, NumberUtils.toInt(address.createBytes()), true);
+
+ }
+
+ @Override
+ public boolean contains(String key) {
+ Integer value = get(key);
+ if (value == null) {
+ return false;
+ }
+ return true;
+
+ }
+
+ @Override
+ public int calMemory() {
+ int value = super.calMemory() + (this.conflicts.getConflictIndex() + 1) * this.conflicts
+ .getBlockSize();
+ return value;
+ }
+
+ /**
+ * TODO remove the key from the sinkcache and return the removed value
+ *
+ * @return
+ */
+ //public Integer remove(String key) {
+ // return null;
+ //}
+ public IntValueKV(int capacity) {
+ super(capacity);
+ }
+
+ public static void main(String[] args) throws Exception {
+ IntValueKV cache = new IntValueKV(5);
+ cache.put("A", 0);
+ cache.put("B", 1);
+ cache.put("C", 2);
+ cache.put("D", 3);
+ cache.put("E", 4);
+ cache.put("F", 5);
+ cache.put("G", 6);
+
+ System.exit(0);
+
+ int size = 10000000;
+ int sampleSize = 1024;
+ int dataSize = 3974534;
+ IntValueKV compressByteMap = new IntValueKV(size);
+ Map<String, Integer> dataMap = new HashMap<>(size);
+ Set<Integer> whiteSet = new HashSet<>(1024);
+ Map<String, Integer> sample1Map = new HashMap<>(1024);
+ Map<String, Integer> sample2Map = new HashMap<>(1024);
+ //init data
+ Random random = new Random();
+ while (true) {
+ if (whiteSet.size() >= sampleSize) {
+ break;
+ }
+ int seed = random.nextInt(dataSize);
+ if (!whiteSet.contains(seed)) {
+ whiteSet.add(seed);
+ }
+ }
+
+ long originWriteCounter = 0;
+ long compressWriteCounter = 0;
+ String path = "/Users/arthur/Downloads/";
+ String blackFile = "2020-11-11-14-08-32_EXPORT_CSV_16231630_392_0.csv";
+ try (BufferedReader reader = new BufferedReader(new InputStreamReader(new FileInputStream(path + blackFile)))) {
+ reader.readLine();
+ String line = null;
+ int counter = 0;
+ while ((line = reader.readLine()) != null) {
+ line = line.replaceAll("\"", "");
+ String[] parts = line.split(",", 2);
+ long begin = System.nanoTime();
+ dataMap.put(parts[1].trim(), Integer.parseInt(parts[0]));
+ originWriteCounter += (System.nanoTime() - begin);
+ if (whiteSet.contains(counter++)) {
+ sample1Map.put(parts[1].trim(), Integer.parseInt(parts[0]));
+ }
+ }
+ }
+ for (int i = 0; i < sampleSize * 100; i++) {
+ sample2Map.put(UUID.randomUUID().toString(), -1);
+ }
+ System.out.println("sample1 size:\t" + sample1Map.size());
+ System.out.println("sample2 size:\t" + sample2Map.size());
+ //System.out.println(
+ // "origin map size(computed by third party):\t" + RamUsageEstimator.humanSizeOf(dataMap) + "\tline's\t"
+ // + dataMap.size());
+ //
+ Iterator<Entry<String, Integer>> iterator = dataMap.entrySet().iterator();
+ while (iterator.hasNext()) {
+ Entry<String, Integer> entry = iterator.next();
+ long begin = System.nanoTime();
+ compressByteMap.put(entry.getKey(), entry.getValue());
+ compressWriteCounter += (System.nanoTime() - begin);
+ }
+ //System.out.println(
+ // "compressed map size(computed by third party):\t" + RamUsageEstimator.humanSizeOf(compressByteMap)
+ // + "\tline's\t"
+ // + compressByteMap.size);
+ System.out.println("compressed map size(computed by it's self)\t" + compressByteMap.calMemory() + " MB");
+ System.out.println(
+ "origin write cost:\t" + originWriteCounter / 1000 + "\tcompress write cost:\t"
+ + compressWriteCounter / 1000);
+ //
+ long originSearchCounter = 0;
+ long compressCounter = 0;
+ Iterator<Entry<String, Integer>> iterator1 = sample1Map.entrySet().iterator();
+ Iterator<Entry<String, Integer>> iterator2 = sample2Map.entrySet().iterator();
+ while (iterator1.hasNext() && iterator2.hasNext()) {
+ Entry<String, Integer> entry1 = iterator1.next();
+ String key1 = entry1.getKey();
+ Integer value1 = entry1.getValue();
+ Entry<String, Integer> entry2 = iterator2.next();
+ String key2 = entry2.getKey();
+ Integer value2 = entry2.getValue();
+ long begin = System.nanoTime();
+ Assert.assertEquals(value1, dataMap.get(key1));
+ Assert.assertNotEquals(value2, dataMap.get(key2));
+ originSearchCounter += (System.nanoTime() - begin);
+ begin = System.nanoTime();
+ Assert.assertEquals(value1, compressByteMap.get(key1));
+ Assert.assertNotEquals(value2, compressByteMap.get(key2));
+ compressCounter += (System.nanoTime() - begin);
+ }
+ System.out.println(
+ "origin search cost:\t" + originSearchCounter / 1000 + "\tcompress search cost:\t"
+ + compressCounter / 1000);
+ }
+
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/compress/impl/KeySet.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/compress/impl/KeySet.java
new file mode 100644
index 0000000..0e5f278
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/compress/impl/KeySet.java
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.cache.compress.impl;
+
+import org.apache.rocketmq.streams.common.cache.compress.CacheKV;
+import org.apache.rocketmq.streams.common.cache.compress.ICacheKV;
+
+/**
+ * 支持key是string，value是int的场景，支持size不大于10000000.只支持int，long，boolean，string类型
+ */
+public class KeySet implements ICacheKV {
+
+ protected CacheKV cacheKV;
+
+ public KeySet(int capacity) {
+ cacheKV = new CacheKV(capacity, 20) {
+ @Override
+ public Object get(String key) {
+ boolean exist = contains(key);
+ if (exist) {
+ return key;
+ }
+ return null;
+ }
+
+ @Override
+ public void put(String key, Object value) {
+ putInner(key, 1, true);
+ }
+
+ @Override
+ public boolean contains(String key) {
+ MapElementContext context = queryMapElementByHashCode(key);
+ if (context.isMatchKey()) {
+ return true;
+ }
+ return false;
+ }
+
+ };
+ }
+
+ public void add(String key) {
+ cacheKV.putInner(key, 1, true);
+ }
+
+ @Override
+ public Object get(String key) {
+ return cacheKV.get(key);
+ }
+
+ @Override
+ public void put(String key, Object value) {
+ cacheKV.put(key, value);
+ }
+
+ @Override
+ public boolean contains(String key) {
+ return cacheKV.contains(key);
+ }
+
+ @Override
+ public int getSize() {
+ return cacheKV.getSize();
+ }
+
+ @Override
+ public int calMemory() {
+ return cacheKV.calMemory();
+ }
+
+ public static void main(String[] args) throws InterruptedException {
+ int count = 10000000;
+ KeySet map = new KeySet(count);
+ long start = System.currentTimeMillis();
+ // Set<String> values=new HashSet<>();
+ //Map<String,Integer> kv=new HashMap<>();
+ for (int i = 0; i < count; i++) {
+ map.add("sdfsdfdds" + i);
+ ;
+ }
+ System.out.println("cost memory:\t" + map.calMemory() + "M");
+ // System.out.println(values.size());
+ System.out.println("cost is:\t" + (System.currentTimeMillis() - start));
+ start = System.currentTimeMillis();
+
+ for (int i = 0; i < count; i++) {
+ boolean v = map.contains("sdfsdfdds" + i);
+ if (!v) {
+ throw new RuntimeException("");
+ }
+ }
+ System.out.println("cost is:\t" + (System.currentTimeMillis() - start));
+ }
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/compress/impl/ListValueKV.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/compress/impl/ListValueKV.java
new file mode 100644
index 0000000..40b54bd
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/compress/impl/ListValueKV.java
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.cache.compress.impl;
+
+import org.apache.rocketmq.streams.common.cache.compress.CacheKV;
+import org.apache.rocketmq.streams.common.datatype.DataType;
+import org.apache.rocketmq.streams.common.datatype.ListDataType;
+import org.apache.rocketmq.streams.common.utils.DataTypeUtil;
+
+import java.util.List;
+
+/**
+ * 支持key是string，value是int的场景，支持size不大于10000000.只支持int，long，boolean，string类型
+ */
+public class ListValueKV<T> extends CacheKV<List<T>> {
+
+ protected StringValueKV stringValueKV;
+ protected DataType<T> datatype;
+ protected ListDataType listDataType;
+ protected transient Class elementClass;
+
+ public ListValueKV(int capacity, Class elementClass) {
+ super(0);
+ stringValueKV = new StringValueKV(capacity, false);
+ datatype = DataTypeUtil.getDataTypeFromClass(elementClass);
+ listDataType = new ListDataType(datatype);
+ this.elementClass = elementClass;
+ }
+
+ @Override
+ public List<T> get(String key) {
+ String value = stringValueKV.get(key);
+ if (value == null) {
+ return null;
+ }
+
+ return listDataType.getData(value);
+ }
+
+ @Override
+ public void put(String key, List<T> values) {
+ String value = listDataType.toDataJson(values);
+ ;
+ stringValueKV.put(key, value);
+ }
+
+ @Override
+ public boolean contains(String key) {
+ return stringValueKV.contains(key);
+ }
+
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/compress/impl/MultiValueKV.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/compress/impl/MultiValueKV.java
new file mode 100644
index 0000000..7881733
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/compress/impl/MultiValueKV.java
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.cache.compress.impl;
+
+import org.apache.rocketmq.streams.common.cache.compress.ICacheKV;
+
+import java.util.ArrayList;
+import java.util.List;
+
+public abstract class MultiValueKV<T> implements ICacheKV<T> {
+ //按固定大小分割存储
+ protected List<ICacheKV<T>> valueKVS = new ArrayList<>();
+ //当前存储的索引
+ protected int currentIndex = 0;
+ //每个分片的大小
+ protected int capacity;
+
+ public MultiValueKV(int capacity) {
+ this.capacity = capacity;
+ }
+
+ @Override
+ public T get(String key) {
+ if (valueKVS == null) {
+ return null;
+ }
+ for (ICacheKV<T> cacheKV : valueKVS) {
+ if (cacheKV != null) {
+ T value = cacheKV.get(key);
+ if (value != null) {
+ return value;
+ }
+ }
+ }
+ return null;
+ }
+
+ @Override
+ public void put(String key, T value) {
+ if (valueKVS == null) {
+ return;
+ }
+ ICacheKV<T> cacheKV = valueKVS.get(currentIndex);
+ if (cacheKV.getSize() >= capacity) {
+ synchronized (this) {
+ cacheKV = valueKVS.get(currentIndex);
+ if (cacheKV.getSize() >= capacity) {
+ cacheKV = create();
+ valueKVS.add(cacheKV);
+ currentIndex++;
+ }
+ }
+ }
+ cacheKV.put(key, value);
+ }
+
+ @Override
+ public boolean contains(String key) {
+ if (valueKVS == null) {
+ return false;
+ }
+ for (ICacheKV<T> cacheKV : valueKVS) {
+ if (cacheKV != null) {
+ boolean isMatch = cacheKV.contains(key);
+ if (isMatch) {
+ return true;
+ }
+ }
+ }
+ return false;
+ }
+
+ protected abstract ICacheKV<T> create();
+
+ @Override
+ public int getSize() {
+ return 0;
+ }
+
+ @Override
+ public int calMemory() {
+ return 0;
+ }
+
+ public int getCurrentIndex() {
+ return currentIndex;
+ }
+
+ public void setCurrentIndex(int currentIndex) {
+ this.currentIndex = currentIndex;
+ }
+
+ public int getCapacity() {
+ return capacity;
+ }
+
+ public void setCapacity(int capacity) {
+ this.capacity = capacity;
+ }
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/compress/impl/MutilValueKV.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/compress/impl/MutilValueKV.java
new file mode 100644
index 0000000..3065a46
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/compress/impl/MutilValueKV.java
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.streams.common.cache.compress.impl;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.rocketmq.streams.common.cache.compress.ICacheKV;
+
+public abstract class MutilValueKV<T> implements ICacheKV<T> {
+ //按固定大小分割存储
+ protected List<ICacheKV<T>> valueKVS=new ArrayList<>();
+ //当前存储的索引
+ protected int currentIndex=0;
+ //每个分片的大小
+ protected int capacity;
+
+ public MutilValueKV(int capacity){
+ this.capacity=capacity;
+ }
+
+
+ @Override
+ public T get(String key) {
+ if(valueKVS==null){
+ return null;
+ }
+ for(ICacheKV<T> cacheKV:valueKVS){
+ if(cacheKV!=null){
+ T value=cacheKV.get(key);
+ if(value!=null){
+ return value;
+ }
+ }
+ }
+ return null;
+ }
+
+ @Override
+ public void put(String key, T value) {
+ if(valueKVS==null){
+ return;
+ }
+ ICacheKV<T> cacheKV= valueKVS.get(currentIndex);
+ if(cacheKV.getSize()>=capacity){
+ synchronized (this){
+ cacheKV= valueKVS.get(currentIndex);
+ if(cacheKV.getSize()>=capacity){
+ cacheKV=create();
+ valueKVS.add(cacheKV);
+ currentIndex++;
+ }
+ }
+ }
+ cacheKV.put(key,value);
+ }
+
+ @Override
+ public boolean contains(String key) {
+ if(valueKVS==null){
+ return false;
+ }
+ for(ICacheKV<T> cacheKV:valueKVS){
+ if(cacheKV!=null){
+ boolean isMatch=cacheKV.contains(key);
+ if(isMatch){
+ return true;
+ }
+ }
+ }
+ return false;
+ }
+
+
+ protected abstract ICacheKV<T> create();
+
+ @Override
+ public int getSize() {
+ return 0;
+ }
+
+ @Override
+ public int calMemory() {
+ return 0;
+ }
+
+ public int getCurrentIndex() {
+ return currentIndex;
+ }
+
+ public void setCurrentIndex(int currentIndex) {
+ this.currentIndex = currentIndex;
+ }
+
+ public int getCapacity() {
+ return capacity;
+ }
+
+ public void setCapacity(int capacity) {
+ this.capacity = capacity;
+ }
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/compress/impl/SplitCache.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/compress/impl/SplitCache.java
new file mode 100644
index 0000000..59aa08a
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/compress/impl/SplitCache.java
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.cache.compress.impl;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+
+/**
+ * sinkcache(key:String,value:String) for special scene which read operation more than write operation, it's not thread safe
+ */
+public class SplitCache<T> {
+
+ private int capacity;
+
+ /**
+ * TODO: try huffman code?
+ */
+ private List<byte[]> values;
+
+ private IntValueKV keys;
+
+ private SplitCache() {
+
+ }
+
+ public SplitCache(int capacity) {
+ this.capacity = capacity;
+ values = new ArrayList<>(capacity);
+ keys = new IntValueKV(capacity);
+ }
+
+ public void put(String key, String value) {
+ int index = Optional.ofNullable(keys.get(key)).orElse(-1);
+ if (index > -1 && index < capacity) {
+ values.set(index, value.getBytes());
+ } else {
+ values.add(value.getBytes());
+ keys.put(key, values.size() - 1);
+ }
+ }
+
+ public String get(String key) {
+ int index = Optional.ofNullable(keys.get(key)).orElse(-1);
+ return (index > -1 && index < capacity) ? new String(values.get(index)) : null;
+ }
+
+ public String remove(String key) {
+ int index = Optional.ofNullable(keys.get(key)).orElse(-1);
+ String value = null;
+ if (index > -1 && index < capacity) {
+ value = new String(values.get(index));
+ //TODO remove from keys
+ values.remove(index);
+ }
+ return value;
+ }
+
+ public int getSize() {
+ return values.size();
+ }
+
+ public void clear() {
+ values = new ArrayList<>(capacity);
+ keys = new IntValueKV(capacity);
+ }
+
+ public int getCapacity() {
+ return capacity;
+ }
+
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/compress/impl/StringValueKV.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/compress/impl/StringValueKV.java
new file mode 100644
index 0000000..341cfbe
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/compress/impl/StringValueKV.java
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.cache.compress.impl;
+
+import org.apache.rocketmq.streams.common.cache.compress.ByteArrayValueKV;
+import org.apache.rocketmq.streams.common.cache.compress.ICacheKV;
+import org.junit.Assert;
+
+import java.io.UnsupportedEncodingException;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * 支持key是string，value是int的场景，支持size不大于10000000.只支持int，long，boolean，string类型 只能一次行load，不能进行更新
+ */
+public class StringValueKV implements ICacheKV<String> {
+
+ protected final static String CODE = "UTF-8";
+ protected ByteArrayValueKV values;
+
+ public StringValueKV(int capacity, boolean isFixedLength) {
+ values = new ByteArrayValueKV(capacity, isFixedLength);
+ }
+
+ @Override
+ public String get(String key) {
+ byte[] bytes = values.get(key);
+ if (bytes == null) {
+ return null;
+ }
+ try {
+ return new String(bytes, CODE);
+ } catch (Exception e) {
+ throw new RuntimeException("can not convert byte 2 string ", e);
+ }
+ }
+
+ @Override
+ public void put(String key, String value) {
+
+ try {
+ byte[] bytes = value.getBytes(CODE);
+ values.put(key, bytes);
+ } catch (UnsupportedEncodingException e) {
+ throw new RuntimeException("can not convert byte 2 string ", e);
+ }
+
+ }
+
+ @Override
+ public boolean contains(String key) {
+ return values.contains(key);
+ }
+
+ @Override
+ public int getSize() {
+ return values.getSize();
+ }
+
+ @Override
+ public int calMemory() {
+ return values.calMemory();
+ }
+
+ public static void main(String[] args) throws InterruptedException {
+ int count = 10000000;
+
+ StringValueKV map = new StringValueKV(count, false);
+ long start = System.currentTimeMillis();
+ for (int i = 0; i < count; i++) {
+ map.put("sdfsdfdds" + i, i + "");
+ }
+ System.out.println("fixed value size: " + map.getSize());
+ //System.out.println("fixed value memory: " + RamUsageEstimator.humanSizeOf(map));
+ System.out.println("fixed value write cost: " + (System.currentTimeMillis() - start));
+
+ start = System.currentTimeMillis();
+ SplitCache splitCache = new SplitCache(count);
+ for (int i = 0; i < count; i++) {
+ splitCache.put("sdfsdfdds" + i, i + "");
+ }
+ System.out.println("free value size: " + splitCache.getSize());
+ // System.out.println("free value memory: " + RamUsageEstimator.humanSizeOf(splitCache));
+ System.out.println("free value cost: " + (System.currentTimeMillis() - start));
+
+ start = System.currentTimeMillis();
+ Map<String, String> originMap = new HashMap<>(count);
+ for (int i = 0; i < count; i++) {
+ originMap.put("sdfsdfdds" + i, i + "");
+ }
+ System.out.println("origin map size: " + originMap.size());
+ // System.out.println("origin map memory: " + RamUsageEstimator.humanSizeOf(originMap));
+ System.out.println("origin map cost: " + (System.currentTimeMillis() - start));
+
+ start = System.currentTimeMillis();
+ for (int i = 0; i < count; i++) {
+ String v = map.get("sdfsdfdds" + i);
+ Assert.assertEquals(v, i + "");
+ v = map.get("asdfasdf" + i);
+ Assert.assertNull(v);
+ }
+ System.out.println("fix value read cost: " + (System.currentTimeMillis() - start));
+
+ start = System.currentTimeMillis();
+ for (int i = 0; i < count; i++) {
+ String v = splitCache.get("sdfsdfdds" + i);
+ Assert.assertEquals(v, i + "");
+ v = splitCache.get("asdfasdf" + i);
+ Assert.assertNull(v);
+ }
+ System.out.println("free value read cost: " + (System.currentTimeMillis() - start));
+
+ start = System.currentTimeMillis();
+ for (int i = 0; i < count; i++) {
+ String v = originMap.get("sdfsdfdds" + i);
+ Assert.assertEquals(v, i + "");
+ v = originMap.get("asdfasdf" + i);
+ Assert.assertNull(v);
+ }
+ System.out.println("origin map read cost: " + (System.currentTimeMillis() - start));
+ }
+
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/softreference/ICache.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/softreference/ICache.java
new file mode 100644
index 0000000..7f97a34
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/softreference/ICache.java
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.cache.softreference;
+
+import java.util.List;
+
+/**
+ * 基于软引用实现的缓存，可以在内存不足时回收，尽量最大限度使用内存的场景使用
+ */
+public interface ICache<K, V> {
+
+ /**
+ * get driver
+ *
+ * @param k
+ * @return
+ */
+ V get(K k);
+
+ /**
+ * put driver
+ *
+ * @param k
+ * @param v
+ */
+ void put(K k, V v);
+
+ /**
+ * remove driver which return the removed value
+ *
+ * @param k
+ * @return
+ */
+ V remove(K k);
+
+ /**
+ * remove all keys which has same prefix and return the removed value in the list
+ *
+ * @param k
+ * @return
+ */
+ List<V> removeByPrefix(K k);
+
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/softreference/RebuildCacheElement.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/softreference/RebuildCacheElement.java
new file mode 100644
index 0000000..5f923dd
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/softreference/RebuildCacheElement.java
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.cache.softreference;
+
+/**
+ * 缓存失效后，重新创建对象
+ */
+public interface RebuildCacheElement<K, V> {
+
+ V create(K k);
+
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/softreference/impl/SoftReferenceCache.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/softreference/impl/SoftReferenceCache.java
new file mode 100644
index 0000000..6a930fd
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/cache/softreference/impl/SoftReferenceCache.java
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.cache.softreference.impl;
+
+import org.apache.rocketmq.streams.common.cache.softreference.ICache;
+import org.apache.rocketmq.streams.common.cache.softreference.RebuildCacheElement;
+
+import java.lang.ref.ReferenceQueue;
+import java.lang.ref.SoftReference;
+import java.util.List;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * 基于软引用实现的缓存，可以在内存不足时回收，尽量最大限度使用内存的场景使用
+ */
+public class SoftReferenceCache<K, V> implements ICache<K, V> {
+
+ protected RebuildCacheElement<K, V> rebuildCacheElement;
+
+ public SoftReferenceCache(RebuildCacheElement<K, V> rebuildCacheElement) {
+ this.rebuildCacheElement = rebuildCacheElement;
+ }
+
+ public SoftReferenceCache() {
+ this.rebuildCacheElement = new RebuildCacheElement<K, V>() {
+ @Override
+ public V create(K k) {
+ return null;
+ }
+ };
+ }
+
+ protected class ExtraInfoReference<T> extends SoftReference<T> {
+
+ private Object info;
+
+ public ExtraInfoReference(Object info, T t, ReferenceQueue<T> refQueue) {
+ super(t, refQueue);
+ this.info = info;
+ }
+
+ public Object getExtraInfo() {
+ return this.info;
+ }
+ }
+
+ // 缓存，用软引用记录
+ private ConcurrentHashMap<K, ExtraInfoReference<V>> cache = new ConcurrentHashMap<K, ExtraInfoReference<V>>();
+ private ReferenceQueue<V> refQueue = new ReferenceQueue<V>();
+
+ @Override
+ public V get(K key) {
+ V value = null;
+
+ if (cache.containsKey(key)) {
+ SoftReference<V> refValue = cache.get(key);
+ if (refValue != null) {
+ value = refValue.get();
+ }
+
+ }
+ // 如果软引用被回收
+ if (value == null) {
+ // 清除软引用队列
+ clearRefQueue();
+ // 创建值并放入缓存
+ value = rebuildCacheElement.create(key);
+ if (value != null) {
+ ExtraInfoReference<V> refValue = new ExtraInfoReference<V>(key, value, refQueue);
+ cache.put(key, refValue);
+ }
+ }
+
+ return value;
+ }
+
+ @Override
+ public void put(K k, V v) {
+ if (v == null) {
+ cache.remove(k);
+ }
+ ExtraInfoReference<V> refValue = new ExtraInfoReference<V>(k, v, refQueue);
+ cache.put(k, refValue);
+ }
+
+ @Override
+ public V remove(K k) {
+ return null;
+ }
+
+ @Override
+ public List<V> removeByPrefix(K k) {
+ return null;
+ }
+
+ public void clear() {
+ clearRefQueue();
+ cache.clear();
+ }
+
+ /**
+ * 从软引用队列中移除无效引用， 同时从cache中删除无效缓存
+ */
+ @SuppressWarnings("unchecked")
+ protected void clearRefQueue() {
+ ExtraInfoReference<V> refValue = null;
+ while ((refValue = (ExtraInfoReference<V>)refQueue.poll()) != null) {
+ K key = (K)refValue.getExtraInfo();
+ cache.remove(key);
+ }
+ }
+
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/calssscaner/AbstractScan.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/calssscaner/AbstractScan.java
new file mode 100644
index 0000000..5b9f58e
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/calssscaner/AbstractScan.java
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.calssscaner;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.rocketmq.streams.common.utils.FileUtil;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+
+import java.io.*;
+import java.net.URL;
+import java.util.*;
+import java.util.jar.JarEntry;
+import java.util.jar.JarFile;
+
+public abstract class AbstractScan {
+
+ private static final Log LOG = LogFactory.getLog(AbstractScan.class);
+
+ private static final String CLASS_REAR = ".class";
+
+ protected Set<String> scanDirs = new HashSet<>();
+
+ public void scanClassDir(File jarFile, String packageName, ClassLoader classLoader) {
+ scanClassInJar(jarFile.getAbsolutePath(), packageName, classLoader);
+ }
+
+ public void scanClassDir(String dir, String packageName, ClassLoader classLoader) {
+ if (StringUtil.isEmpty(dir)) {
+ return;
+ }
+ if (classLoader == null) {
+ classLoader = this.getClass().getClassLoader();
+ }
+ if (!dir.endsWith(File.separator)) {
+ dir += File.separator;
+ }
+ if (packageName.contains(".")) {
+ dir += packageName.replace(".", File.separator);
+ }
+ File dirs = new File(dir);
+ if (!dirs.exists()) {
+ return;
+ }
+
+ File[] files = dirs.listFiles();
+ if (files.length == 0) {
+ return;
+ }
+ for (File file : files) {
+ try {
+ String className = file.getName();
+ if (className.endsWith(CLASS_REAR)) {
+ Class clazz = classLoader.loadClass(packageName + "." + className.replace(CLASS_REAR, ""));
+ doProcessor(clazz);
+ }
+ } catch (ClassNotFoundException e) {
+ LOG.error("load class error " + file.getName(), e);
+ continue;
+ }
+ }
+ }
+
+ protected void scanDir(String dir) {
+ if (inJar(dir)) {
+ scanClassInJar(dir);
+ } else {
+ scanClassInDir(dir);
+ }
+ }
+
+ public void scanPackages(String... packageNames) {
+ if (packageNames != null) {
+ for (String packageName : packageNames) {
+ scanPackage(packageName);
+ }
+ }
+ }
+
+ public void scanPackage(String packageName) {
+ if (scanDirs.contains(packageName)) {
+ return;
+ }
+ scanDirs.add(packageName);
+ String packageDir = "/" + packageName.replace(".", "/");
+ Set<String> hasScan = new HashSet<>();
+ if (inJar(packageDir)) {
+ scanClassInJar(packageDir);
+ } else {
+ List<String> dirs = scanPackageDir(packageDir);
+ if (dirs == null) {
+ return;
+ }
+
+ for (String dir : dirs) {
+ if (hasScan.contains(dir)) {
+ continue;
+ } else {
+ scanDir(dir);
+ hasScan.add(dir);
+ }
+
+ }
+ }
+
+ }
+
+ protected List<String> scanPackageDir(String packageDir) {
+ try {
+
+ URL url = this.getClass().getResource(packageDir);
+ if (url == null) {
+ return null;
+ }
+ BufferedReader br = new BufferedReader(new InputStreamReader(url.openStream()));
+ String line = br.readLine();
+ boolean hasScane = false;
+ List<String> dirs = new ArrayList<>();
+ while (line != null) {
+ if (!line.endsWith(CLASS_REAR)) {
+ String dir = packageDir + "/" + line;
+ File fileDir = new File(url.getPath() + "/" + line);
+ if (fileDir.isDirectory()) {
+ List<String> result = scanPackageDir(dir);
+ if (result != null) {
+ dirs.addAll(result);
+ }
+ }
+ } else {
+ if (!hasScane) {
+ dirs.add(packageDir);
+ hasScane = true;
+ }
+ }
+ line = br.readLine();
+ }
+ return dirs;
+ } catch (Exception e) {
+ LOG.error("包扫描异常：", e);
+ // throw new RuntimeException("scan error "+packageDir,e);
+ }
+ return null;
+
+ }
+
+ protected boolean inJar(String dir) {
+ return FileUtil.inJar(dir, this.getClass());
+ }
+
+ protected void scanClassInJar(String dirName) {
+ URL url = null;
+ try {
+ url = this.getClass().getResource(dirName);
+ if (url == null) {
+ return;
+ }
+ } catch (Exception e) {
+ LOG.error("ScanFunctionService scanClassInJar error", e);
+ }
+
+ // jar:file:/Users/yuanxiaodong/alibaba/rule-engine-feature/5/rules-engine/engine/target/ruleengine
+ // .jar!/com/aliyun/filter/function/expression
+
+ String jarUrl = url.toString().replace("jar:file:", "");
+ int index = jarUrl.indexOf("!/");
+ String packageName = createPackageName(dirName);
+ jarUrl = jarUrl.substring(0, index);
+ scanClassInJar(jarUrl, packageName, this.getClass().getClassLoader());
+
+ }
+
+ protected void scanClassInJar(String jarPath, String packageName, ClassLoader classLoader) {
+ try {
+ if (classLoader == null) {
+ classLoader = this.getClass().getClassLoader();
+ }
+ JarFile jarFile = new JarFile(jarPath);
+ Enumeration<JarEntry> entries = jarFile.entries();
+
+ while (entries.hasMoreElements()) {
+ String className = entries.nextElement().getName().replace("/", ".");
+ if (className.startsWith(packageName) && className.endsWith(".class")) {
+ className = className.replace(CLASS_REAR, "");
+ doRegisterFunction(className, classLoader);
+ }
+
+ }
+ } catch (Exception e) {
+ LOG.error("ScanFunctionService scanClassInJar JarFile error", e);
+ }
+ }
+
+ protected void scanClassInDir(String dirName) {
+
+ InputStream in = this.getClass().getResourceAsStream(dirName);
+ if (in == null) {
+ return;
+ }
+ BufferedReader br = new BufferedReader(new InputStreamReader(in));
+ String packageName = createPackageName(dirName);
+ try {
+ String line = br.readLine();
+ while (line != null) {
+ if (!line.endsWith(CLASS_REAR)) {
+ line = br.readLine();
+ continue;
+ }
+ String className = line.replace(CLASS_REAR, "");
+ doRegisterFunction(packageName + className);
+ line = br.readLine();
+ }
+ } catch (IOException e) {
+ LOG.error("注册类错误" + e.getMessage(), e);
+ }
+
+ }
+
+ private String createPackageName(String dirName) {
+ if (dirName.startsWith("/")) {
+ String packageName = dirName.substring(1).replace("/", ".") + ".";
+ return packageName;
+ } else {
+ return this.getClass().getPackage().getName() + "." + dirName + ".";
+ }
+ }
+
+ protected void doRegisterFunction(String className) {
+ doRegisterFunction(className, this.getClass().getClassLoader());
+ }
+
+ protected void doRegisterFunction(String className, ClassLoader classLoader) {
+ Class clazz = null;
+ try {
+ clazz = Class.forName(className, true, classLoader);
+ doProcessor(clazz);
+ } catch (Exception e) {
+ e.printStackTrace();
+ LOG.error("初始化类错误" + e.getMessage(), e);
+ }
+ }
+
+ protected abstract void doProcessor(Class clazz);
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/AbstractChannel.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/AbstractChannel.java
new file mode 100644
index 0000000..d002fa1
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/AbstractChannel.java
@@ -0,0 +1,220 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.channel;
+
+import org.apache.rocketmq.streams.common.channel.sink.ISink;
+import org.apache.rocketmq.streams.common.channel.source.AbstractSource;
+import org.apache.rocketmq.streams.common.channel.source.ISource;
+import org.apache.rocketmq.streams.common.channel.split.ISplit;
+import org.apache.rocketmq.streams.common.checkpoint.CheckPointMessage;
+import org.apache.rocketmq.streams.common.configurable.BasedConfigurable;
+import org.apache.rocketmq.streams.common.context.IMessage;
+import org.apache.rocketmq.streams.common.context.MessageOffset;
+import org.apache.rocketmq.streams.common.interfaces.IStreamOperator;
+import org.apache.rocketmq.streams.common.topology.builder.PipelineBuilder;
+import org.apache.rocketmq.streams.common.utils.Base64Utils;
+import org.apache.rocketmq.streams.common.utils.InstantiationUtil;
+
+import java.util.*;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import com.alibaba.fastjson.JSONObject;
+
+/**
+ * channel的抽象，实现了消息的封装，发送等核心逻辑
+ */
+public abstract class AbstractChannel extends BasedConfigurable implements IChannel<IChannel> {
+ protected transient ISink sink;
+ protected transient ISource source;
+
+ protected abstract ISink createSink();
+
+ protected abstract ISource createSource();
+
+ protected transient AtomicBoolean hasCreated = new AtomicBoolean(false);
+
+ @Override
+ protected boolean initConfigurable() {
+ create();
+ return true;
+ }
+
+ protected void create() {
+ if (hasCreated.compareAndSet(false, true)) {
+ sink = createSink();
+ source = createSource();
+ sink.init();
+ source.init();
+ }
+
+ }
+
+ @Override
+ public void getJsonObject(JSONObject jsonObject) {
+
+ super.getJsonObject(jsonObject);
+ String sinkValue = jsonObject.getString("sink");
+ String sourceValue = jsonObject.getString("source");
+ create();
+ if (sourceValue != null) {
+ source = InstantiationUtil.deserializeObject(Base64Utils.decode(sourceValue));
+ source.init();
+ }
+ if (sinkValue != null) {
+ sink = InstantiationUtil.deserializeObject(Base64Utils.decode(sinkValue));
+ sink.init();
+ }
+
+ }
+
+ @Override
+ public boolean flush(String... splitIds) {
+ return sink.flush(splitIds);
+ }
+
+ @Override
+ protected void setJsonObject(JSONObject jsonObject) {
+ super.setJsonObject(jsonObject);
+ jsonObject.put("sink", Base64Utils.encode(InstantiationUtil.serializeObject(sink)));
+ jsonObject.put("source", Base64Utils.encode(InstantiationUtil.serializeObject(source)));
+ }
+
+ @Override
+ public void removeSplit(Set<String> splitIds) {
+ source.removeSplit(splitIds);
+ }
+
+ @Override
+ public void addNewSplit(Set<String> splitIds) {
+ source.addNewSplit(splitIds);
+ }
+
+ @Override
+ public Map<String, MessageOffset> getFinishedQueueIdAndOffsets(CheckPointMessage checkPointMessage) {
+ return sink.getFinishedQueueIdAndOffsets(checkPointMessage);
+ }
+
+ @Override
+ public boolean flush(Set<String> splitIds) {
+ return sink.flush(splitIds);
+ }
+
+ @Override
+ public boolean flushMessage(List<IMessage> messages) {
+ return sink.flushMessage(messages);
+ }
+
+ @Override
+ public IChannel createStageChain(PipelineBuilder pipelineBuilder) {
+ return this;
+ }
+
+ @Override
+ public void addConfigurables(PipelineBuilder pipelineBuilder) {
+ pipelineBuilder.addConfigurables(this);
+ }
+
+ @Override
+ public ISink getSink() {
+ return sink;
+ }
+
+ @Override
+ public ISource getSource() {
+ return source;
+ }
+
+ @Override
+ public boolean batchAdd(IMessage fieldName2Value, ISplit split) {
+ return sink.batchAdd(fieldName2Value, split);
+ }
+
+ @Override
+ public boolean batchAdd(IMessage fieldName2Value) {
+ return sink.batchAdd(fieldName2Value);
+ }
+
+ @Override
+ public boolean batchSave(List<IMessage> messages) {
+ return sink.batchSave(messages);
+ }
+
+ @Override
+ public boolean flush() {
+ return sink.flush();
+ }
+
+ @Override
+ public void openAutoFlush() {
+ sink.openAutoFlush();
+ }
+
+ @Override
+ public void closeAutoFlush() {
+ sink.closeAutoFlush();
+ }
+
+ @Override
+ public void setBatchSize(int batchSize) {
+ sink.setBatchSize(batchSize);
+ }
+
+ @Override
+ public int getBatchSize() {
+ return sink.getBatchSize();
+ }
+
+ @Override
+ public boolean start(IStreamOperator receiver) {
+ return source.start(receiver);
+ }
+
+ @Override
+ public String getGroupName() {
+ return source.getGroupName();
+ }
+
+ @Override
+ public void setGroupName(String groupName) {
+ source.setGroupName(groupName);
+ }
+
+ @Override
+ public int getMaxThread() {
+ return source.getMaxThread();
+ }
+
+ @Override
+ public void setMaxThread(int maxThread) {
+ source.setMaxThread(maxThread);
+ }
+
+ @Override
+ public void setMaxFetchLogGroupSize(int size) {
+ source.setMaxFetchLogGroupSize(size);
+ }
+
+ @Override
+ public long getCheckpointTime() {
+ return source.getCheckpointTime();
+ }
+
+ public void setJsonData(Boolean isJsonData) {
+ create();
+ ((AbstractSource)source).setJsonData(isJsonData);
+ }
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/IChannel.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/IChannel.java
new file mode 100644
index 0000000..f0384a3
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/IChannel.java
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.channel;
+
+import org.apache.rocketmq.streams.common.channel.sink.ISink;
+import org.apache.rocketmq.streams.common.channel.source.ISource;
+
+import java.io.Serializable;
+
+/**
+ * source和sink的结合，dipper框架中，不单独区分source和sink，统一用channel 接口表示
+ */
+public interface IChannel<T extends IChannel>
+ extends ISink<T>, ISource<T>, Serializable {
+ String TYPE = "channel";
+
+ ISink getSink();
+
+ ISource getSource();
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/builder/AbstractSupportShuffleChannelBuilder.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/builder/AbstractSupportShuffleChannelBuilder.java
new file mode 100644
index 0000000..9153292
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/builder/AbstractSupportShuffleChannelBuilder.java
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.channel.builder;
+
+import com.alibaba.fastjson.JSONObject;
+
+import org.apache.rocketmq.streams.common.channel.source.ISource;
+import org.apache.rocketmq.streams.common.utils.ConfigurableUtil;
+
+public abstract class AbstractSupportShuffleChannelBuilder implements IChannelBuilder, IShuffleChannelBuilder {
+
+ @Override
+ public ISource copy(ISource pipelineSource) {
+ JSONObject jsonObject = JSONObject.parseObject(pipelineSource.toJson());
+ return (ISource)ConfigurableUtil.create(pipelineSource.getNameSpace(), pipelineSource.getConfigureName(), jsonObject, pipelineSource.getClass().getName());
+ }
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/builder/IChannelBuilder.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/builder/IChannelBuilder.java
new file mode 100644
index 0000000..0c26811
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/builder/IChannelBuilder.java
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.channel.builder;
+
+import com.alibaba.fastjson.JSONObject;
+import org.apache.rocketmq.streams.common.channel.sink.ISink;
+import org.apache.rocketmq.streams.common.channel.source.ISource;
+import org.apache.rocketmq.streams.common.metadata.MetaData;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+
+import java.util.Properties;
+
+/**
+ * 创建channel，如果需要扩展channel，需要实现这个接口，跟定属性文件，能够创建channel对象 如果想和sql对接，实现这个接口，properties中的kv是sql中with部分的内容
+ */
+public interface IChannelBuilder {
+
+ /**
+ * 主要完成sql中的with 属性和source/sink字段名的映射
+ *
+ * @param formatProperties
+ * @param inputProperties
+ * @param formatName
+ * @param inputName
+ */
+ static void formatPropertiesName(JSONObject formatProperties, Properties inputProperties, String formatName, String inputName) {
+ String inputValue = inputProperties.getProperty(inputName);
+ if (StringUtil.isNotEmpty(inputValue)) {
+ formatProperties.put(formatName, inputValue);
+ }
+ }
+
+ /**
+ * 创建channel
+ *
+ * @param properties
+ * @return
+ */
+ ISource createSource(String namespace, String name, Properties properties, MetaData metaData);
+
+ /**
+ * 返回channel类型，和blink语句中的type值一致
+ *
+ * @return
+ */
+ String getType();
+
+ /**
+ * 创建channel
+ *
+ * @param properties
+ * @return
+ */
+ ISink createSink(String namespace, String name, Properties properties, MetaData metaData);
+
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/builder/IShuffleChannelBuilder.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/builder/IShuffleChannelBuilder.java
new file mode 100644
index 0000000..a6e64f4
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/builder/IShuffleChannelBuilder.java
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.channel.builder;
+
+import org.apache.rocketmq.streams.common.channel.sink.ISink;
+import org.apache.rocketmq.streams.common.channel.source.ISource;
+
+public interface IShuffleChannelBuilder {
+
+ /**
+ * 根据数据源source 创建一个source shuffle出来
+ *
+ * @param pipelineSource
+ * @return
+ */
+ ISource copy(ISource pipelineSource);
+
+ /**
+ * 根据数据源source 创建一个source shuffle出来
+ *
+ * @param pipelineSource
+ * @return
+ */
+ ISink createBySource(ISource pipelineSource);
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/impl/OutputPrintChannel.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/impl/OutputPrintChannel.java
new file mode 100644
index 0000000..775581c
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/impl/OutputPrintChannel.java
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.channel.impl;
+
+import org.apache.commons.lang3.time.StopWatch;
+import org.apache.rocketmq.streams.common.channel.sink.AbstractSink;
+import org.apache.rocketmq.streams.common.context.IMessage;
+import org.apache.rocketmq.streams.common.utils.PrintUtil;
+
+import java.util.List;
+
+/**
+ * 测试使用，输出就是把消息打印出来
+ */
+public class OutputPrintChannel extends AbstractSink {
+
+ private static int counter = 1;
+ private transient boolean start = false;
+ private static long startTime = System.currentTimeMillis();
+ private static long begin = startTime;
+ private static int step = 40000;
+
+ @Override
+ protected boolean batchInsert(List<IMessage> messages) {
+ StringBuilder stringBuilder = new StringBuilder();
+ for (IMessage msg : messages) {
+ stringBuilder.append(msg.getMessageValue().toString() + PrintUtil.LINE);
+ }
+ System.out.println(stringBuilder.toString());
+ return false;
+ }
+
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/impl/PrintChannelBuilder.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/impl/PrintChannelBuilder.java
new file mode 100644
index 0000000..d094315
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/impl/PrintChannelBuilder.java
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.channel.impl;
+
+import java.util.Properties;
+
+import com.google.auto.service.AutoService;
+import org.apache.rocketmq.streams.common.channel.builder.IChannelBuilder;
+import org.apache.rocketmq.streams.common.channel.sink.ISink;
+import org.apache.rocketmq.streams.common.channel.source.ISource;
+import org.apache.rocketmq.streams.common.metadata.MetaData;
+import org.apache.rocketmq.streams.common.model.ServiceName;
+
+@AutoService(IChannelBuilder.class)
+@ServiceName(value = PrintChannelBuilder.TYPE, aliasName = "OutputPrintChannel")
+public class PrintChannelBuilder implements IChannelBuilder {
+ public static final String TYPE = "print";
+
+ @Override
+ public ISource createSource(String namespace, String name, Properties properties, MetaData metaData) {
+ throw new RuntimeException("can not support this method");
+ }
+
+ @Override
+ public String getType() {
+ return TYPE;
+ }
+
+ @Override
+ public ISink createSink(String namespace, String name, Properties properties, MetaData metaData) {
+ return new OutputPrintChannel();
+ }
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/impl/file/FileChannelBuilder.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/impl/file/FileChannelBuilder.java
new file mode 100644
index 0000000..559a9ec
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/impl/file/FileChannelBuilder.java
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.channel.impl.file;
+
+import java.util.Properties;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import com.alibaba.fastjson.JSONObject;
+
+import com.google.auto.service.AutoService;
+import org.apache.rocketmq.streams.common.channel.builder.AbstractSupportShuffleChannelBuilder;
+import org.apache.rocketmq.streams.common.channel.builder.IChannelBuilder;
+import org.apache.rocketmq.streams.common.channel.impl.memory.MemoryChannel;
+import org.apache.rocketmq.streams.common.channel.impl.memory.MemorySink;
+import org.apache.rocketmq.streams.common.channel.impl.memory.MemorySource;
+import org.apache.rocketmq.streams.common.channel.sink.ISink;
+import org.apache.rocketmq.streams.common.channel.source.ISource;
+import org.apache.rocketmq.streams.common.metadata.MetaData;
+import org.apache.rocketmq.streams.common.model.ServiceName;
+import org.apache.rocketmq.streams.common.utils.ConfigurableUtil;
+
+@AutoService(IChannelBuilder.class)
+@ServiceName(value = FileChannelBuilder.TYPE, aliasName = "FileSource")
+public class FileChannelBuilder extends AbstractSupportShuffleChannelBuilder {
+
+ public static final String TYPE = "file";
+
+ /**
+ * @param namespace
+ * @param name
+ * @param properties
+ * @return
+ */
+ @Override
+ public ISource createSource(String namespace, String name, Properties properties, MetaData metaData) {
+
+ FileSource fileSource = (FileSource)ConfigurableUtil.create(FileSource.class.getName(), namespace, name, createFormatProperty(properties), null);
+ return fileSource;
+ }
+
+ @Override
+ public String getType() {
+ return TYPE;
+ }
+
+ @Override
+ public ISink createSink(String namespace, String name, Properties properties, MetaData metaData) {
+ FileSink fileSink = (FileSink)ConfigurableUtil.create(FileSink.class.getName(), namespace, name, createFormatProperty(properties), null);
+ return fileSink;
+ }
+
+ /**
+ * 创建标准的属性文件
+ *
+ * @param properties
+ * @return
+ */
+ protected JSONObject createFormatProperty(Properties properties) {
+ JSONObject formatProperties = new JSONObject();
+ for (Object object : properties.keySet()) {
+ String key = (String)object;
+ if ("type".equals(key)) {
+ continue;
+ }
+ formatProperties.put(key, properties.getProperty(key));
+ }
+ IChannelBuilder.formatPropertiesName(formatProperties, properties, "filePath", "filePath");
+ IChannelBuilder.formatPropertiesName(formatProperties, properties, "filePath", "fileName");
+ IChannelBuilder.formatPropertiesName(formatProperties, properties, "maxThread", "thread.max.count");
+ return formatProperties;
+ }
+
+ @Override
+ public ISource copy(ISource pipelineSource) {
+ return new MemoryChannel();
+ }
+
+ @Override
+ public ISink createBySource(ISource pipelineSource) {
+ return new MemoryChannel();
+ }
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/impl/file/FileSink.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/impl/file/FileSink.java
new file mode 100644
index 0000000..1fbc3cc
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/impl/file/FileSink.java
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.channel.impl.file;
+
+import org.apache.rocketmq.streams.common.channel.sink.AbstractSupportShuffleSink;
+import org.apache.rocketmq.streams.common.channel.sink.ISink;
+import org.apache.rocketmq.streams.common.channel.split.ISplit;
+import org.apache.rocketmq.streams.common.configurable.annotation.ENVDependence;
+import org.apache.rocketmq.streams.common.context.IMessage;
+import org.apache.rocketmq.streams.common.utils.FileUtil;
+import org.apache.rocketmq.streams.common.utils.PrintUtil;
+
+import java.io.BufferedWriter;
+import java.io.File;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * 文件的输入输出，source是把指定的文件数据加载到内存，for循环输出到后续节点 sink，把内容写入文件，可以配置写入模式，是追加还是覆盖
+ */
+public class FileSink extends AbstractSupportShuffleSink {
+
+ /**
+ * 文件全路径。如果目录是环境变量，可以写成evnDir/文件名
+ */
+ @ENVDependence
+ private String filePath;
+
+ /**
+ * 文件写入时，是否追加
+ */
+ private boolean needAppend = true;
+
+ protected transient BufferedWriter writer;
+ /**
+ * FileChannel 中 writer 完成 初始化 标识
+ */
+ private volatile transient boolean writerInitFlag = false;
+
+ public FileSink(String filePath) {
+ this(filePath, false);
+
+ }
+
+ public FileSink(String filePath, boolean needAppend) {
+ this();
+ this.filePath = filePath;
+ this.needAppend = needAppend;
+
+ }
+
+ @Override
+ public String getShuffleTopicFieldName() {
+ return "filePath";
+ }
+
+ @Override
+ protected void createTopicIfNotExist(int splitNum) {
+
+ }
+
+ @Override
+ public List<ISplit> getSplitList() {
+ File file = new File(filePath);
+ List<ISplit> splits = new ArrayList<>();
+ splits.add(new FileSplit(file));
+ return splits;
+ }
+
+ @Override
+ public int getSplitNum() {
+ return 1;
+ }
+
+ @Override
+ protected boolean batchInsert(List<IMessage> messages) {
+ // 初始化 write 防止 文件不存在导致异常
+ initWrite();
+ if (messages != null) {
+ try {
+ for (IMessage message : messages) {
+ writer.write(message.getMessageValue().toString() + PrintUtil.LINE);
+ }
+ writer.flush();
+ } catch (IOException e) {
+ throw new RuntimeException("write line error " + filePath, e);
+ }
+ }
+ return true;
+ }
+
+ @Override
+ public void destroy() {
+
+ try {
+ if (writer != null) {
+ writer.flush();
+ writer.close();
+ }
+
+ } catch (IOException e) {
+ throw new RuntimeException("close error " + filePath, e);
+ }
+ }
+
+ public FileSink() {
+ setType(ISink.TYPE);
+ }
+
+ private static final String PREFIX = "dipper.upgrade.channel.file.envkey";
+
+ /**
+ * 初始化 witer 防止文件不存在异常
+ */
+ private void initWrite() {
+ if (!writerInitFlag) {
+ synchronized (this) {
+ if (!writerInitFlag) {
+ try {
+ writer = new BufferedWriter(new FileWriter(filePath, needAppend));
+ writerInitFlag = true;
+ } catch (Exception e) {
+ throw new RuntimeException("create write error " + filePath, e);
+ }
+ }
+ }
+ }
+ }
+
+ public String getFilePath() {
+ return filePath;
+ }
+
+ public void setFilePath(String filePath) {
+ this.filePath = filePath;
+ }
+
+ public boolean isNeedAppend() {
+ return needAppend;
+ }
+
+ public void setNeedAppend(boolean needAppend) {
+ this.needAppend = needAppend;
+ }
+
+}
+
+
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/impl/file/FileSource.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/impl/file/FileSource.java
new file mode 100644
index 0000000..dbf29e9
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/impl/file/FileSource.java
@@ -0,0 +1,232 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.channel.impl.file;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import com.alibaba.fastjson.JSONObject;
+
+import org.apache.rocketmq.streams.common.channel.source.AbstractBatchSource;
+import org.apache.rocketmq.streams.common.channel.source.ISource;
+import org.apache.rocketmq.streams.common.configurable.annotation.ENVDependence;
+import org.apache.rocketmq.streams.common.utils.DateUtil;
+
+/**
+ * 文件的输入输出，source是把指定的文件数据加载到内存，for循环输出到后续节点 sink，把内容写入文件，可以配置写入模式，是追加还是覆盖
+ */
+public class FileSource extends AbstractBatchSource {
+
+ /**
+ * 文件全路径。如果目录是环境变量，可以写成evnDir/文件名
+ */
+ @ENVDependence
+ private String filePath;
+
+ protected transient BufferedReader reader;
+ protected transient ExecutorService executorService;
+
+ public FileSource(String filePath) {
+ this();
+ this.filePath = filePath;
+ }
+
+ @Override
+ protected boolean initConfigurable() {
+ super.initConfigurable();
+ File file = new File(filePath);
+ if (file.exists() && file.isDirectory()) {
+ executorService = new ThreadPoolExecutor(maxThread, maxThread,
+ 0L, TimeUnit.MILLISECONDS,
+ new LinkedBlockingQueue<Runnable>(1000));
+ }
+ return true;
+ }
+
+ @Override
+ protected boolean startSource() {
+
+ LinkedBlockingQueue<FileIterator> queue = createIteratorList();
+ AtomicInteger count = new AtomicInteger(0);
+ long startTime = System.currentTimeMillis();
+ CountDownLatch countDownLatch = new CountDownLatch(queue.size());
+ try {
+ FileIterator fileIterator = queue.poll();
+ while (fileIterator != null) {
+ ReadTask readTask = new ReadTask(fileIterator, count, countDownLatch);
+ if (executorService != null) {
+ executorService.execute(readTask);
+ } else {
+ readTask.run();
+ }
+ fileIterator = queue.poll();
+ }
+ countDownLatch.await();
+ } catch (Exception e) {
+ e.printStackTrace();
+ }
+
+ System.out.println("process data cost :" + (System.currentTimeMillis() - startTime) + ", the count is " + count.get() + " now " + DateUtil.getCurrentTimeString());
+
+ return true;
+ }
+
+ /**
+ * 如果是目录，每个文件一个iterator，如果是文件只生成一个iterator
+ */
+ protected LinkedBlockingQueue<FileIterator> createIteratorList() {
+ LinkedBlockingQueue<FileIterator> iterators = new LinkedBlockingQueue<>(1000);
+ File file = new File(filePath);
+ if (file.exists() == false) {
+ return null;
+ }
+ try {
+ if (file.isDirectory() == false) {
+ iterators.put(new FileIterator(file));
+ return iterators;
+ }
+ File[] files = file.listFiles();
+ for (File subFile : files) {
+ iterators.add(new FileIterator(subFile));
+ }
+ return iterators;
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+
+ }
+
+ /**
+ * 一个文件生成一个Iterator，每次加载一行数据
+ */
+ public static class FileIterator implements Iterator<String> {
+ protected File file;
+ private String line;
+ protected int index=0;
+ protected BufferedReader reader = null;
+
+ public FileIterator(File file) throws FileNotFoundException {
+ this.file = file;
+ reader = new BufferedReader(new FileReader(file));
+ }
+
+ @Override
+ public boolean hasNext() {
+ try {
+ line = reader.readLine();
+ index++;
+ if (line != null) {
+ return true;
+ }
+ } catch (IOException e) {
+ throw new RuntimeException("read error ", e);
+ }
+
+ return false;
+ }
+
+ public int getIndex() {
+ return index;
+ }
+
+ public void close() {
+ try {
+ reader.close();
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ @Override
+ public String next() {
+ return line;
+ }
+ }
+
+ /**
+ * 如果是个目录，1个文件一个线程
+ */
+ protected class ReadTask implements Runnable {
+ protected FileIterator fileIterator;
+ protected AtomicInteger count;
+ private CountDownLatch countDownLatch;
+
+ public ReadTask(FileIterator fileIterator, AtomicInteger count, CountDownLatch countDownLatch) {
+ this.fileIterator = fileIterator;
+ this.count = count;
+ this.countDownLatch = countDownLatch;
+ }
+
+ @Override
+ public void run() {
+ if (fileIterator != null) {
+ int offset = 1;
+ while (fileIterator.hasNext()) {
+ String line = fileIterator.next();
+ doReceiveMessage(line, false, fileIterator.file.getName(), offset + "");
+ offset++;
+ count.incrementAndGet();
+
+ }
+ sendCheckpoint(fileIterator.file.getName());
+ fileIterator.close();
+ countDownLatch.countDown();
+ ;
+ }
+
+ }
+ }
+
+ @Override
+ public void destroy() {
+
+ try {
+ if (reader != null) {
+ reader.close();
+ }
+
+ } catch (IOException e) {
+ String realFilePath = filePath;
+ throw new RuntimeException("close error " + realFilePath, e);
+ }
+ }
+
+ public FileSource() {
+ setType(ISource.TYPE);
+ }
+
+ public String getFilePath() {
+ return filePath;
+ }
+
+ public void setFilePath(String filePath) {
+ this.filePath = filePath;
+ }
+
+}
+
+
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/impl/file/FileSplit.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/impl/file/FileSplit.java
new file mode 100644
index 0000000..134c1e6
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/impl/file/FileSplit.java
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.channel.impl.file;
+
+import java.io.File;
+
+import com.alibaba.fastjson.JSONObject;
+
+import org.apache.rocketmq.streams.common.channel.split.ISplit;
+import org.apache.rocketmq.streams.common.configurable.BasedConfigurable;
+
+public class FileSplit extends BasedConfigurable implements ISplit<FileSplit, File> {
+ protected transient File file;
+ private String filePath;
+ protected int splitId;
+
+ @Override
+ public String getQueueId() {
+ return filePath + "_" + splitId;
+ }
+
+ @Override
+ public String getPlusQueueId() {
+ return filePath + "_" + (splitId + 1);
+ }
+
+ @Override
+ public File getQueue() {
+ return file;
+ }
+
+ @Override
+ public int compareTo(FileSplit o) {
+ return splitId - o.splitId;
+ }
+
+ @Override
+ protected void getJsonObject(JSONObject jsonObject) {
+ super.getJsonObject(jsonObject);
+ File file = new File(filePath);
+ splitId = Integer.valueOf(file.getName().replace("split_", ""));
+ this.file = file;
+ }
+
+ public FileSplit(File file) {
+ this.filePath = file.getAbsolutePath();
+ splitId = Integer.valueOf(file.getName().replace("split_", ""));
+ this.file = file;
+ }
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/impl/memory/MemoryCache.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/impl/memory/MemoryCache.java
new file mode 100644
index 0000000..050e0e6
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/impl/memory/MemoryCache.java
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.channel.impl.memory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import com.alibaba.fastjson.JSONObject;
+
+import org.apache.rocketmq.streams.common.configurable.BasedConfigurable;
+import org.apache.rocketmq.streams.common.context.IMessage;
+import org.apache.rocketmq.streams.common.utils.Base64Utils;
+import org.apache.rocketmq.streams.common.utils.InstantiationUtil;
+
+public class MemoryCache extends BasedConfigurable {
+ public static String TYPE = "memoryCache";
+ protected List<String> cache = new ArrayList<>();
+ protected transient ConcurrentLinkedQueue<String> queue;
+
+ public <T> MemoryCache(T[] array) {
+ this();
+ for (T t : array) {
+ cache.add(Base64Utils.encode(InstantiationUtil.serializeObject(t)));
+ }
+ }
+
+ public MemoryCache(List<?> list) {
+ this();
+ for (Object t : list) {
+ cache.add(Base64Utils.encode(InstantiationUtil.serializeObject(t)));
+ }
+ }
+
+ public MemoryCache() {
+ setType(TYPE);
+ }
+
+ @Override
+ protected boolean initConfigurable() {
+ queue = new ConcurrentLinkedQueue();
+ for (String t : cache) {
+ byte[] bytes = Base64Utils.decode(t);
+ queue.offer(InstantiationUtil.deserializeObject(bytes));
+ }
+ return super.initConfigurable();
+ }
+
+ public List<String> getCache() {
+ return cache;
+ }
+
+ public void setCache(List<String> cache) {
+ this.cache = cache;
+ }
+
+ public ConcurrentLinkedQueue getQueue() {
+ return queue;
+ }
+
+ public void addMessage(String msg) {
+ queue.offer(msg);
+ }
+
+ public void addMessage(JSONObject msg) {
+ addMessage(msg.toJSONString());
+ }
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/impl/memory/MemoryChannel.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/impl/memory/MemoryChannel.java
new file mode 100644
index 0000000..ad6752a
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/impl/memory/MemoryChannel.java
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.channel.impl.memory;
+
+import org.apache.rocketmq.streams.common.channel.AbstractChannel;
+import org.apache.rocketmq.streams.common.channel.sink.AbstractSink;
+import org.apache.rocketmq.streams.common.channel.sink.ISink;
+import org.apache.rocketmq.streams.common.channel.source.AbstractUnreliableSource;
+import org.apache.rocketmq.streams.common.channel.source.ISource;
+import org.apache.rocketmq.streams.common.context.IMessage;
+import org.apache.rocketmq.streams.common.context.Message;
+
+import java.util.List;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * 消息产生的source数据，就是通过sink写入的消息
+ */
+public class MemoryChannel extends AbstractChannel {
+ /**
+ * 是否启动qps的统计
+ */
+ protected transient volatile boolean startQPSCount = false;
+ /**
+ * 总处理数据数
+ */
+ protected transient AtomicLong COUNT = new AtomicLong(0);
+ /**
+ * 最早的处理时间
+ */
+ protected transient long firstReceiveTime = System.currentTimeMillis();
+
+ public void setStartQPSCount(boolean startQPSCount) {
+ this.startQPSCount = startQPSCount;
+ }
+
+ @Override
+ protected ISink createSink() {
+ return new AbstractSink() {
+ @Override
+ protected boolean batchInsert(List<IMessage> messages) {
+ if (startQPSCount) {
+ long count = COUNT.addAndGet(messages.size());
+ long second = ((System.currentTimeMillis() - firstReceiveTime) / 1000);
+ double qps = count / second;
+ System.out.println("qps is " + qps + "。the count is " + count + ".the process time is " + second);
+ }
+ for (IMessage msg : messages) {
+ ((AbstractUnreliableSource)source).doUnreliableReceiveMessage(msg.getMessageValue());
+ }
+ return true;
+ }
+ };
+ }
+
+ @Override
+ protected ISource createSource() {
+ return new AbstractUnreliableSource() {
+ @Override
+ protected boolean startSource() {
+ return super.startSource();
+ }
+ };
+ }
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/impl/memory/MemoryChannelBuilder.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/impl/memory/MemoryChannelBuilder.java
new file mode 100644
index 0000000..224b62b
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/impl/memory/MemoryChannelBuilder.java
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.channel.impl.memory;
+
+import java.util.Properties;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import com.alibaba.fastjson.JSONObject;
+
+import com.google.auto.service.AutoService;
+import org.apache.rocketmq.streams.common.channel.builder.AbstractSupportShuffleChannelBuilder;
+import org.apache.rocketmq.streams.common.channel.builder.IChannelBuilder;
+import org.apache.rocketmq.streams.common.channel.impl.file.FileSink;
+import org.apache.rocketmq.streams.common.channel.impl.file.FileSource;
+import org.apache.rocketmq.streams.common.channel.sink.ISink;
+import org.apache.rocketmq.streams.common.channel.source.ISource;
+import org.apache.rocketmq.streams.common.metadata.MetaData;
+import org.apache.rocketmq.streams.common.model.ServiceName;
+import org.apache.rocketmq.streams.common.utils.ConfigurableUtil;
+
+@AutoService(IChannelBuilder.class)
+@ServiceName(value = MemoryChannelBuilder.TYPE, aliasName = "MemorySource")
+public class MemoryChannelBuilder extends AbstractSupportShuffleChannelBuilder {
+
+ public static final String TYPE = "memory";
+
+ /**
+ * @param namespace
+ * @param name
+ * @param properties
+ * @return
+ */
+ @Override
+ public ISource createSource(String namespace, String name, Properties properties, MetaData metaData) {
+ return new MemorySource();
+ }
+
+ @Override
+ public String getType() {
+ return TYPE;
+ }
+
+ @Override
+ public ISink createSink(String namespace, String name, Properties properties, MetaData metaData) {
+ return new MemorySink();
+ }
+
+ @Override
+ public ISource copy(ISource pipelineSource) {
+ return new MemoryChannel();
+ }
+
+ @Override
+ public ISink createBySource(ISource pipelineSource) {
+ return new MemoryChannel();
+ }
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/impl/memory/MemorySink.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/impl/memory/MemorySink.java
new file mode 100644
index 0000000..c9cdf11
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/impl/memory/MemorySink.java
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.channel.impl.memory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.apache.rocketmq.streams.common.channel.sink.AbstractSupportShuffleSink;
+import org.apache.rocketmq.streams.common.channel.split.ISplit;
+import org.apache.rocketmq.streams.common.configurable.IAfterConfiguableRefreshListerner;
+import org.apache.rocketmq.streams.common.configurable.IConfigurableService;
+import org.apache.rocketmq.streams.common.context.IMessage;
+
+public class MemorySink extends AbstractSupportShuffleSink implements IAfterConfiguableRefreshListerner {
+ /**
+ * 是否启动qps的统计
+ */
+ protected transient volatile boolean startQPSCount = false;
+ /**
+ * 总处理数据数
+ */
+ protected transient AtomicLong COUNT = new AtomicLong(0);
+ /**
+ * 最早的处理时间
+ */
+ protected transient long firstReceiveTime = System.currentTimeMillis();
+ protected String cacheName;
+ protected transient MemoryCache memoryCache;
+
+ public MemorySink() {}
+
+ @Override
+ protected boolean batchInsert(List<IMessage> messages) {
+ if (startQPSCount) {
+ long count = COUNT.addAndGet(messages.size());
+ long second = ((System.currentTimeMillis() - firstReceiveTime) / 1000);
+ double qps = count / second;
+ System.out.println("qps is " + qps + "。the count is " + count + ".the process time is " + second);
+ }
+ try {
+ for (IMessage msg : messages) {
+ memoryCache.queue.offer(msg.getMessageValue().toString());
+ }
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+
+ return true;
+ }
+
+ @Override
+ public String getShuffleTopicFieldName() {
+ return null;
+ }
+
+ @Override
+ protected void createTopicIfNotExist(int splitNum) {
+
+ }
+
+ @Override
+ public List<ISplit> getSplitList() {
+ List<ISplit> splits = new ArrayList<>();
+ splits.add(new MemorySplit());
+ return splits;
+ }
+
+ @Override
+ public int getSplitNum() {
+ return 1;
+ }
+
+ @Override
+ public void doProcessAfterRefreshConfigurable(IConfigurableService configurableService) {
+ memoryCache = configurableService.queryConfigurable(MemoryCache.TYPE, cacheName);
+ }
+
+ public String getCacheName() {
+ return cacheName;
+ }
+
+ public void setCacheName(String cacheName) {
+ this.cacheName = cacheName;
+ }
+
+ public void setMemoryCache(MemoryCache memoryCache) {
+ this.memoryCache = memoryCache;
+ setCacheName(memoryCache.getConfigureName());
+
+ }
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/impl/memory/MemorySource.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/impl/memory/MemorySource.java
new file mode 100644
index 0000000..806c36d
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/impl/memory/MemorySource.java
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.channel.impl.memory;
+
+import org.apache.rocketmq.streams.common.channel.source.AbstractUnreliableSource;
+import org.apache.rocketmq.streams.common.configurable.IAfterConfiguableRefreshListerner;
+import org.apache.rocketmq.streams.common.configurable.IConfigurableService;
+
+public class MemorySource extends AbstractUnreliableSource implements IAfterConfiguableRefreshListerner {
+
+ protected String cacheName;
+ protected transient MemoryCache memoryCache;
+
+ public MemorySource() {
+
+ }
+
+ @Override
+ protected boolean initConfigurable() {
+ this.enableAsyncReceive = false;
+ return super.initConfigurable();
+ }
+
+ @Override
+ protected boolean startSource() {
+ boolean success = super.startSource();
+ Thread thread = new Thread(new Runnable() {
+ @Override
+ public void run() {
+ try {
+ while (true) {
+ Object message = memoryCache.queue.poll();
+ while (message != null) {
+ doUnreliableReceiveMessage(message);
+ message = memoryCache.queue.poll();
+ }
+ sendCheckpoint(getQueueId());
+ Thread.sleep(1000);
+ }
+
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+
+ }
+ });
+ thread.start();
+ return true;
+ }
+
+ @Override
+ public String getQueueId() {
+ return "1";
+ }
+
+ public String getCacheName() {
+ return cacheName;
+ }
+
+ public void setCacheName(String cacheName) {
+ this.cacheName = cacheName;
+ }
+
+ @Override
+ public void doProcessAfterRefreshConfigurable(IConfigurableService configurableService) {
+ memoryCache = configurableService.queryConfigurable(MemoryCache.TYPE, cacheName);
+ }
+
+ public void setMemoryCache(MemoryCache memoryCache) {
+ this.memoryCache = memoryCache;
+ setCacheName(memoryCache.getConfigureName());
+
+ }
+
+ public MemoryCache getMemoryCache() {
+ return memoryCache;
+ }
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/impl/memory/MemorySplit.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/impl/memory/MemorySplit.java
new file mode 100644
index 0000000..0525df8
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/impl/memory/MemorySplit.java
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.channel.impl.memory;
+
+import com.alibaba.fastjson.JSONObject;
+import org.apache.rocketmq.streams.common.channel.split.ISplit;
+import org.apache.rocketmq.streams.common.configurable.BasedConfigurable;
+
+public class MemorySplit extends BasedConfigurable implements ISplit<MemorySplit, MemorySplit> {
+ @Override
+ public String getQueueId() {
+ return 1 + "";
+ }
+
+ @Override
+ public String getPlusQueueId() {
+ return 2 + "";
+ }
+
+ @Override
+ public MemorySplit getQueue() {
+ return this;
+ }
+
+ @Override
+ public int compareTo(MemorySplit o) {
+ return 0;
+ }
+
+ @Override
+ protected void getJsonObject(JSONObject jsonObject) {
+ super.getJsonObject(jsonObject);
+
+ }
+
+ public MemorySplit() {
+ }
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/sink/AbstractSink.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/sink/AbstractSink.java
new file mode 100644
index 0000000..2f24491
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/sink/AbstractSink.java
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.channel.sink;
+
+import com.alibaba.fastjson.JSONObject;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.rocketmq.streams.common.channel.sinkcache.IMessageCache;
+import org.apache.rocketmq.streams.common.channel.sinkcache.impl.MultiSplitMessageCache;
+import org.apache.rocketmq.streams.common.channel.source.ISource;
+import org.apache.rocketmq.streams.common.channel.split.ISplit;
+import org.apache.rocketmq.streams.common.checkpoint.CheckPointManager;
+import org.apache.rocketmq.streams.common.checkpoint.CheckPointManager.SourceState;
+import org.apache.rocketmq.streams.common.checkpoint.CheckPointMessage;
+import org.apache.rocketmq.streams.common.configurable.BasedConfigurable;
+import org.apache.rocketmq.streams.common.configurable.IConfigurableIdentification;
+import org.apache.rocketmq.streams.common.context.IMessage;
+import org.apache.rocketmq.streams.common.context.MessageOffset;
+import org.apache.rocketmq.streams.common.topology.builder.PipelineBuilder;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * 输出的接口抽象，针对json消息的场景
+ */
+public abstract class AbstractSink extends BasedConfigurable implements ISink<AbstractSink> {
+ private static final Log LOG = LogFactory.getLog(AbstractSink.class);
+ public static String TARGET_QUEUE = "target_queue";//指定发送queue
+
+ public static final int DEFAULT_BATCH_SIZE = 3000;
+
+ protected transient IMessageCache<IMessage> messageCache;
+
+ protected volatile int batchSize = DEFAULT_BATCH_SIZE;
+ protected transient volatile Map<String, SourceState> sourceName2State = new HashMap<>();//保存完成刷新的queueid和offset
+
+ public AbstractSink() {
+ setType(TYPE);
+
+ }
+
+ @Override
+ protected boolean initConfigurable() {
+ messageCache = new MultiSplitMessageCache(this);
+ messageCache.openAutoFlush();
+ return super.initConfigurable();
+ }
+
+ @Override
+ public boolean batchAdd(IMessage fieldName2Value, ISplit split) {
+ fieldName2Value.getMessageBody().put(TARGET_QUEUE, split);
+ return batchAdd(fieldName2Value);
+ }
+
+ public ISplit getSPlit(IMessage message) {
+ return (ISplit)message.getMessageBody().get(TARGET_QUEUE);
+ }
+
+ @Override
+ public boolean batchAdd(IMessage fieldName2Value) {
+ messageCache.addCache(fieldName2Value);
+
+ return true;
+ }
+
+ @Override
+ public void openAutoFlush() {
+ messageCache.openAutoFlush();
+ }
+
+ @Override
+ public boolean batchSave(List<IMessage> messages) {
+ if (messages == null || messages.size() == 0) {
+ //LOG.warn("has empty data to insert");
+ return true;
+ }
+ int batchSize = this.batchSize;
+ if (batchSize == -1) {
+ batchSize = DEFAULT_BATCH_SIZE;
+ }
+ int length = messages.size();
+ if (length <= batchSize) {
+ batchInsert(messages);
+ return true;
+ }
+ int count = length / batchSize;
+ if (length % batchSize > 0) {
+ count++;
+ }
+ int startIndex = 0;
+ int endIndex = batchSize;
+ if (endIndex > length) {
+ endIndex = length;
+ }
+ for (int i = 0; i < count; i++) {
+ List<IMessage> batchItem = messages.subList(startIndex, endIndex);
+ batchInsert(batchItem);
+ startIndex = endIndex;
+ endIndex += batchSize;
+ if (endIndex > length) {
+ endIndex = length;
+ }
+ }
+ return true;
+ }
+
+ @Override
+ public boolean flush(Set<String> splitIds) {
+ int size = messageCache.flush(splitIds);
+ return size > 0;
+ }
+
+ @Override
+ public boolean flush(String... splitIds) {
+ if(splitIds==null){
+ return true;
+ }
+ Set<String> splitIdSet =new HashSet<>();
+ for(String splitId:splitIds){
+ splitIdSet.add(splitId);
+ }
+ return flush(splitIdSet);
+ }
+
+ protected abstract boolean batchInsert(List<IMessage> messages);
+
+ @Override
+ public void closeAutoFlush() {
+ messageCache.closeAutoFlush();
+ }
+
+ @Override
+ public boolean flushMessage(List<IMessage> messages) {
+
+ boolean success = batchSave(messages);
+ for (IMessage message : messages) {
+ String queueId = message.getHeader().getQueueId();
+ MessageOffset messageOffset = message.getHeader().getMessageOffset();
+ ISource source = message.getHeader().getSource();
+ String pipelineName = message.getHeader().getPiplineName();
+ String sourceName = CheckPointManager.createSourceName(source, pipelineName);
+ SourceState sourceState = this.sourceName2State.get(sourceName);
+ if (sourceState == null) {
+ sourceState = new SourceState();
+ sourceState.setSourceName(sourceName);
+ }
+ sourceState.getQueueId2Offsets().put(queueId, messageOffset);
+ }
+ return success;
+ }
+
+ @Override
+ public boolean flush() {
+ String name = getConfigureName();
+ if (StringUtil.isEmpty(name)) {
+ name = getClass().getName();
+ }
+ int size = messageCache.flush();
+ if (size > 0) {
+ System.out.println(name + " finish flush data " + size);
+ }
+
+ return true;
+ }
+
+ /**
+ * 把message对象转化成jsonobject
+ *
+ * @param messageList
+ * @return
+ */
+ protected List<JSONObject> convertJsonObjectFromMessage(List<IMessage> messageList) {
+ List<JSONObject> messages = new ArrayList<>();
+ for (IMessage message : messageList) {
+ messages.add(message.getMessageBody());
+ }
+ return messages;
+ }
+
+ @Override
+ public AbstractSink createStageChain(PipelineBuilder pipelineBuilder) {
+ return this;
+ }
+
+ @Override
+ public void addConfigurables(PipelineBuilder pipelineBuilder) {
+ pipelineBuilder.addConfigurables(this);
+ }
+
+ @Override
+ public int getBatchSize() {
+ return batchSize;
+ }
+
+ @Override
+ public void setBatchSize(int batchSize) {
+ this.batchSize = batchSize;
+ }
+
+ public IMessageCache<IMessage> getMessageCache() {
+ return messageCache;
+ }
+
+ @Override
+ public Map<String, MessageOffset> getFinishedQueueIdAndOffsets(CheckPointMessage checkPointMessage) {
+ String piplineName = null;
+ if (IConfigurableIdentification.class.isInstance(checkPointMessage.getStreamOperator())) {
+ IConfigurableIdentification configurable = (IConfigurableIdentification)checkPointMessage.getStreamOperator();
+ piplineName = configurable.getConfigureName();
+ }
+ SourceState sourceState = this.sourceName2State.get(
+ CheckPointManager.createSourceName(checkPointMessage.getSource(), piplineName));
+ if (sourceState != null) {
+ return sourceState.getQueueId2Offsets();
+ }
+ return new HashMap<>();
+ }
+
+ public void setMessageCache(
+ IMessageCache<IMessage> messageCache) {
+ this.messageCache = messageCache;
+ }
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/sink/AbstractSupportShuffleSink.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/sink/AbstractSupportShuffleSink.java
new file mode 100644
index 0000000..efcce01
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/sink/AbstractSupportShuffleSink.java
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.channel.sink;
+
+import java.util.List;
+
+import org.apache.rocketmq.streams.common.channel.IChannel;
+import org.apache.rocketmq.streams.common.channel.split.ISplit;
+import org.apache.rocketmq.streams.common.context.IMessage;
+
+public abstract class AbstractSupportShuffleSink extends AbstractSink {
+
+ protected transient int splitNum = 10;//分片个数
+
+ //sls对应的project和logstore初始化是否完成标志
+ protected volatile transient boolean hasCreated = false;
+
+ /**
+ * 获取sink的主题，在sls中是logStore，metaq是topic
+ *
+ * @return
+ */
+ public abstract String getShuffleTopicFieldName();
+
+ /**
+ * 创建一个消息队列主题，需要判断是否已经存在，如果已经存在，不需要重复创建
+ */
+ protected abstract void createTopicIfNotExist(int splitNum);
+
+ /**
+ * 获取所有的分片
+ *
+ * @return
+ */
+ public abstract List<ISplit> getSplitList();
+
+ @Override
+ protected boolean initConfigurable() {
+ boolean success = super.initConfigurable();
+ hasCreated = false;
+ checkAndCreateTopic();
+ return success;
+ }
+
+ /**
+ * 创建主题，只创建一次
+ */
+ protected void checkAndCreateTopic() {
+ if (!hasCreated) {
+ synchronized (this) {
+ if (!hasCreated) {
+ createTopicIfNotExist(splitNum);
+ hasCreated = true;
+ }
+
+ }
+ }
+
+ }
+
+ public void setSplitNum(int splitNum) {
+ this.splitNum = splitNum;
+ }
+
+ public abstract int getSplitNum();
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/sink/ISink.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/sink/ISink.java
new file mode 100644
index 0000000..1261320
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/sink/ISink.java
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.channel.sink;
+
+import org.apache.rocketmq.streams.common.channel.sinkcache.IMessageFlushCallBack;
+import org.apache.rocketmq.streams.common.channel.split.ISplit;
+import org.apache.rocketmq.streams.common.checkpoint.CheckPointMessage;
+import org.apache.rocketmq.streams.common.configurable.IConfigurable;
+import org.apache.rocketmq.streams.common.context.IMessage;
+import org.apache.rocketmq.streams.common.context.MessageOffset;
+import org.apache.rocketmq.streams.common.topology.builder.IStageBuilder;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+
+public interface ISink<T extends ISink> extends IConfigurable, IStageBuilder<T>, IMessageFlushCallBack<IMessage> {
+
+ String TYPE = "sink";
+
+ /**
+ * 根据channel推断 meta，或者不需要meta，如消息对垒
+ *
+ * @param fieldName2Value
+ * @return
+ */
+ boolean batchAdd(IMessage fieldName2Value, ISplit split);
+
+ /**
+ * 根据channel推断 meta，或者不需要meta，如消息对垒
+ *
+ * @param fieldName2Value
+ * @return
+ */
+ boolean batchAdd(IMessage fieldName2Value);
+
+ /**
+ * 直接存储存储，不过缓存
+ *
+ * @param messages
+ * @return
+ */
+ boolean batchSave(List<IMessage> messages);
+
+ /**
+ * 刷新某个分片
+ *
+ * @return
+ */
+ boolean flush(Set<String> splitId);
+
+
+ /**
+ * 刷新某个分片
+ *
+ * @return
+ */
+ boolean flush(String... splitIds);
+ /**
+ * 如果支持批量保存，此方法完成数据的全部写入
+ *
+ * @return
+ */
+ boolean flush();
+
+ /**
+ * 调用这个方法后，不必调用flush，由框架定时或定批完成刷新
+ */
+ void openAutoFlush();
+
+ void closeAutoFlush();
+
+ /**
+ * 设置缓存大小，超过条数，强制刷新
+ *
+ * @param batchSize
+ */
+ void setBatchSize(int batchSize);
+
+ /**
+ * 获取批次大小
+ *
+ * @return
+ */
+ int getBatchSize();
+
+ Map<String, MessageOffset> getFinishedQueueIdAndOffsets(CheckPointMessage checkPointMessage);
+
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/sinkcache/DataSourceAutoFlushTask.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/sinkcache/DataSourceAutoFlushTask.java
new file mode 100644
index 0000000..78b7547
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/sinkcache/DataSourceAutoFlushTask.java
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.channel.sinkcache;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.rocketmq.streams.common.channel.sink.AbstractSink;
+import org.apache.rocketmq.streams.common.channel.sinkcache.impl.MessageCache;
+
+/**
+ * 自动刷新缓存的任务，开始openAutoFlush后，可以由独立线程完成数据的flush，不必显式调用
+ */
+public class DataSourceAutoFlushTask implements Runnable {
+
+ private static final Log LOG = LogFactory.getLog(DataSourceAutoFlushTask.class);
+
+ private volatile boolean isAutoFlush = false;
+ private volatile IMessageCache messageCache;
+ protected transient Long lastUpdateTime;
+
+ public DataSourceAutoFlushTask(boolean isAutoFlush,
+ IMessageCache messageCache) {
+ this.isAutoFlush = isAutoFlush;
+ this.messageCache = messageCache;
+ }
+
+ @Override
+ public void run() {
+ while (isAutoFlush) {
+ try {
+ if (messageCache.getMessageCount() < 300 && (lastUpdateTime != null && (System.currentTimeMillis() - lastUpdateTime) < 300)) {
+ Thread.sleep(100);
+ continue;
+ }
+
+ messageCache.flush();
+ lastUpdateTime = System.currentTimeMillis();
+ } catch (Exception e) {
+ e.printStackTrace();
+ }
+ }
+ }
+
+ public boolean isAutoFlush() {
+ return isAutoFlush;
+ }
+
+ public void setAutoFlush(boolean autoFlush) {
+ isAutoFlush = autoFlush;
+ }
+
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/sinkcache/IMessageCache.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/sinkcache/IMessageCache.java
new file mode 100644
index 0000000..d462336
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/sinkcache/IMessageCache.java
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.channel.sinkcache;
+
+import java.util.Set;
+
+import org.apache.rocketmq.streams.common.metadata.MetaData;
+
+/**
+ * 消息缓存组件，支持消息缓存，刷新逻辑
+ *
+ * @param <R>
+ */
+public interface IMessageCache<R> {
+
+ /**
+ * 把消息插入缓存中
+ *
+ * @param msg 待缓存的消息
+ * @return
+ */
+ int addCache(R msg);
+
+ /**
+ * 把队列排空，并写入到存储中
+ *
+ * @return
+ */
+ int flush();
+
+ /**
+ * 刷新某个分片
+ *
+ * @return
+ */
+ int flush(Set<String> splitId);
+
+ /**
+ * 返回消息个数
+ *
+ * @return
+ */
+ Integer getMessageCount();
+
+ /**
+ * 调用这个方法后，不必调用flush，由框架定时或定批完成刷新
+ */
+ void openAutoFlush();
+
+ void closeAutoFlush();
+
+ /**
+ * 设置缓存大小，超过条数，强制刷新
+ *
+ * @param batchSize
+ */
+ void setBatchSize(int batchSize);
+
+ /**
+ * 获取批次大小
+ *
+ * @return
+ */
+ int getBatchSize();
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/sinkcache/IMessageFlushCallBack.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/sinkcache/IMessageFlushCallBack.java
new file mode 100644
index 0000000..325b4e8
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/sinkcache/IMessageFlushCallBack.java
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.channel.sinkcache;
+
+import java.util.List;
+
+import org.apache.rocketmq.streams.common.context.IMessage;
+
+public interface IMessageFlushCallBack<R> {
+ /**
+ * 直接存储存储，不过缓存
+ *
+ * @param messages
+ * @return
+ */
+ boolean flushMessage(List<R> messages);
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/sinkcache/impl/AbstractMutilSplitMessageCache.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/sinkcache/impl/AbstractMutilSplitMessageCache.java
new file mode 100644
index 0000000..c2da923
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/sinkcache/impl/AbstractMutilSplitMessageCache.java
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.channel.sinkcache.impl;
+
+import org.apache.rocketmq.streams.common.channel.sinkcache.IMessageCache;
+import org.apache.rocketmq.streams.common.channel.sinkcache.IMessageFlushCallBack;
+import org.apache.rocketmq.streams.common.context.IMessage;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+
+public abstract class AbstractMutilSplitMessageCache<R> extends MessageCache<R> {
+ protected ConcurrentHashMap<String, MessageCache<IMessage>> queueMessageCaches = new ConcurrentHashMap();
+
+ public AbstractMutilSplitMessageCache(
+ IMessageFlushCallBack<R> flushCallBack) {
+ super(null);
+ this.flushCallBack = new MessageFlushCallBack(flushCallBack);
+ }
+
+ @Override
+ public int addCache(R msg) {
+ String queueId = createSplitId(msg);
+ MessageCache messageCache = new MessageCache(flushCallBack);
+ MessageCache existMessageCache = queueMessageCaches.putIfAbsent(queueId, messageCache);
+ if (existMessageCache != null) {
+ messageCache = existMessageCache;
+ } else {
+ messageCache.setBatchSize(batchSize);
+ messageCache.openAutoFlush();
+ }
+ messageCache.addCache(msg);
+ int size = messageCount.incrementAndGet();
+ if (batchSize > 0 && size >= batchSize) {
+ flush(queueId);
+ size = messageCount.get();
+ }
+ return size;
+ }
+
+ protected abstract String createSplitId(R msg);
+
+ @Override
+ public int flush() {
+ int size = 0;
+ for (IMessageCache cache : this.queueMessageCaches.values()) {
+ size += cache.flush();
+ }
+ return size;
+ }
+
+ @Override
+ public int flush(Set<String> splitIds) {
+ int size=0;
+ if(queueMessageCaches==null||queueMessageCaches.size()==0){
+ return 0;
+ }
+ for(String splitId:splitIds){
+
+ IMessageCache cache= queueMessageCaches.get(splitId);
+ if(cache!=null){
+ size+=cache.flush();
+ }
+
+ }
+ return size;
+ }
+
+ protected int flush(String splitId) {
+ Set<String> splitIds = new HashSet<>();
+ splitIds.add(splitId);
+ return flush(splitIds);
+ }
+
+ @Override
+ public Integer getMessageCount() {
+ return messageCount.get();
+ }
+
+ @Override
+ public void openAutoFlush() {
+ if (this.queueMessageCaches == null) {
+ return;
+ }
+ for (IMessageCache cache : this.queueMessageCaches.values()) {
+ cache.openAutoFlush();
+ }
+ }
+
+ @Override
+ public void closeAutoFlush() {
+ for (IMessageCache cache : this.queueMessageCaches.values()) {
+ cache.closeAutoFlush();
+ }
+ }
+
+ @Override
+ public void setBatchSize(int batchSize) {
+ this.batchSize = batchSize;
+ }
+
+ @Override
+ public int getBatchSize() {
+ return batchSize;
+ }
+
+ protected class MessageFlushCallBack implements IMessageFlushCallBack<R> {
+ protected IMessageFlushCallBack<R> callBack;
+
+ public MessageFlushCallBack(IMessageFlushCallBack<R> callBack) {
+ this.callBack = callBack;
+ }
+
+ @Override
+ public boolean flushMessage(List<R> messages) {
+ boolean success = callBack.flushMessage(messages);
+ messageCount.addAndGet(-messages.size());
+ return success;
+ }
+ }
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/sinkcache/impl/MessageCache.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/sinkcache/impl/MessageCache.java
new file mode 100644
index 0000000..00c0a58
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/sinkcache/impl/MessageCache.java
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.channel.sinkcache.impl;
+
+import org.apache.rocketmq.streams.common.channel.sinkcache.DataSourceAutoFlushTask;
+import org.apache.rocketmq.streams.common.channel.sinkcache.IMessageCache;
+import org.apache.rocketmq.streams.common.channel.sinkcache.IMessageFlushCallBack;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * 消息缓存的实现，通过消息队列做本地缓存。目前多是用了这个实现
+ */
+public class MessageCache<R> implements IMessageCache<R> {
+
+ protected IMessageFlushCallBack<R> flushCallBack;
+
+ protected volatile AtomicInteger messageCount = new AtomicInteger(0);//缓存中的数据条数
+ protected int batchSize = 1000;//最大缓存条数，超过后需要，刷新出去，做内存保护
+ protected transient DataSourceAutoFlushTask autoFlushTask;//自动任务刷新，可以均衡实时性和吞吐率
+ protected volatile transient ConcurrentLinkedQueue<R> dataQueue = new ConcurrentLinkedQueue<>();//缓存数据的消息队列
+
+ protected AtomicBoolean openAutoFlushLock = new AtomicBoolean(false);
+
+ public MessageCache(IMessageFlushCallBack<R> flushCallBack) {
+ this.flushCallBack = flushCallBack;
+ }
+
+ /**
+ * 把待插入的数据写入队列 如果缓存超过batchsize，需要强制刷新
+ *
+ * @param msg
+ * @return
+ */
+ @Override
+ public int addCache(R msg) {
+ offerQueue(msg);
+ int size = messageCount.incrementAndGet();
+ if (batchSize > 0 && size >= batchSize) {
+ flush();
+ }
+ return size;
+ }
+
+ @Override
+ public void openAutoFlush() {
+ if (openAutoFlushLock.compareAndSet(false, true)) {//可重入锁
+ autoFlushTask = new DataSourceAutoFlushTask(true, this);
+ Thread thread = new Thread(autoFlushTask);
+ thread.start();
+ }
+ }
+
+ @Override
+ public void closeAutoFlush() {
+ if (autoFlushTask != null) {
+ autoFlushTask.setAutoFlush(false);
+ openAutoFlushLock.set(false);
+ }
+ }
+
+ protected synchronized void offerQueue(R msg) {
+ dataQueue.offer(msg);
+ }
+
+ protected List<R> getMessagesFromQueue() {
+ List<R> messages = new ArrayList<>();
+ ConcurrentLinkedQueue<R> tmp = dataQueue;
+ dataQueue = new ConcurrentLinkedQueue<>();
+ while (!tmp.isEmpty()) {
+ R msg = tmp.poll();
+ messages.add(msg);
+ }
+ return messages;
+ }
+
+ @Override
+ public Integer getMessageCount() {
+ return messageCount.get();
+ }
+
+ /**
+ * 把队列排空，并写入到存储中
+ *
+ * @return
+ */
+ @Override
+ public int flush() {
+ if (getMessageCount() == 0) {
+ return 0;
+ }
+ synchronized (this) {
+ if (getMessageCount() == 0) {
+ return 0;
+ }
+ List<R> messages = null;
+ messageCount = new AtomicInteger(0);
+ messages = getMessagesFromQueue();
+ flushCallBack.flushMessage(messages);
+ return messages.size();
+ }
+
+ }
+
+ @Override
+ public int flush(Set<String> splitIds) {
+ return flush();
+ }
+
+ @Override
+ public int getBatchSize() {
+ return batchSize;
+ }
+
+ @Override
+ public void setBatchSize(int batchSize) {
+ this.batchSize = batchSize;
+ }
+
+ public IMessageFlushCallBack<R> getFlushCallBack() {
+ return flushCallBack;
+ }
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/sinkcache/impl/MultiSplitMessageCache.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/sinkcache/impl/MultiSplitMessageCache.java
new file mode 100644
index 0000000..622b07b
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/sinkcache/impl/MultiSplitMessageCache.java
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.channel.sinkcache.impl;
+
+import org.apache.rocketmq.streams.common.channel.sinkcache.IMessageFlushCallBack;
+import org.apache.rocketmq.streams.common.context.IMessage;
+
+public class MultiSplitMessageCache extends AbstractMutilSplitMessageCache<IMessage> {
+
+ public MultiSplitMessageCache(
+ IMessageFlushCallBack<IMessage> flushCallBack) {
+ super(flushCallBack);
+ }
+
+ @Override
+ protected String createSplitId(IMessage msg) {
+ return msg.getHeader().getQueueId();
+ }
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/sinkcache/impl/MutilSplitMessageCache.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/sinkcache/impl/MutilSplitMessageCache.java
new file mode 100644
index 0000000..ba8ea90
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/sinkcache/impl/MutilSplitMessageCache.java
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.streams.common.channel.sinkcache.impl;
+
+import org.apache.rocketmq.streams.common.channel.sinkcache.IMessageFlushCallBack;
+import org.apache.rocketmq.streams.common.context.IMessage;
+
+public class MutilSplitMessageCache extends AbstractMutilSplitMessageCache<IMessage>{
+
+ public MutilSplitMessageCache(IMessageFlushCallBack<IMessage> flushCallBack) {
+ super(flushCallBack);
+ }
+
+ @Override
+ protected String createSplitId(IMessage msg) {
+ return msg.getHeader().getQueueId();
+ }
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/source/AbstractBatchSource.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/source/AbstractBatchSource.java
new file mode 100644
index 0000000..e6a16e5
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/source/AbstractBatchSource.java
@@ -0,0 +1,202 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.channel.source;
+
+import java.util.List;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+
+import com.alibaba.fastjson.JSONObject;
+
+import org.apache.rocketmq.streams.common.context.AbstractContext;
+import org.apache.rocketmq.streams.common.context.BatchMessageOffset;
+import org.apache.rocketmq.streams.common.context.Message;
+import org.apache.rocketmq.streams.common.topology.builder.PipelineBuilder;
+import org.apache.rocketmq.streams.common.utils.RuntimeUtil;
+
+/**
+ * 用批处理实现数据流 比如通过sql，定时获取数据，这类非消息队列数据源，没有offset和queueId，系统会模拟实现 也会增加offset的存储，实现断点续传
+ */
+public abstract class AbstractBatchSource extends AbstractSource {
+
+ /**
+ * 一个批次能处理的最大数据量
+ */
+ private static final Long MAX_BATCH_SIZE = 10000000000L;
+
+ /**
+ * 因为是批次数据，所以只有一个queue
+ */
+ private static final String BATCH_MESSAGE_QUEUE_ID = "1";
+
+ /**
+ * 定时发checkpoint
+ */
+ protected transient ScheduledExecutorService scheduled;
+
+ /**
+ * 模拟offset生成，递增产生
+ */
+ protected transient AtomicLong offsetGenerator;
+
+ /**
+ * 最后一次提交的时间，用来判断是否需要checkpoint
+ */
+ protected transient long lastCommitTime;
+
+ private transient BatchMessageOffset progress;//如果需要保存offset，通过这个对象保存
+
+ public AbstractBatchSource() {
+ setBatchMessage(true);
+ }
+
+ @Override
+ protected boolean initConfigurable() {
+ scheduled = new ScheduledThreadPoolExecutor(2);
+ offsetGenerator = new AtomicLong(System.currentTimeMillis());
+ long lastCommitTime = System.currentTimeMillis();
+ return super.initConfigurable();
+ }
+
+ @Override
+ protected boolean startSource() {
+ String queueId = getQueueId();
+ scheduled.scheduleWithFixedDelay(new Runnable() {
+ @Override
+ public void run() {
+ if (System.currentTimeMillis() - lastCommitTime > getCheckpointTime()) {
+ lastCommitTime = System.currentTimeMillis();
+ sendCheckpoint(queueId);
+ }
+
+ }
+ }, 0, getCheckpointTime(), TimeUnit.SECONDS);
+ return true;
+ }
+
+ public AbstractContext doReceiveMessage(JSONObject message) {
+ return doReceiveMessage(message, false);
+ }
+
+ public AbstractContext doReceiveMessage(String message, boolean needFlush) {
+ String queueId = getQueueId();
+ String offset = this.offsetGenerator.incrementAndGet() + "";
+ return doReceiveMessage(message, needFlush, queueId, offset);
+ }
+
+ public AbstractContext doReceiveMessage(JSONObject message, boolean needFlush) {
+ String queueId = getQueueId();
+ String offset = this.offsetGenerator.incrementAndGet() + "";
+ return doReceiveMessage(message, needFlush, queueId, offset);
+ }
+
+ @Override
+ public boolean supportNewSplitFind() {
+ return false;
+ }
+
+ /**
+ * 设置初始化参数和进度
+ *
+ * @param msg
+ */
+ public void setProgress(JSONObject msg) {
+ BatchMessageOffset offset = new BatchMessageOffset();
+ if (msg != null) {
+ offset.setCurrentMessage(msg.toJSONString());
+ }
+ offset.setOwnerType(this.getType());
+ this.progress = offset;
+ }
+
+ @Override
+ public void addConfigurables(PipelineBuilder pipelineBuilder) {
+ super.addConfigurables(pipelineBuilder);
+ if (progress != null) {
+ pipelineBuilder.addConfigurables(progress);
+ }
+ }
+
+ /**
+ * 提供单条消息的处理逻辑，默认不会加入checkpoint
+ *
+ * @param message
+ * @return
+ */
+ @Override
+ public AbstractContext doReceiveMessage(JSONObject message, boolean needSetCheckPoint, String queueId,
+ String offset) {
+ Message msg = createMessage(message, queueId, offset, needSetCheckPoint);
+ return executeMessage(msg);
+ }
+
+ @Override
+ protected boolean isNotDataSplit(String queueId) {
+ return false;
+ }
+
+ /**
+ * 对于批量接入的消息，可以在消息中加入checkpoint，在这批消息执行完成后，flush所有的输出节点，确保消息至少被消费一次
+ *
+ * @param messages 这批消息会作为一个批次
+ * @param needSetCheckPoint 是否在最后一条消息加入checkpoint标志
+ * @return
+ */
+ public AbstractContext doReceiveMessage(List<JSONObject> messages, boolean needSetCheckPoint) {
+ if (messages == null || messages.size() == 0) {
+ return null;
+ }
+
+ AbstractContext context = null;
+ int i = 0;
+ for (JSONObject jsonObject : messages) {
+
+ if (i == messages.size() - 1) {
+ doReceiveMessage(jsonObject, needSetCheckPoint);
+ } else {
+ doReceiveMessage(jsonObject, false);
+ }
+ i++;
+ }
+ return context;
+ }
+
+ @Override
+ public boolean supportRemoveSplitFind() {
+ return false;
+ }
+
+ @Override
+ public boolean supportOffsetRest() {
+ return false;
+ }
+
+ @Override
+ public boolean isBatchMessage() {
+ return true;
+ }
+
+ public String getQueueId() {
+ return RuntimeUtil.getDipperInstanceId();
+ }
+
+ public Long createOffset() {
+ return offsetGenerator.incrementAndGet();
+ }
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/source/AbstractSource.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/source/AbstractSource.java
new file mode 100644
index 0000000..722175f
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/source/AbstractSource.java
@@ -0,0 +1,536 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.channel.source;
+
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import com.alibaba.fastjson.JSONArray;
+import com.alibaba.fastjson.JSONObject;
+
+import org.apache.rocketmq.streams.common.channel.source.systemmsg.NewSplitMessage;
+import org.apache.rocketmq.streams.common.channel.source.systemmsg.RemoveSplitMessage;
+import org.apache.rocketmq.streams.common.checkpoint.CheckPointManager;
+import org.apache.rocketmq.streams.common.checkpoint.CheckPointMessage;
+import org.apache.rocketmq.streams.common.configurable.BasedConfigurable;
+import org.apache.rocketmq.streams.common.configurable.annotation.ENVDependence;
+import org.apache.rocketmq.streams.common.context.AbstractContext;
+import org.apache.rocketmq.streams.common.context.Context;
+import org.apache.rocketmq.streams.common.context.IMessage;
+import org.apache.rocketmq.streams.common.context.Message;
+import org.apache.rocketmq.streams.common.context.MessageHeader;
+import org.apache.rocketmq.streams.common.context.UserDefinedMessage;
+import org.apache.rocketmq.streams.common.interfaces.IStreamOperator;
+import org.apache.rocketmq.streams.common.topology.builder.PipelineBuilder;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+
+/**
+ * channel的抽象，实现了消息的封装，发送等核心逻辑
+ */
+public abstract class AbstractSource extends BasedConfigurable implements ISource<AbstractSource> {
+
+ public static String CHARSET = "UTF-8";
+
+ protected Boolean isJsonData = true;//输入的消息是否为json
+ protected Boolean msgIsJsonArray = false;//输入的消息是否为json array
+ @ENVDependence
+ protected String groupName;//group name
+ protected int maxThread = Runtime.getRuntime().availableProcessors();
+
+ @ENVDependence
+ protected String topic = "";
+
+ /**
+ * 多长时间做一次checkpoint
+ */
+ protected long checkpointTime = 1000 * 60 * 2;
+ /**
+ * 是否是批量消息，批量消息会一批做一次checkpoint，比如通过数据库加载的批消息
+ */
+ protected boolean isBatchMessage = false;
+ /**
+ * 每次拉取的最大条数，多用于消息队列
+ */
+ protected int maxFetchLogGroupSize = 100;
+
+ /**
+ * 数据源投递消息的算子，此算子用来接收source的数据，做处理
+ */
+ protected volatile transient IStreamOperator receiver;
+ /**
+ * 开启mock模式，则会收集mock数据，如果数据源没数据，则会发送mock数据
+ */
+ protected transient volatile Boolean openMock = false;
+
+ protected transient AtomicBoolean hasStart = new AtomicBoolean(false);
+
+ /**
+ * 做checkpoint的管理
+ */
+ protected transient CheckPointManager checkPointManager = new CheckPointManager();
+
+ @Override
+ protected boolean initConfigurable() {
+ hasStart = new AtomicBoolean(false);
+ openMock = false;
+ return super.initConfigurable();
+ }
+
+ @Override
+ public boolean start(IStreamOperator receiver) {
+ this.receiver = receiver;
+ boolean isStartSucess = true;
+ if (hasStart.compareAndSet(false, true)) {
+ isStartSucess = startSource();
+ }
+
+ return isStartSucess;
+ }
+
+ /**
+ * 启动 source
+ *
+ * @return
+ */
+ protected abstract boolean startSource();
+
+ public AbstractSource() {
+ setType(ISource.TYPE);
+ }
+
+ /**
+ * 提供单条消息的处理逻辑，默认不会加入checkpoint
+ *
+ * @param message
+ * @return
+ */
+ public AbstractContext doReceiveMessage(JSONObject message, boolean needSetCheckPoint, String queueId,
+ String offset) {
+ Message msg = createMessage(message, queueId, offset, needSetCheckPoint);
+ AbstractContext context = executeMessage(msg);
+ return context;
+ }
+
+ /**
+ * 处理消息，并且判断是否需要进行加入check point表识别
+ *
+ * @param message
+ * @param needSetCheckPoint
+ * @return
+ */
+ public AbstractContext doReceiveMessage(String message, boolean needSetCheckPoint, String queueId, String offset) {
+ if (this.msgIsJsonArray) {
+ JSONArray jsonArray = JSONObject.parseArray(message);
+ if (jsonArray == null || jsonArray.size() == 0) {
+ return null;
+ }
+ AbstractContext context = null;
+ for (int i = 0; i < jsonArray.size(); i++) {
+ JSONObject msgBody = jsonArray.getJSONObject(i);
+ boolean checkpoint = false;
+ if (needSetCheckPoint && i == jsonArray.size() - 1) {
+ checkpoint = true;
+ }
+ context = doReceiveMessage(msgBody, checkpoint, queueId, createBatchOffset(offset, i));
+ if (!context.isContinue()) {
+ continue;
+ }
+ }
+ return context;
+ } else {
+ JSONObject jsonObject = create(message);
+ return doReceiveMessage(jsonObject, needSetCheckPoint, queueId, offset);
+ }
+ }
+
+ /**
+ * 发送一个系统消息，执行组件不可见，告诉所有组件刷新存储
+ *
+ * @param queueId
+ */
+ public void sendCheckpoint(String queueId) {
+ Set<String> queues = new HashSet<>();
+ queues.add(queueId);
+ sendCheckpoint(queues);
+ }
+
+ /**
+ * 发送系统消息，执行组件不可见，告诉所有组件刷新存储
+ *
+ * @param queueIds
+ */
+ public void sendCheckpoint(Set<String> queueIds) {
+ JSONObject msg = new JSONObject();
+ Message message = createMessage(msg, null, null, true);
+ message.getMessageBody().put("_queues", queueIds);
+ message.getHeader().setCheckpointQueueIds(queueIds);
+ message.getHeader().setNeedFlush(true);
+ message.getHeader().setSystemMessage(true);
+ if (supportOffsetRest()) {
+ message.getHeader().setNeedFlush(false);
+ }
+
+ CheckPointMessage checkPointMessage = new CheckPointMessage();
+ checkPointMessage.setStreamOperator(this.receiver);
+ checkPointMessage.setSource(this);
+ message.setSystemMessage(checkPointMessage);
+ executeMessage(message);
+ if (checkPointMessage.isValidate() && supportOffsetRest()) {
+ saveCheckpoint(checkPointMessage);
+ }
+ }
+
+ protected void saveCheckpoint(CheckPointMessage checkPointMessage) {
+ this.checkPointManager.addCheckPointMessage(checkPointMessage);
+ }
+
+ public JSONObject createJson(Object message) {
+ JSONObject jsonObject = null;
+ if (!isJsonData) {
+ jsonObject = new UserDefinedMessage(message);
+ jsonObject.put(IMessage.DATA_KEY, message);
+ jsonObject.put(IMessage.IS_NOT_JSON_MESSAGE, true);
+ } else {
+ jsonObject = Message.parseObject(message.toString());
+ }
+ return jsonObject;
+
+ }
+
+ public JSONObject create(String message) {
+ return createJson(message);
+ }
+
+ /**
+ * 交给receiver执行后续逻辑
+ *
+ * @param channelMessage
+ * @return
+ */
+ public AbstractContext executeMessage(Message channelMessage) {
+ AbstractContext context = new Context(channelMessage);
+ if (isSplitInRemoving(channelMessage)) {
+ return context;
+ }
+ if (!channelMessage.getHeader().isSystemMessage()) {
+ messageQueueChangedCheck(channelMessage.getHeader());
+ }
+
+ boolean needFlush = channelMessage.getHeader().isSystemMessage() == false && channelMessage.getHeader().isNeedFlush();
+
+ if (receiver != null) {
+ receiver.doMessage(channelMessage, context);
+ }
+ if (needFlush) {
+ sendCheckpoint(channelMessage.getHeader().getQueueId());
+ }
+ executeMessageAfterReceiver(channelMessage, context);
+ return context;
+ }
+
+ protected boolean isSplitInRemoving(Message channelMessage) {
+ return this.checkPointManager.isRemovingSplit(channelMessage.getHeader().getQueueId());
+ }
+
+ /**
+ * source 能否自动返现新增的分片，如果不支持，系统将会模拟实现
+ *
+ * @return
+ */
+ public abstract boolean supportNewSplitFind();
+
+ /**
+ * 能否发现分片移走了，如果不支持，系统会模拟实现
+ *
+ * @return
+ */
+ public abstract boolean supportRemoveSplitFind();
+
+ /**
+ * 是否运行中，在分片发现时，自动设置分片的offset。必须支持supportNewSplitFind
+ *
+ * @return
+ */
+ public abstract boolean supportOffsetRest();
+
+ /**
+ * 系统模拟新分片发现，把消息中的分片保存下来，如果第一次收到，认为是新分片
+ *
+ * @param header
+ */
+ protected void messageQueueChangedCheck(MessageHeader header) {
+ if (supportNewSplitFind() && supportRemoveSplitFind()) {
+ return;
+ }
+ Set<String> queueIds = new HashSet<>();
+ String msgQueueId = header.getQueueId();
+ if (StringUtil.isNotEmpty(msgQueueId)) {
+ queueIds.add(msgQueueId);
+ }
+ Set<String> checkpointQueueIds = header.getCheckpointQueueIds();
+ if (checkpointQueueIds != null) {
+ queueIds.addAll(checkpointQueueIds);
+ }
+ Set<String> newQueueIds = new HashSet<>();
+ Set<String> removeQueueIds = new HashSet<>();
+ for (String queueId : queueIds) {
+ if (isNotDataSplit(queueId)) {
+ continue;
+ }
+ if (StringUtil.isNotEmpty(queueId)) {
+ if (!this.checkPointManager.contains(queueId)) {
+ synchronized (this) {
+ if (!this.checkPointManager.contains(queueId)) {
+ this.checkPointManager.addSplit(queueId);
+ newQueueIds.add(queueId);
+ }
+ }
+ } else {
+
+ this.checkPointManager.updateLastUpdate(queueId);
+ //if(this.checkPointManager.isRemovedSplit(queueId)){
+ // this.checkPointManager.removeSplit(queueId);
+ // removeQueueIds.add(queueId);
+ //}else {
+ //
+ //}
+ }
+ }
+ }
+ //if(!supportRemoveSplitFind()){
+ // removeSplit(removeQueueIds);
+ //}
+ if (!supportNewSplitFind()) {
+ addNewSplit(newQueueIds);
+ }
+
+ }
+
+ protected abstract boolean isNotDataSplit(String queueId);
+
+ @Override
+ public void removeSplit(Set<String> splitIds) {
+ if (splitIds == null || splitIds.size() == 0) {
+ return;
+ }
+ this.checkPointManager.addRemovingSplit(splitIds);
+ sendRemoveSplitSystemMessage(splitIds);
+ //先保存所有的分片
+ sendCheckpoint(splitIds);
+ this.checkPointManager.flush();
+ synchronized (this) {
+ for (String splitId : splitIds) {
+ this.checkPointManager.removeSplit(splitId);
+ }
+
+ }
+ }
+
+ @Override
+ public void addNewSplit(Set<String> splitIds) {
+ if (splitIds == null || splitIds.size() == 0) {
+ return;
+ }
+ this.checkPointManager.deleteRemovingSplit(splitIds);
+
+ JSONObject msg = new JSONObject();
+ Message message = createMessage(msg, null, null, false);
+ message.getMessageBody().put("_queues", splitIds);
+ //message.getHeader().setCheckpointQueueIds(queueIds);
+
+ message.getHeader().setNeedFlush(false);
+ message.getHeader().setSystemMessage(true);
+ NewSplitMessage systemMessage = new NewSplitMessage(splitIds, this.checkPointManager.getCurrentSplits());
+ systemMessage.setStreamOperator(this.receiver);
+ systemMessage.setSource(this);
+ message.setSystemMessage(systemMessage);
+ executeMessage(message);
+ }
+
+ /**
+ * 发送系统消息，执行组件不可见，告诉所有组件刷新存储
+ *
+ * @param queueIds
+ */
+ public void sendRemoveSplitSystemMessage(Set<String> queueIds) {
+ JSONObject msg = new JSONObject();
+ Message message = createMessage(msg, null, null, true);
+ message.getMessageBody().put("_queues", queueIds);
+ //message.getHeader().setCheckpointQueueIds(queueIds);
+ message.getHeader().setNeedFlush(true);
+ message.getHeader().setSystemMessage(true);
+ Set<String> currentSplitIds = new HashSet<>();
+ currentSplitIds.addAll(this.checkPointManager.getCurrentSplits());
+ for (String queueId : queueIds) {
+ currentSplitIds.remove(queueId);
+ }
+ RemoveSplitMessage systemMessage = new RemoveSplitMessage(queueIds, currentSplitIds);
+ systemMessage.setStreamOperator(this.receiver);
+ systemMessage.setSource(this);
+ message.setSystemMessage(systemMessage);
+ executeMessage(message);
+ }
+
+ /**
+ * 如果存在offset，做更新，这里的offset是批流的offset，有系统创建和保存，多用于数据库查询结果场景
+ *
+ * @param channelMessage
+ * @param context
+ */
+ protected void executeMessageAfterReceiver(Message channelMessage, AbstractContext context) {
+ //如果有进度，则保存进度
+ if (channelMessage.getHeader() != null && channelMessage.getHeader().getProgress() != null) {
+ JSONObject msg = channelMessage.getHeader().getProgress().getCurrentMsg();
+ Iterator<Entry<String, Object>> it = msg.entrySet().iterator();
+ JSONObject newMsg = new JSONObject();
+ newMsg.putAll(msg);
+ while (it.hasNext()) {
+ Entry<String, Object> entry = it.next();
+ String key = entry.getKey();
+ if (channelMessage.getMessageBody().containsKey(key)) {
+ newMsg.put(key, channelMessage.getMessageBody().get(key));
+ }
+ }
+ channelMessage.getHeader().getProgress().setCurrentMessage(newMsg.toJSONString());
+ channelMessage.getHeader().getProgress().update();
+ }
+
+ }
+
+ /**
+ * 把json 转换成一个message对象
+ *
+ * @param msg
+ * @return
+ */
+ public Message createMessage(JSONObject msg, String queueId, String offset, boolean checkpoint) {
+ Message channelMessage = new Message(msg);
+ channelMessage.getHeader().setSource(this);
+ channelMessage.getHeader().setOffset(offset);
+ channelMessage.getHeader().setQueueId(queueId);
+ channelMessage.getHeader().setNeedFlush(checkpoint);
+ channelMessage.setJsonMessage(isJsonData);
+ return channelMessage;
+ }
+
+ /**
+ * 每批次通过加小序号来区分offset的大小
+ *
+ * @param offset
+ * @param i
+ * @return
+ */
+ private String createBatchOffset(String offset, int i) {
+ String index = "" + i;
+ for (int j = index.length(); j < 5; j++) {
+ index = "0" + index;
+ }
+ return offset + index;
+ }
+
+ @Override
+ public void setMaxFetchLogGroupSize(int size) {
+ this.maxFetchLogGroupSize = size;
+ }
+
+ @Override
+ public AbstractSource createStageChain(PipelineBuilder pipelineBuilder) {
+ return this;
+ }
+
+ @Override
+ public void addConfigurables(PipelineBuilder pipelineBuilder) {
+ pipelineBuilder.addConfigurables(this);
+ }
+
+ @Override
+ public String getGroupName() {
+ return groupName;
+ }
+
+ @Override
+ public void setGroupName(String groupName) {
+ this.groupName = groupName;
+ }
+
+ @Override
+ public int getMaxThread() {
+ return maxThread;
+ }
+
+ @Override
+ public void setMaxThread(int maxThread) {
+ this.maxThread = maxThread;
+ }
+
+ public IStreamOperator getReceiver() {
+ return receiver;
+ }
+
+ public void setReceiver(IStreamOperator receiver) {
+ this.receiver = receiver;
+ }
+
+ public Boolean getJsonData() {
+ return isJsonData;
+ }
+
+ public void setJsonData(Boolean jsonData) {
+ isJsonData = jsonData;
+ }
+
+ public Boolean getMsgIsJsonArray() {
+ return msgIsJsonArray;
+ }
+
+ public void setMsgIsJsonArray(Boolean msgIsJsonArray) {
+ this.msgIsJsonArray = msgIsJsonArray;
+ }
+
+ public void setBatchMessage(boolean batchMessage) {
+ isBatchMessage = batchMessage;
+ }
+
+ public int getMaxFetchLogGroupSize() {
+ return maxFetchLogGroupSize;
+ }
+
+ public String getTopic() {
+ return topic;
+ }
+
+ public void setTopic(String topic) {
+ this.topic = topic;
+ }
+
+ public void setCheckpointTime(long checkpointTime) {
+ this.checkpointTime = checkpointTime;
+ }
+
+ @Override
+ public long getCheckpointTime() {
+ return checkpointTime;
+ }
+
+ public boolean isBatchMessage() {
+ return isBatchMessage;
+ }
+
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/source/AbstractSupportOffsetResetSource.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/source/AbstractSupportOffsetResetSource.java
new file mode 100644
index 0000000..78b6597
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/source/AbstractSupportOffsetResetSource.java
@@ -0,0 +1,21 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.channel.source;
+
+public abstract class AbstractSupportOffsetResetSource extends AbstractSource {
+
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/source/AbstractUnreliableSource.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/source/AbstractUnreliableSource.java
new file mode 100644
index 0000000..410c96d
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/source/AbstractUnreliableSource.java
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.channel.source;
+
+import com.alibaba.fastjson.JSONObject;
+import com.lmax.disruptor.EventHandler;
+import com.lmax.disruptor.YieldingWaitStrategy;
+import com.lmax.disruptor.dsl.Disruptor;
+import com.lmax.disruptor.dsl.ProducerType;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.rocketmq.streams.common.context.AbstractContext;
+import org.apache.rocketmq.streams.common.context.Message;
+import org.apache.rocketmq.streams.common.disruptor.BufferFullFunction;
+import org.apache.rocketmq.streams.common.disruptor.DisruptorEvent;
+import org.apache.rocketmq.streams.common.disruptor.DisruptorEventFactory;
+import org.apache.rocketmq.streams.common.disruptor.DisruptorProducer;
+
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * 不可靠的消息源，如http，syslog，可以继承这个类。做了系统保护，如果消息发送太快，可能会出现丢失。
+ */
+public abstract class AbstractUnreliableSource extends AbstractBatchSource {
+ private static final Log LOG = LogFactory.getLog(AbstractUnreliableSource.class);
+
+ protected Boolean enableAsyncReceive = true;
+ protected boolean isSingleType = false;//是否只有单个生产者，如果是，则为true
+
+ private transient ExecutorService cachedThreadPool = null;
+ private transient int bufferSize = 1024;
+ private transient Disruptor<DisruptorEvent> disruptor;
+ private transient DisruptorProducer<Message> disruptorProducer;
+ private transient BufferFullFunction bufferFullFunction;
+ protected transient boolean discard = false;//如果过快，直接丢弃。只有enableAsyncReceive生效时使用
+ private transient EventHandler<DisruptorEvent> eventEventHandler;
+
+ @Override
+ protected boolean initConfigurable() {
+ bufferSize = 1024;
+ boolean discard = false;//如果过快，直接丢弃。只有enableAsyncReceive生效时使用
+ return super.initConfigurable();
+ }
+
+ public AbstractUnreliableSource() {
+ super();
+ if (!enableAsyncReceive) {
+ return;
+ }
+ cachedThreadPool = new ThreadPoolExecutor(maxThread, maxThread,
+ 0L, TimeUnit.MILLISECONDS,
+ new LinkedBlockingQueue<Runnable>());
+ ProducerType producerType = ProducerType.MULTI;
+ if (isSingleType) {
+ producerType = ProducerType.SINGLE;
+ }
+ disruptor = new Disruptor<>(new DisruptorEventFactory(), bufferSize, cachedThreadPool, producerType,
+ new YieldingWaitStrategy());
+ eventEventHandler = new MessageEventHandler();
+ disruptor.handleEventsWith(eventEventHandler);
+ disruptor.start();
+ disruptorProducer = new DisruptorProducer<>(disruptor);
+ //bufferFullFunction = data -> batchAdd((JSONObject)data);
+ bufferFullFunction = new BufferFullFunction() {
+ @Override
+ public void process(Object data) {
+ LOG.warn("discard data ");
+ }
+ };
+ }
+
+ @Override
+ public AbstractContext executeMessage(Message channelMessage) {
+ if (enableAsyncReceive) {
+ disruptorProducer.publish(channelMessage, bufferFullFunction, discard);
+ return null;
+ } else {
+ return executeMessageBySupper(channelMessage);
+ }
+
+ }
+
+ @Override
+ public boolean supportRemoveSplitFind() {
+ return false;
+ }
+
+ @Override
+ public boolean supportOffsetRest() {
+ return false;
+ }
+
+ @Override
+ protected boolean isNotDataSplit(String queueId) {
+ return false;
+ }
+
+ /**
+ * 收到消息发送出去，因为是不可靠队列，如果对象不是json，则用UserDefinedMessage处理
+ *
+ * @param t
+ * @param <T>
+ * @return
+ */
+ public <T> AbstractContext doUnreliableReceiveMessage(T t) {
+ return super.doReceiveMessage(createJson(t));
+ }
+
+ @Override
+ public AbstractContext doReceiveMessage(JSONObject message) {
+ return doUnreliableReceiveMessage(message);
+ }
+
+ @Override
+ protected void executeMessageAfterReceiver(Message channelMessage, AbstractContext context) {
+ }
+
+ public Boolean getEnableAsyncReceive() {
+ return enableAsyncReceive;
+ }
+
+ public void setEnableAsyncReceive(Boolean enableAsyncReceive) {
+ this.enableAsyncReceive = enableAsyncReceive;
+ }
+
+ public AbstractContext executeMessageBySupper(Message msg) {
+ return super.executeMessage(msg);
+
+ }
+
+ protected class MessageEventHandler implements EventHandler<DisruptorEvent> {
+ @Override
+ public void onEvent(DisruptorEvent event, long sequence, boolean endOfBatch) throws Exception {
+ // LOG.info("get event " + event);
+ Message msg = (Message)event.getData();
+ executeMessageBySupper(msg);
+ }
+ }
+
+ public boolean isSingleType() {
+ return isSingleType;
+ }
+
+ public void setSingleType(boolean singleType) {
+ isSingleType = singleType;
+ }
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/source/ISource.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/source/ISource.java
new file mode 100644
index 0000000..5b79b10
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/source/ISource.java
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.channel.source;
+
+import java.util.Set;
+
+import org.apache.rocketmq.streams.common.channel.split.ISplit;
+import org.apache.rocketmq.streams.common.configurable.IConfigurable;
+import org.apache.rocketmq.streams.common.interfaces.IStreamOperator;
+import org.apache.rocketmq.streams.common.topology.builder.IStageBuilder;
+
+public interface ISource<T extends ISource> extends IConfigurable, IStageBuilder<T> {
+ String TYPE = "source";
+
+ /**
+ * 开始接收数据，把收到的数据交给receiver处理
+ *
+ * @param receiver 处理流数据
+ * @return 是否正常启动
+ */
+ boolean start(IStreamOperator receiver);
+
+ /**
+ * 同一个group name共同消费一份数据，主要针对消息队列，如果实现的子类用到这个字段，需要保持语义
+ *
+ * @return 组名
+ */
+ String getGroupName();
+
+ /**
+ * 设置组名
+ *
+ * @param groupName 组名
+ */
+ void setGroupName(String groupName);
+
+ /**
+ * 需要的最大处理线程
+ *
+ * @return 最大处理线程
+ */
+ int getMaxThread();
+
+ /**
+ * 设置线程
+ *
+ * @param maxThread 线程数
+ */
+ void setMaxThread(int maxThread);
+
+ /**
+ * 每次最大抓取个数，只针对消息队列适用，这个参数主要可以控制内存占用
+ *
+ * @param size 每次最大抓取个数
+ */
+ void setMaxFetchLogGroupSize(int size);
+
+ /**
+ * 消息超过多长时间，会被checkpoint一次，对于批量消息无效
+ *
+ * @return checkpoint时间
+ */
+ long getCheckpointTime();
+
+ /**
+ * 当分片被移走前需要做的回调
+ *
+ * @param splitIds 要移走的分片
+ */
+ void removeSplit(Set<String> splitIds);
+
+ /**
+ * 当新增分片时，需要做的回调
+ */
+ void addNewSplit(Set<String> splitIds);
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/source/ISplitChangedListener.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/source/ISplitChangedListener.java
new file mode 100644
index 0000000..fc959ab
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/source/ISplitChangedListener.java
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.channel.source;
+
+import org.apache.rocketmq.streams.common.channel.source.systemmsg.NewSplitMessage;
+import org.apache.rocketmq.streams.common.channel.source.systemmsg.RemoveSplitMessage;
+import org.apache.rocketmq.streams.common.checkpoint.CheckPointMessage;
+import org.apache.rocketmq.streams.common.context.AbstractContext;
+import org.apache.rocketmq.streams.common.context.IMessage;
+
+public interface ISplitChangedListener {
+ /**
+ * 当channel保存commit offset时，发送系统消息，收到系统消息的stage需要完成state保存。
+ *
+ * @param message
+ * @param context
+ */
+ void addNewSplit(IMessage message, AbstractContext context, NewSplitMessage newSplitMessage);
+
+ /**
+ * 当channel保存commit offset时，发送系统消息，收到系统消息的stage需要完成state保存。
+ *
+ * @param message
+ * @param context
+ */
+ void removeSplit(IMessage message, AbstractContext context, RemoveSplitMessage removeSplitMessage);
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/source/systemmsg/NewSplitMessage.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/source/systemmsg/NewSplitMessage.java
new file mode 100644
index 0000000..a3c6bc5
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/source/systemmsg/NewSplitMessage.java
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.channel.source.systemmsg;
+
+import java.util.Set;
+
+public class NewSplitMessage extends SplitChangedMessage {
+
+ public NewSplitMessage(Set<String> splitIds, Set<String> currentSplitIds) {
+ super(splitIds, currentSplitIds, false);
+ }
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/source/systemmsg/RemoveSplitMessage.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/source/systemmsg/RemoveSplitMessage.java
new file mode 100644
index 0000000..59540bd
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/source/systemmsg/RemoveSplitMessage.java
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.channel.source.systemmsg;
+
+import java.util.Set;
+
+public class RemoveSplitMessage extends SplitChangedMessage {
+
+ public RemoveSplitMessage(Set<String> splitIds, Set<String> currentSplitIds) {
+ super(splitIds, currentSplitIds, true);
+ }
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/source/systemmsg/SplitChangedMessage.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/source/systemmsg/SplitChangedMessage.java
new file mode 100644
index 0000000..909ebd7
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/source/systemmsg/SplitChangedMessage.java
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.channel.source.systemmsg;
+
+import java.util.Set;
+
+import org.apache.rocketmq.streams.common.channel.source.ISource;
+import org.apache.rocketmq.streams.common.interfaces.IStreamOperator;
+import org.apache.rocketmq.streams.common.interfaces.ISystemMessage;
+
+public class SplitChangedMessage implements ISystemMessage {
+ protected Set<String> splitIds;
+ protected boolean needFlush;//需要同步刷新
+ protected ISource source;//数据源对象
+ protected IStreamOperator streamOperator;//当前的pipline
+ protected Set<String> currentSplitIds = null;
+
+ public SplitChangedMessage(Set<String> splitIds, Set<String> currentSplitIds, boolean needFlush) {
+ this.splitIds = splitIds;
+ this.needFlush = needFlush;
+ this.currentSplitIds = currentSplitIds;
+ }
+
+ public Set<String> getSplitIds() {
+ return splitIds;
+ }
+
+ public void setSplitIds(Set<String> splitIds) {
+ this.splitIds = splitIds;
+ }
+
+ public boolean isNeedFlush() {
+ return needFlush;
+ }
+
+ public void setNeedFlush(boolean needFlush) {
+ this.needFlush = needFlush;
+ }
+
+ public ISource getSource() {
+ return source;
+ }
+
+ public void setSource(ISource source) {
+ this.source = source;
+ }
+
+ public IStreamOperator getStreamOperator() {
+ return streamOperator;
+ }
+
+ public void setStreamOperator(IStreamOperator streamOperator) {
+ this.streamOperator = streamOperator;
+ }
+
+ public Set<String> getCurrentSplitIds() {
+ return currentSplitIds;
+ }
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/split/ISplit.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/split/ISplit.java
new file mode 100644
index 0000000..679f0cd
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/channel/split/ISplit.java
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.channel.split;
+
+import org.apache.rocketmq.streams.common.datatype.IJsonable;
+
+import java.io.Serializable;
+
+/**
+ * 对消息队列分片的抽象。代表一个分片
+ *
+ * @param <T>
+ * @param <Q>
+ */
+public interface ISplit<T, Q> extends Comparable<T>, Serializable, IJsonable {
+
+ String getQueueId();
+
+ /**
+ * 比当前queueId大的queueId的值，最好只大一点点，主要应用在rocksdb 做范围查询,因为结束部分是开区间，只要大一点点就行
+ *
+ * @return
+ */
+ String getPlusQueueId();
+
+ /**
+ * 获取具体的队列 获取具体的队列
+ *
+ * @return
+ */
+ Q getQueue();
+
+ // public T getQueue();
+
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/checkpoint/CheckPoint.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/checkpoint/CheckPoint.java
new file mode 100644
index 0000000..d8d1305
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/checkpoint/CheckPoint.java
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.checkpoint;
+
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.rocketmq.streams.common.channel.source.ISource;
+import org.apache.rocketmq.streams.common.model.Entity;
+
+/**
+ * model for checkpoint，need save in store
+ */
+public class CheckPoint extends Entity {
+ protected String sourceNamespace;
+ protected String sourceName;
+ protected String queueId;
+ protected String offset;
+
+ public static String loadOffset(ISource source, String splitId) {
+ return null;
+ //Set<String> splits=new HashSet<>();
+ //splits.add(splitId);
+ //Map<String,String> queueId2Offset=loadOffset(source,splits);
+ //if(queueId2Offset==null||queueId2Offset.containsKey(splitId)==false){
+ // return null;
+ //}
+ //return queueId2Offset.get(splitId);
+ }
+
+ public String getQueueId() {
+ return queueId;
+ }
+
+ public void setQueueId(String queueId) {
+ this.queueId = queueId;
+ }
+
+ public String getOffset() {
+ return offset;
+ }
+
+ public void setOffset(String offset) {
+ this.offset = offset;
+ }
+
+ public String getSourceNamespace() {
+ return sourceNamespace;
+ }
+
+ public void setSourceNamespace(String sourceNamespace) {
+ this.sourceNamespace = sourceNamespace;
+ }
+
+ public String getSourceName() {
+ return sourceName;
+ }
+
+ public void setSourceName(String sourceName) {
+ this.sourceName = sourceName;
+ }
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/checkpoint/CheckPointManager.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/checkpoint/CheckPointManager.java
new file mode 100644
index 0000000..e1eae3d
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/checkpoint/CheckPointManager.java
@@ -0,0 +1,279 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.checkpoint;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+
+import org.apache.rocketmq.streams.common.channel.sinkcache.IMessageCache;
+import org.apache.rocketmq.streams.common.channel.sinkcache.IMessageFlushCallBack;
+import org.apache.rocketmq.streams.common.channel.sinkcache.impl.MessageCache;
+import org.apache.rocketmq.streams.common.channel.source.ISource;
+import org.apache.rocketmq.streams.common.configurable.IConfigurableIdentification;
+import org.apache.rocketmq.streams.common.context.MessageOffset;
+import org.apache.rocketmq.streams.common.utils.MapKeyUtil;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+
+public class CheckPointManager {
+ protected IMessageCache<CheckPointMessage> messageCache;
+ protected transient Map<String, Long> currentSplitAndLastUpdateTime = new HashMap<>();//保存这个实例处理的分片数
+
+ protected transient Map<String, Long> removingSplits = new HashMap<>();//正在删除的分片
+
+ public CheckPointManager() {
+ messageCache = new MessageCache<>(new IMessageFlushCallBack<CheckPointMessage>() {
+ @Override
+ public boolean flushMessage(List<CheckPointMessage> messages) {
+ //合并最近的checkpoint，只存储一次，为了
+ Map<String, SourceState> sourceStateMap = mergeSourceState(messages);
+ saveCheckPoint(sourceStateMap);
+ return true;
+ }
+ });
+ messageCache.openAutoFlush();
+ }
+
+ public void flush() {
+ messageCache.flush();
+ }
+
+ public synchronized void addSplit(String splitId) {
+ this.currentSplitAndLastUpdateTime.put(splitId, System.currentTimeMillis());
+ }
+
+ public synchronized void removeSplit(String splitId) {
+ this.currentSplitAndLastUpdateTime.remove(splitId);
+ }
+
+ public boolean contains(String splitId) {
+ return this.currentSplitAndLastUpdateTime.containsKey(splitId);
+ }
+
+ /**
+ * 可能有多次的offset合并在一起，对offset合并 合并包含两个逻辑：1.同1个CheckPointMessage中，选择最小的作为本次的offset
+ *
+ * @param messages
+ */
+ protected Map<String, SourceState> mergeSourceState(List<CheckPointMessage> messages) {
+ Map<String, SourceState> sourceStateMap = new HashMap<>();
+ for (CheckPointMessage checkPointMessage : messages) {
+ SourceState sourceState = createSourceState(checkPointMessage);
+ if (sourceState == null) {
+ continue;
+ }
+ String sourceName = sourceState.getSourceName();
+ SourceState existSourceState = sourceStateMap.get(sourceName);
+ if (existSourceState != null) {
+ SourceState lastSourceState = merge(sourceState, existSourceState);
+ sourceStateMap.put(sourceName, lastSourceState);
+ }
+ }
+ return sourceStateMap;
+ }
+
+ /**
+ * 一个pipeline流程中，找最小的offset提交保存
+ *
+ * @param checkPointMessage
+ * @return
+ */
+ protected SourceState createSourceState(CheckPointMessage checkPointMessage) {
+ SourceState sourceState = new SourceState();
+ String pipelineName = null;
+ if (checkPointMessage.getStreamOperator() instanceof IConfigurableIdentification) {
+ IConfigurableIdentification configurable = (IConfigurableIdentification)checkPointMessage.getCheckPointStates();
+ pipelineName = configurable.getConfigureName();
+ }
+ Map<String, MessageOffset> queueId2Offsets = new HashMap<>();
+ sourceState.setSourceName(createSourceName(checkPointMessage.getSource(), pipelineName));
+ sourceState.setQueueId2Offsets(queueId2Offsets);
+
+ for (CheckPointState checkPointState : checkPointMessage.getCheckPointStates()) {
+ if (checkPointState.isReplyAnyOny()) {
+ continue;
+ }
+ if (checkPointState.isReplyRefuse()) {
+ return null;
+ }
+ for (Entry<String, MessageOffset> entry : checkPointState.getQueueIdAndOffset().entrySet()) {
+ String queueId = entry.getKey();
+ MessageOffset offset = entry.getValue();
+ MessageOffset existOffset = queueId2Offsets.get(queueId);
+ if (existOffset == null) {
+ queueId2Offsets.put(queueId, offset);
+ } else {
+ boolean isGreateThan = existOffset.greateThan(offset.getOffsetStr());
+ if (isGreateThan) {
+ queueId2Offsets.put(queueId, offset);
+ } else {
+ queueId2Offsets.put(queueId, existOffset);
+ }
+ }
+ }
+ }
+ return sourceState;
+ }
+
+ /**
+ * 先查询现在数据源的分片，如果已经不处理的分片，不做保存 否则把结果保存到db中
+ *
+ * @param sourceStateMap
+ */
+ protected void saveCheckPoint(Map<String, SourceState> sourceStateMap) {
+ List<CheckPoint> checkPoints = new ArrayList<>();
+ for (SourceState sourceState : sourceStateMap.values()) {
+ CheckPoint checkPoint = new CheckPoint();
+ // checkPoint.setOffset();
+ }
+ }
+
+ /**
+ * 如果多次的checkpoint在一起，先合并再保存
+ *
+ * @param sourceState
+ * @param existSourceState
+ * @return
+ */
+ protected SourceState merge(SourceState sourceState, SourceState existSourceState) {
+ Iterator<Entry<String, MessageOffset>> it = sourceState.getQueueId2Offsets().entrySet()
+ .iterator();
+ while (it.hasNext()) {
+ Entry<String, MessageOffset> entry = it.next();
+ String queueId = entry.getKey();
+ MessageOffset offset = entry.getValue();
+ MessageOffset existOffset = existSourceState.getQueueId2Offsets().get(queueId);
+ if (existOffset == null) {
+ existSourceState.getQueueId2Offsets().put(queueId, offset);
+ } else {
+ boolean isGreaterThan = offset.greateThan(existOffset.getOffsetStr());
+ if (isGreaterThan) {
+ existSourceState.getQueueId2Offsets().put(queueId, offset);
+ }
+ }
+ }
+ return existSourceState;
+ }
+
+ public void addCheckPointMessage(CheckPointMessage message) {
+ this.messageCache.addCache(message);
+ }
+
+ //public boolean isRemovedSplit(String queueId) {
+ // Long lastUpdateTime=this.currentSplitAndLastUpdateTime.get(queueId);
+ // if(lastUpdateTime==null){
+ // return false;
+ // }
+ // if(System.currentTimeMillis()-lastUpdateTime>10000*1000){
+ // return true;
+ // }
+ // return false;
+ //}
+
+ public void updateLastUpdate(String queueId) {
+ addSplit(queueId);
+ }
+
+ public Set<String> getCurrentSplits() {
+ return this.currentSplitAndLastUpdateTime.keySet();
+ }
+
+ public static class SourceState {
+ protected String sourceName;
+ protected Map<String, MessageOffset> queueId2Offsets = new HashMap<>();
+
+ public String getSourceName() {
+ return sourceName;
+ }
+
+ public void setSourceName(String sourceName) {
+ this.sourceName = sourceName;
+ }
+
+ public Map<String, MessageOffset> getQueueId2Offsets() {
+ return queueId2Offsets;
+ }
+
+ public void setQueueId2Offsets(
+ Map<String, MessageOffset> queueId2Offsets) {
+ this.queueId2Offsets = queueId2Offsets;
+ }
+ }
+
+ /**
+ * 根据source进行划分，主要是针对双流join的场景
+ *
+ * @param source
+ * @return
+ */
+ public static String createSourceName(ISource source, String piplineName) {
+ String namespace = null;
+ String name = null;
+ if (source != null) {
+ namespace = source.getNameSpace();
+ name = source.getConfigureName();
+ }
+
+ if (StringUtil.isEmpty(namespace)) {
+ namespace = "default_namespace";
+ }
+ if (StringUtil.isEmpty(name)) {
+ name = "default_name";
+ }
+ if (StringUtil.isEmpty(piplineName)) {
+ piplineName = "default_piplineName";
+ }
+ return MapKeyUtil.createKey(namespace, piplineName, name);
+ }
+
+ public Map<String, Long> getCurrentSplitAndLastUpdateTime() {
+ return currentSplitAndLastUpdateTime;
+ }
+
+ public synchronized void addRemovingSplit(Set<String> removingSplits) {
+ long removingTime = System.currentTimeMillis();
+ for (String split : removingSplits) {
+ this.removingSplits.put(split, removingTime);
+ }
+ }
+
+ public synchronized void deleteRemovingSplit(Set<String> removingSplits) {
+ for (String split : removingSplits) {
+ this.removingSplits.remove(split);
+ }
+
+ }
+
+ public synchronized boolean isRemovingSplit(String splitId) {
+ Long removingTime = this.removingSplits.get(splitId);
+ if (removingTime == null) {
+ return false;
+ }
+ //超过10秒才允许当作新分片进来
+ if (System.currentTimeMillis() - removingTime > 10 * 1000) {
+ this.removingSplits.remove(splitId);
+ return false;
+ }
+ return true;
+ }
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/checkpoint/CheckPointMessage.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/checkpoint/CheckPointMessage.java
new file mode 100644
index 0000000..975ccb8
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/checkpoint/CheckPointMessage.java
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.checkpoint;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import javafx.stage.Stage;
+import org.apache.rocketmq.streams.common.channel.source.ISource;
+import org.apache.rocketmq.streams.common.interfaces.IStreamOperator;
+import org.apache.rocketmq.streams.common.interfaces.ISystemMessage;
+import org.apache.rocketmq.streams.common.topology.ChainPipeline;
+import org.apache.rocketmq.streams.common.topology.ChainStage;
+
+public class CheckPointMessage implements ISystemMessage {
+ protected ISource source;//数据源对象
+ protected IStreamOperator streamOperator;//当前的pipline
+ protected List<CheckPointState> checkPointStates = new ArrayList<>();
+ protected boolean isValidate = true;
+
+ public ISource getSource() {
+ return source;
+ }
+
+ public void setSource(ISource source) {
+ this.source = source;
+ }
+
+ public IStreamOperator getStreamOperator() {
+ return streamOperator;
+ }
+
+ public void setStreamOperator(IStreamOperator streamOperator) {
+ this.streamOperator = streamOperator;
+ }
+
+ public List<CheckPointState> getCheckPointStates() {
+ return checkPointStates;
+ }
+
+ public void setCheckPointStates(
+ List<CheckPointState> checkPointStates) {
+ this.checkPointStates = checkPointStates;
+ }
+
+ public void reply(CheckPointState checkPointState) {
+ checkPointStates.add(checkPointState);
+ }
+
+ public void replyAnyone() {
+
+ }
+
+ public void replyRefuse() {
+ isValidate = false;
+ }
+
+ public boolean isValidate() {
+ return isValidate;
+ }
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/checkpoint/CheckPointState.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/checkpoint/CheckPointState.java
new file mode 100644
index 0000000..73508b0
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/checkpoint/CheckPointState.java
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.checkpoint;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.rocketmq.streams.common.context.MessageOffset;
+import org.apache.rocketmq.streams.common.topology.ChainPipeline;
+import org.apache.rocketmq.streams.common.topology.ChainStage;
+
+public class CheckPointState {
+ protected Map<String, MessageOffset> queueIdAndOffset = new HashMap<>();//存储已经处理完成的队列id和最大的offset
+ /**
+ * 0：基于queueIdAndOffset作为已经完成的状态；-1，本次状态不反馈，请忽略，此时不做offset保存；1.我已经反馈，但我不存储状态，以其他stage状态为主
+ */
+ protected int resultType = 0;
+
+ public Map<String, MessageOffset> getQueueIdAndOffset() {
+ return queueIdAndOffset;
+ }
+
+ public void setQueueIdAndOffset(
+ Map<String, MessageOffset> queueIdAndOffset) {
+ this.queueIdAndOffset = queueIdAndOffset;
+ }
+
+ public boolean isReplyRefuse() {
+ return this.resultType == -1;
+ }
+
+ public boolean isReplyAnyOny() {
+ return this.resultType == 1;
+ }
+
+ public void replyRefuse() {
+ this.resultType = -1;
+ }
+
+ public void replyAnyOne() {
+ this.resultType = 1;
+ }
+
+ public int getResultType() {
+ return resultType;
+ }
+
+ public void setResultType(int resultType) {
+ this.resultType = resultType;
+ }
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/checkpoint/ICheckPoint.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/checkpoint/ICheckPoint.java
new file mode 100644
index 0000000..50d9c7a
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/checkpoint/ICheckPoint.java
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.checkpoint;
+
+import org.apache.rocketmq.streams.common.context.AbstractContext;
+import org.apache.rocketmq.streams.common.context.IMessage;
+
+public interface ICheckPoint {
+
+ /**
+ * 当channel保存commit offset时，发送系统消息，收到系统消息的stage需要完成state保存。
+ *
+ * @param message
+ * @param context
+ */
+ void checkpoint(IMessage message, AbstractContext context, CheckPointMessage checkPointMessage);
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/classloader/ByteClassLoader.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/classloader/ByteClassLoader.java
new file mode 100644
index 0000000..92f3f36
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/classloader/ByteClassLoader.java
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.classloader;
+
+public class ByteClassLoader extends ClassLoader {
+ public ByteClassLoader(ClassLoader classLoader) {
+ super(classLoader);
+ }
+
+ public Class<?> defineClass(String name, byte[] b) {
+ // ClassLoader是个抽象类，而ClassLoader.defineClass 方法是protected的
+ // 所以我们需要定义一个子类将这个方法暴露出来
+ return super.defineClass(name, b, 0, b.length);
+ }
+
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/classloader/FileClassLoader.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/classloader/FileClassLoader.java
new file mode 100644
index 0000000..0bf6c10
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/classloader/FileClassLoader.java
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.classloader;
+
+import java.io.*;
+
+/**
+ * 加载指定目录的class
+ */
+public class FileClassLoader extends ClassLoader {
+
+ protected String dir;
+
+ public FileClassLoader(String dir, ClassLoader parent) {
+ super(parent);
+ this.dir = dir;
+ }
+
+ //用于寻找类文件
+ @Override
+ public Class findClass(String name) {
+ byte[] b = loadClassData(name);
+ return defineClass(name, b, 0, b.length);
+ }
+
+ //用于加载类文件
+ private byte[] loadClassData(String fileName) {
+ String classDir = dir;
+ if (!dir.endsWith(File.separator)) {
+ classDir += File.separator;
+ }
+ if (fileName.indexOf(".") != -1) {
+ fileName = fileName.replace(".", File.separator);
+ }
+ String name = classDir + fileName + ".class";
+
+ //使用输入流读取类文件
+ InputStream in = null;
+ //使用byteArrayOutputStream保存类文件。然后转化为byte数组
+ ByteArrayOutputStream out = null;
+ try {
+ in = new FileInputStream(new File(name));
+ out = new ByteArrayOutputStream();
+ int i = 0;
+ while ((i = in.read()) != -1) {
+ out.write(i);
+ }
+
+ } catch (Exception e) {
+ } finally {
+ try {
+ out.close();
+ in.close();
+ } catch (IOException e) {
+ e.printStackTrace();
+ }
+
+ }
+
+ return out.toByteArray();
+
+ }
+
+ public static void main(String[] args) throws ClassNotFoundException {
+ FileClassLoader fileClassLoader = new FileClassLoader("/tmp", FileClassLoader.class.getClassLoader());
+ Class clazz = fileClassLoader.loadClass("com.aliyun.yundun.dipper.channel.self.UpgradeFunction");
+ System.out.println(clazz.getName());
+ }
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/classloader/IsolationClassLoader.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/classloader/IsolationClassLoader.java
new file mode 100644
index 0000000..cd86623
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/classloader/IsolationClassLoader.java
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.classloader;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.net.URLClassLoader;
+
+/**
+ * 加载指定目录的class
+ */
+public class IsolationClassLoader extends URLClassLoader {
+
+ public IsolationClassLoader(String libDir, ClassLoader parent) {
+ super(new URL[] {}, parent);
+ tryLoadJarInDir(libDir);
+ }
+
+ public IsolationClassLoader(String libDir) {
+ super(new URL[] {}, null);
+ tryLoadJarInDir(libDir);
+ }
+
+ @Override
+ public Class<?> loadClass(String name) throws ClassNotFoundException {
+ return super.loadClass(name);
+ }
+
+ @Override
+ protected Class<?> findClass(String name) throws ClassNotFoundException {
+ try {
+ return super.findClass(name);
+ } catch (ClassNotFoundException e) {
+ return IsolationClassLoader.class.getClassLoader().loadClass(name);
+ }
+ }
+
+ private void tryLoadJarInDir(String dirPath) {
+ File dir = new File(dirPath);
+ // 自动加载目录下的jar包
+ if (dir.exists() && dir.isDirectory()) {
+ for (File file : dir.listFiles()) {
+ if (file.isFile() && file.getName().endsWith(".jar") && file.getName().startsWith("operator")) {
+ this.addURL(file);
+ continue;
+ }
+ }
+ } else if (dir.exists() && dir.getName().endsWith(".jar") && dir.getName().startsWith("operator")) {
+ this.addURL(dir);
+ }
+ }
+
+ private void addURL(File file) {
+ try {
+ super.addURL(new URL("file", null, file.getCanonicalPath()));
+ } catch (MalformedURLException e) {
+ e.printStackTrace();
+ } catch (IOException e) {
+ e.printStackTrace();
+ }
+ }
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/compiler/CustomJavaCompiler.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/compiler/CustomJavaCompiler.java
new file mode 100644
index 0000000..d8b277e
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/compiler/CustomJavaCompiler.java
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.compiler;
+
+import org.apache.rocketmq.streams.common.utils.FileUtil;
+
+import javax.tools.*;
+import java.io.ByteArrayOutputStream;
+import java.io.File;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * 编译java源码
+ */
+public class CustomJavaCompiler {
+ //类全名
+ private String fullClassName;
+ private String sourceCode;
+ //存放编译之后的字节码(key:类全名,value:编译之后输出的字节码)
+ private Map<String, ByteJavaFileObject> javaFileObjectMap = new ConcurrentHashMap<>();
+ //获取java的编译器
+ private JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
+ //存放编译过程中输出的信息
+ private DiagnosticCollector<JavaFileObject> diagnosticsCollector = new DiagnosticCollector<>();
+ //执行结果（控制台输出的内容）
+ private String runResult;
+ //编译耗时(单位ms)
+ private long compilerTakeTime;
+ //运行耗时(单位ms)
+ private long runTakeTime;
+
+ /**
+ * @param sourceFile java 文件
+ */
+ public CustomJavaCompiler(File sourceFile) {
+ this(FileUtil.loadFileContent(sourceFile.getAbsolutePath()));
+ }
+
+ /**
+ * @param sourceCode java 源码的字符串形式
+ */
+ public CustomJavaCompiler(String sourceCode) {
+ this.sourceCode = sourceCode;
+ this.fullClassName = getFullClassName(sourceCode);
+ }
+
+ public <T> T compileAndNewInstance() {
+ try {
+ Class aClass = compileClass();
+ return (T)aClass.newInstance();
+ } catch (Exception e) {
+ throw new RuntimeException("compile class error, the class name is " + fullClassName, e);
+ }
+ }
+
+ public Class compileClass() {
+ compiler();
+ StringClassLoader scl = new StringClassLoader();
+ Class<?> aClass = null;
+ try {
+ aClass = scl.findClass(fullClassName);
+ if (aClass == null) {
+ return null;
+ }
+ return aClass;
+ } catch (Exception e) {
+ throw new RuntimeException("compile class error, the class name is " + fullClassName, e);
+ }
+ }
+
+ /**
+ * 编译字符串源代码,编译失败在 diagnosticsCollector 中获取提示信息
+ *
+ * @return true:编译成功 false:编译失败
+ */
+ protected boolean compiler() {
+ long startTime = System.currentTimeMillis();
+ //标准的内容管理器,更换成自己的实现，覆盖部分方法
+ StandardJavaFileManager standardFileManager = compiler.getStandardFileManager(diagnosticsCollector, null, null);
+ JavaFileManager javaFileManager = new StringJavaFileManage(standardFileManager);
+ //构造源代码对象
+ JavaFileObject javaFileObject = new StringJavaFileObject(fullClassName, sourceCode);
+ //获取一个编译任务
+ JavaCompiler.CompilationTask task = compiler.getTask(null, javaFileManager, diagnosticsCollector, null, null, Arrays
+ .asList(javaFileObject));
+ //设置编译耗时
+ compilerTakeTime = System.currentTimeMillis() - startTime;
+ return task.call();
+ }
+
+ /**
+ * @return 编译信息(错误 警告)
+ */
+ public String getCompilerMessage() {
+ StringBuilder sb = new StringBuilder();
+ List<Diagnostic<? extends JavaFileObject>> diagnostics = diagnosticsCollector.getDiagnostics();
+ for (Diagnostic diagnostic : diagnostics) {
+ sb.append(diagnostic.toString()).append("\r\n");
+ }
+ return sb.toString();
+ }
+
+ /**
+ * @return 控制台打印的信息
+ */
+ public String getRunResult() {
+ return runResult;
+ }
+
+ public long getCompilerTakeTime() {
+ return compilerTakeTime;
+ }
+
+ public long getRunTakeTime() {
+ return runTakeTime;
+ }
+
+ /**
+ * 获取类的全名称
+ *
+ * @param sourceCode 源码
+ * @return 类的全名称
+ */
+ public static String getFullClassName(String sourceCode) {
+ String className = "";
+ Pattern pattern = Pattern.compile("package\\s+\\S+\\s*;");
+ Matcher matcher = pattern.matcher(sourceCode);
+ if (matcher.find()) {
+ className = matcher.group().replaceFirst("package", "").replace(";", "").trim() + ".";
+ }
+
+ pattern = Pattern.compile("class\\s+\\S+.*\\{");
+ matcher = pattern.matcher(sourceCode);
+ if (matcher.find()) {
+ className += matcher.group().replaceFirst("class", "").replace("{", "").trim();
+ }
+ int index = className.indexOf(" ");
+ if (index != -1) {
+ //如果类有实现接口或继承等，只取类名部分
+ className = className.substring(0, index);
+ }
+ return className;
+ }
+
+ /**
+ * 自定义一个字符串的源码对象
+ */
+ private class StringJavaFileObject extends SimpleJavaFileObject {
+ //等待编译的源码字段
+ private String contents;
+
+ //java源代码 => StringJavaFileObject对象 的时候使用
+ public StringJavaFileObject(String className, String contents) {
+ super(URI.create("string:///" + className.replaceAll("\\.", "/") + Kind.SOURCE.extension), Kind.SOURCE);
+ this.contents = contents;
+ }
+
+ //字符串源码会调用该方法
+ @Override
+ public CharSequence getCharContent(boolean ignoreEncodingErrors) throws IOException {
+ return contents;
+ }
+
+ }
+
+ /**
+ * 自定义一个编译之后的字节码对象
+ */
+ private class ByteJavaFileObject extends SimpleJavaFileObject {
+ //存放编译后的字节码
+ private ByteArrayOutputStream outPutStream;
+
+ public ByteJavaFileObject(String className, Kind kind) {
+ super(URI.create("string:///" + className.replaceAll("\\.", "/") + Kind.SOURCE.extension), kind);
+ }
+
+ //StringJavaFileManage 编译之后的字节码输出会调用该方法（把字节码输出到outputStream）
+ @Override
+ public OutputStream openOutputStream() {
+ outPutStream = new ByteArrayOutputStream();
+ return outPutStream;
+ }
+
+ //在类加载器加载的时候需要用到
+ public byte[] getCompiledBytes() {
+ return outPutStream.toByteArray();
+ }
+ }
+
+ /**
+ * 自定义一个JavaFileManage来控制编译之后字节码的输出位置
+ */
+ private class StringJavaFileManage extends ForwardingJavaFileManager {
+ StringJavaFileManage(JavaFileManager fileManager) {
+ super(fileManager);
+ }
+
+ //获取输出的文件对象，它表示给定位置处指定类型的指定类。
+ @Override
+ public JavaFileObject getJavaFileForOutput(Location location, String className, JavaFileObject.Kind kind, FileObject sibling) throws IOException {
+ ByteJavaFileObject javaFileObject = new ByteJavaFileObject(className, kind);
+ javaFileObjectMap.put(className, javaFileObject);
+ return javaFileObject;
+ }
+ }
+
+ /**
+ * 自定义类加载器, 用来加载动态的字节码
+ */
+ private class StringClassLoader extends ClassLoader {
+ @Override
+ protected Class<?> findClass(String name) throws ClassNotFoundException {
+ ByteJavaFileObject fileObject = javaFileObjectMap.get(name);
+ if (fileObject != null) {
+ byte[] bytes = fileObject.getCompiledBytes();
+ return defineClass(name, bytes, 0, bytes.length);
+ }
+ try {
+ return ClassLoader.getSystemClassLoader().loadClass(name);
+ } catch (Exception e) {
+ return super.findClass(name);
+ }
+ }
+ }
+
+}
\ No newline at end of file
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/component/AbstractComponent.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/component/AbstractComponent.java
new file mode 100644
index 0000000..e0aa35d
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/component/AbstractComponent.java
@@ -0,0 +1,208 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.component;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.rocketmq.streams.common.configure.ConfigureFileKey;
+import org.apache.rocketmq.streams.common.utils.ENVUtile;
+import org.apache.rocketmq.streams.common.utils.PropertiesUtils;
+
+import java.util.Iterator;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+public abstract class AbstractComponent<T> implements IComponent<T>, ConfigureFileKey {
+
+ private static final Log LOG = LogFactory.getLog(AbstractComponent.class);
+
+ public final static String ENV_JDBC_URL = "rocketmq_streams_sync_jdbc_url";
+
+ public final static String ENV_JDBC_USERNAME = "rocketmq_streams_sync_jdbc_username";
+
+ public final static String ENV_JDBC_PASSWORD = "rocketmq_streams_sync_jdbc_password";
+
+ public final static String ENV_JDBC_DRIVER = "rocketmq_streams_sync_jdbc_driver";
+
+ public final static String HTTP_AK = "rocketmq.streams.channel.ak";
+
+ public final static String HTTP_SK = "rocketmq.streams.channel.sk";
+
+ public final static String DEFAULT_JDBC_DRIVER = "com.mysql.jdbc.Driver";
+
+ public final static String DEFAULT_JDBC_TABLE_NAME = "rocketmq_streams_checkpoint_table";
+
+ /**
+ * xml的位置，如果没有即默认位置
+ */
+ protected PropertyConfigureDescriptorManager configureDiscriptorManager = new PropertyConfigureDescriptorManager();
+
+ private Properties properties;
+
+ @Override
+ public boolean init() {
+ Properties properties = createDefaultProperty();
+ return initProperty(properties);
+ }
+
+ protected Properties createDefaultProperty() {
+ //createENVProperties();
+ Properties properties = null;
+ properties = ComponentCreator.getProperties();
+ if (properties == null) {
+ properties = getDefaultProperties();
+ }
+ if (properties == null) {
+ properties = new Properties();
+ }
+ addSystemProperties(properties);
+ return properties;
+ }
+
+ public void initConfigurableServiceDescriptor() {
+ addConfigureDescriptor(new ConfigureDescriptor("jdbc", JDBC_URL, null, true, ENV_JDBC_URL));
+ addConfigureDescriptor(new ConfigureDescriptor("jdbc", JDBC_USERNAME, null, true, ENV_JDBC_USERNAME));
+ addConfigureDescriptor(new ConfigureDescriptor("jdbc", JDBC_PASSWORD, null, true, ENV_JDBC_PASSWORD));
+ addConfigureDescriptor(
+ new ConfigureDescriptor("jdbc", JDBC_DRIVER, DEFAULT_JDBC_DRIVER, false, ENV_JDBC_DRIVER));
+ addConfigureDescriptor(new ConfigureDescriptor("http", HTTP_AK, true));
+ addConfigureDescriptor(new ConfigureDescriptor("http", HTTP_SK, true));
+ }
+
+ protected void addConfigureDescriptor(ConfigureDescriptor configureDiscriptor) {
+ configureDiscriptorManager.addConfigureDescriptor(configureDiscriptor);
+ }
+
+ /**
+ * 如果系统属性中有对应的属性配置，覆盖掉环境变量和文件的属性，主要用于测试，不推荐用在生产环境
+ *
+ * @param properties
+ */
+ protected void addSystemProperties(Properties properties) {
+ for (List<ConfigureDescriptor> configureDescriptors : configureDiscriptorManager.getGroupByConfigures().values()) {
+ for (ConfigureDescriptor configureDiscriptor : configureDescriptors) {
+ String key = configureDiscriptor.getPropertyKey();
+ String value = ENVUtile.getSystemParameter(key);
+ if (value != null) {
+ properties.put(key, value);
+ }
+ }
+ }
+ }
+
+ /**
+ * 根据配置文件的配置情况，进行环境变量检查，如果所有的必须参数都在环境变量中，则创建属性文件。支持多组属性，如configure，既支持http配置，也支持db配置
+ *
+ * @return
+ */
+ protected Properties createENVProperties() {
+ if (configureDiscriptorManager.getGroupByConfigures() == null) {
+ return null;
+ }
+ Iterator<List<ConfigureDescriptor>> it = configureDiscriptorManager.getGroupByConfigures().values().iterator();
+ Properties properties = new Properties();
+ boolean hasProperties = false;
+ while (it.hasNext()) {
+ List<ConfigureDescriptor> configureDiscriptors = it.next();
+ Properties p = configureDiscriptorManager.createENVProperties(configureDiscriptors);
+ if (p != null) {
+ properties.putAll(p);
+ hasProperties = true;
+ }
+ }
+ if (hasProperties) {
+ return properties;
+ }
+ return null;
+ }
+
+ @Override
+ public boolean initByClassPath(String propertiesPath) {
+ Properties properties = PropertiesUtils.getResourceProperties(propertiesPath);
+ return initProperty(properties);
+ }
+
+ @Override
+ public boolean initByFilePath(String filePath) {
+ Properties properties = PropertiesUtils.loadPropertyByFilePath(filePath);
+ return initProperty(properties);
+ }
+
+ @Override
+ public boolean initByPropertiesStr(String... kvs) {
+ Properties properties = createDefaultProperty();
+ if (kvs != null && kvs.length > 0) {
+ for (String ky : kvs) {
+ PropertiesUtils.putProperty(ky, ":", properties);
+ }
+ }
+ return initProperty(properties);
+ }
+
+ /**
+ * 根据属性文件init component
+ *
+ * @param properties
+ * @return
+ */
+ protected boolean initProperty(Properties properties) {
+ boolean success = initProperties(properties);
+ this.properties = properties;
+ return success;
+ }
+
+ protected Properties getDefaultProperties() {
+ try {
+ return PropertiesUtils.getResourceProperties(getComponentPropertyPath());
+ } catch (Exception e) {
+ LOG.error("load jar file error", e);
+ return null;
+ }
+
+ }
+
+ private String getComponentPropertyPath() {
+ return PropertiesUtils.getComponentPropertyPath(this.getClass());
+ }
+
+ protected AtomicBoolean isStart = new AtomicBoolean(false);
+
+ @Override
+ public boolean start(String name) {
+ if (isStart.compareAndSet(false, true)) {
+ startComponent(name);
+ }
+ return true;
+ }
+
+ protected abstract boolean startComponent(String name);
+
+ protected void finishStart() {
+ isStart.set(true);
+ }
+
+ public Properties getProperties() {
+ return properties;
+ }
+
+ protected abstract boolean initProperties(Properties properties);
+
+ public boolean directInitProperties(Properties properties) {
+ return initProperties(properties);
+ }
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/component/ComponentCreator.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/component/ComponentCreator.java
new file mode 100644
index 0000000..de1f5df
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/component/ComponentCreator.java
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.component;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.rocketmq.streams.common.configurable.AbstractConfigurable;
+import org.apache.rocketmq.streams.common.configurable.IConfigurableService;
+import org.apache.rocketmq.streams.common.utils.MapKeyUtil;
+import org.apache.rocketmq.streams.common.utils.PropertiesUtils;
+import org.apache.rocketmq.streams.common.utils.SQLUtil;
+
+import java.net.URL;
+import java.util.*;
+import java.util.Map.Entry;
+
+/**
+ * 创建组件，如果参数未发生变化（如果未传入，则是配置文件的参数），返回同一个组件对象，如果发生变化，返回不同的组件对象
+ */
+@SuppressWarnings({"rawtypes", "unchecked"})
+public class ComponentCreator {
+
+ private static final Log LOG = LogFactory.getLog(ComponentCreator.class);
+
+ /**
+ * 代理dbchannel的class，需要继承JDBCDataSource抽象类。如果配置这个参数，则会给dbchannel增加一层代理，所有需要db访问的，都是通过open api发送sql给代理
+ */
+ public static final String DB_PROXY_CLASS_NAME = "db_proxy_class_name";
+
+ /**
+ * 创建channel的服务
+ */
+ public static final String DIPPER_INSTANCE_CHANNEL_CREATOR_SERVICE_NAME
+ = "dipper_instance_channel_creator_service_name";
+
+ /**
+ * blink jar包所在的路径
+ */
+ public static final String BLINK_UDF_JAR_PATH = "dipper.blink.udf.jar.path";
+ private static final Map<String, IComponent> key2Component = new HashMap<>();
+ private static Properties properties;
+ public static String propertiesPath;//属性文件位置，便于定期刷新
+
+ static {
+ Properties properties1 = PropertiesUtils.getResourceProperties("dipper.properties");
+ if (properties1 == null) {
+ ComponentCreator.createMemoryProperties(10000L);
+ } else {
+ ComponentCreator.setProperties(properties1);
+ }
+ }
+
+ public static String getDBProxyClassName() {
+ return properties.getProperty(DB_PROXY_CLASS_NAME);
+ }
+
+ public static void setProperties(String propertiesPath) {
+ ComponentCreator.propertiesPath = propertiesPath;
+ createProperties(propertiesPath);
+ }
+
+ public static void setProperties(Properties properties) {
+ ComponentCreator.properties = properties;
+ }
+
+ public static String[] createKV(Properties properties) {
+ List<String> keys = new ArrayList<>();
+ Iterator<Object> keyIterator = properties.keySet().iterator();
+ while (keyIterator.hasNext()) {
+ keys.add(keyIterator.next().toString());
+ }
+ Collections.sort(keys);
+ Iterator<String> it = keys.iterator();
+ String[] kvs = new String[properties.size()];
+ int i = 0;
+ while (it.hasNext()) {
+ String key = it.next();
+ String value = properties.getProperty(key);
+ kvs[i] = key + ":" + value;
+ i++;
+ }
+ return kvs;
+ }
+
+ public static void createProperties(Properties properties) {
+ ComponentCreator.properties = loadOtherProperty(properties);
+ }
+
+ public static void createDBProperties(String url, String userName, String password, String tableName, String driverClass) {
+ Properties properties = new Properties();
+ properties.put(AbstractComponent.JDBC_DRIVER, driverClass);
+ properties.put(AbstractComponent.JDBC_URL, url);
+ properties.put(AbstractComponent.JDBC_USERNAME, userName);
+ properties.put(AbstractComponent.JDBC_PASSWORD, password);
+ properties.put(AbstractComponent.JDBC_TABLE_NAME, tableName);
+ properties.put(AbstractComponent.CONNECT_TYPE, IConfigurableService.DEFAULT_SERVICE_NAME);
+ ComponentCreator.properties = loadOtherProperty(properties);
+ }
+
+ public static void createDBProperties(String url, String userName, String password, String tableName, Long pollingTime, String... kvs) {
+ Properties properties = new Properties();
+ properties.put(AbstractComponent.JDBC_DRIVER, AbstractComponent.DEFAULT_JDBC_DRIVER);
+ properties.put(AbstractComponent.JDBC_URL, url);
+ properties.put(AbstractComponent.JDBC_USERNAME, userName);
+ properties.put(AbstractComponent.JDBC_PASSWORD, password);
+ properties.put(AbstractComponent.JDBC_TABLE_NAME, tableName);
+ properties.put(AbstractComponent.POLLING_TIME, pollingTime + "");
+ properties.put(AbstractComponent.CONNECT_TYPE, IConfigurableService.DEFAULT_SERVICE_NAME);
+ ComponentCreator.properties = loadOtherProperty(properties, kvs);
+ }
+
+ public static Properties createFileProperties(String filePathName, Long pollingTime, String... kvs) {
+ Properties properties = new Properties();
+ properties.put(AbstractComponent.CONNECT_TYPE, IConfigurableService.FILE_SERVICE_NAME);
+ properties.put(IConfigurableService.FILE_PATH_NAME, filePathName);
+ properties.put(AbstractComponent.POLLING_TIME, pollingTime + "");
+ Properties properties1 = loadOtherProperty(properties, kvs);
+ ComponentCreator.properties = properties1;
+ return properties1;
+ }
+
+ public static void createMemoryProperties(Long pollingTime, String... kvs) {
+ Properties properties = new Properties();
+ properties.put(AbstractComponent.CONNECT_TYPE, IConfigurableService.MEMORY_SERVICE_NAME);
+ properties.put(AbstractComponent.POLLING_TIME, pollingTime + "");
+ ComponentCreator.properties = loadOtherProperty(properties, kvs);
+ }
+
+ public static void createProperties(String propertiesFilePath, String... kvs) {
+ Properties properties = PropertiesUtils.getResourceProperties(propertiesFilePath);
+ if (properties == null) {
+ properties = PropertiesUtils.loadPropertyByFilePath(propertiesFilePath);
+ }
+ ComponentCreator.properties = loadOtherProperty(properties, kvs);
+ }
+
+ private static Properties loadOtherProperty(Properties tmp, String... kvs) {
+ Properties properties = new Properties();
+ for (Entry<Object, Object> entry : tmp.entrySet()) {
+ String key = (String)entry.getKey();
+ String value = (String)entry.getValue();
+ String realValue = value;
+ if (value.contains("#{")) {
+ realValue = SQLUtil.parseIbatisSQL(tmp, value, true);
+ if (realValue != null && realValue.startsWith("'")) {
+ realValue = realValue.replace("'", "");
+ }
+ }
+
+ properties.put(key, realValue);
+ }
+ if (kvs == null || kvs.length == 0) {
+ return properties;
+ }
+ for (String kv : kvs) {
+ int startIndex = kv.indexOf(":");
+ String key = kv.substring(0, startIndex);
+ String value = kv.substring(startIndex + 1);
+ properties.put(key, value);
+
+ }
+ return properties;
+ }
+
+ public static <T extends IComponent> T getComponent(String namespace, Class componentType, String... kvs) {
+ return getComponent(namespace, componentType, true, kvs);
+ }
+
+ protected static <T extends IComponent> T getComponent(String namespace, Class componentType, boolean isStart,
+ String... kvs) {
+ Properties properties = loadOtherProperty(ComponentCreator.properties, kvs);
+ String[] kvArray = createKV(properties);
+ return (T)getComponentInner(namespace, componentType, isStart, kvArray);
+ }
+
+ public static <T extends IComponent> T getComponent(String namespace, Class componentType) {
+ return (T)getComponent(namespace, componentType, true, createKV(properties));
+ }
+
+ public static <T extends IComponent> T getComponentNotStart(String namespace, Class componentType) {
+ return (T)getComponentInner(namespace, componentType, false, createKV(properties));
+ }
+
+ public static <T extends IComponent> T getComponentNotStart(String namespace, Class componentType, String... kvs) {
+ return getComponent(namespace, componentType, false, kvs);
+ }
+
+ @Deprecated
+ public static <T extends IComponent> T getComponentUsingPropertiesFile(String namespace, Class componentType,
+ String propertiesPath) {
+ return (T)getComponentInner(namespace, componentType, true, propertiesPath);
+ }
+
+ private static IComponent getComponentInner(String namespace, Class<IComponent> componentType, boolean needStart, Object o) {
+ String key = createKey(componentType, namespace, o);
+ if (key2Component.containsKey(key) && key2Component.get(key) != null) {
+ return key2Component.get(key);
+ }
+ synchronized (ComponentCreator.class) {
+ if (key2Component.containsKey(key) && key2Component.get(key) != null) {
+ return key2Component.get(key);
+ }
+ try {
+ IComponent component = componentType.newInstance();
+ initComponent(component, o);
+ key2Component.put(key, component);
+ if (needStart) {
+ component.start(namespace);
+ }
+
+ return component;
+ } catch (Exception e) {
+ LOG.error("can not get component. namespace is " + namespace + ", type is " + componentType.getName(),
+ e);
+ throw new RuntimeException(e.getMessage(), e);
+ }
+ }
+
+ }
+
+ private static String createKey(Class<IComponent> componentType, String namespace, Object o) {
+ String key = MapKeyUtil.createKey(componentType.getName(), namespace);
+ if (o == null) {
+ return key;
+ } else if (o instanceof String) {
+ String propertiesPath = (String)o;
+ return MapKeyUtil.createKey(key, propertiesPath);
+ } else if (o.getClass().isArray()) {
+ String[] properties = (String[])o;
+ String pk = MapKeyUtil.createKeyBySign("_", properties);
+ return MapKeyUtil.createKey(key, pk);
+ }
+ return key;
+ }
+
+ /**
+ * 初始化组件，支持多种初始化方法
+ *
+ * @param component
+ * @param o
+ */
+ private static void initComponent(IComponent component, Object o) {
+ if (o == null) {
+ component.init();
+ } else if (o instanceof String) {
+ String propertiesPath = (String)o;
+ URL url = PropertiesUtils.class.getClassLoader().getResource(propertiesPath);
+ if (url != null) {
+ component.initByClassPath(propertiesPath);
+ } else {
+ component.initByFilePath(propertiesPath);
+ }
+ } else if (o.getClass().isArray()) {
+ component.initByPropertiesStr((String[])o);
+ }
+ return;
+ }
+
+
+ public static void createProperty(String outputFilePath) {
+ Properties properties = ComponentCreator.getProperties();
+ PropertiesUtils.flush(properties, outputFilePath);
+ }
+
+ public static Properties getProperties() {
+ return properties;
+ }
+
+ public static boolean getPropertyBooleanValue(String key) {
+ String value = properties.getProperty(key);
+ if (value == null) {
+ return false;
+ }
+ if ("true".equals(value)) {
+ return true;
+ }
+ return false;
+ }
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/component/ConfigureDescriptor.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/component/ConfigureDescriptor.java
new file mode 100644
index 0000000..ab5a336
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/component/ConfigureDescriptor.java
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.component;
+
+public class ConfigureDescriptor {
+
+ private String propertyKey;
+ private String defaultValue;
+ /**
+ * 这个值是否必须用户设置
+ */
+ private boolean requiredSet = false;
+ private String groupName;
+ /**
+ * 环境变量对应的可以
+ */
+ private String envPropertyKey;
+
+ public ConfigureDescriptor(String groupName, String propertyKey, String defaultValue, boolean requiredSet,
+ String envPropertyKey) {
+ this(groupName, propertyKey, defaultValue, requiredSet);
+ this.envPropertyKey = envPropertyKey;
+ }
+
+ public ConfigureDescriptor(String groupName, String propertyKey, String defaultValue, boolean requiredSet) {
+ this.groupName = groupName;
+ this.propertyKey = propertyKey;
+ this.defaultValue = defaultValue;
+ this.requiredSet = requiredSet;
+ this.envPropertyKey = propertyKey;
+ }
+
+ public ConfigureDescriptor(String groupName, String propertyKey, boolean requiredSet) {
+ this(groupName, propertyKey, null, requiredSet);
+ }
+
+ public ConfigureDescriptor(String propertyKey, boolean requiredSet) {
+ this(null, propertyKey, requiredSet);
+ }
+
+ public ConfigureDescriptor(String propertyKey, boolean requiredSet, String defaultValue) {
+ this(null, propertyKey, defaultValue, requiredSet);
+ }
+
+ public String getPropertyKey() {
+ return propertyKey;
+ }
+
+ public void setPropertyKey(String propertyKey) {
+ this.propertyKey = propertyKey;
+ }
+
+ public String getDefaultValue() {
+ return defaultValue;
+ }
+
+ public void setDefaultValue(String defaultValue) {
+ this.defaultValue = defaultValue;
+ }
+
+ public boolean isRequiredSet() {
+ return requiredSet;
+ }
+
+ public void setRequiredSet(boolean requiredSet) {
+ this.requiredSet = requiredSet;
+ }
+
+ public String getGroupName() {
+ return groupName;
+ }
+
+ public void setGroupName(String groupName) {
+ this.groupName = groupName;
+ }
+
+ public String getEnvPropertyKey() {
+ return envPropertyKey;
+ }
+
+ public void setEnvPropertyKey(String envPropertyKey) {
+ this.envPropertyKey = envPropertyKey;
+ }
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/component/IComponent.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/component/IComponent.java
new file mode 100644
index 0000000..1402f26
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/component/IComponent.java
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.component;
+
+/**
+ * 对组件的封装，隐藏组件的实现细节，组件的使用更简单
+ */
+public interface IComponent<T> {
+
+ /**
+ * 通过默认的配置加载组件，默认配置存放的路径为component／组件名称.default.properties
+ *
+ * @return
+ */
+ boolean init();
+
+ /**
+ * 启动组建，用到的服务需要进行初始化，默认逻辑是谁创建，谁初始化
+ *
+ * @return
+ */
+ boolean start(String namespace);
+
+ /**
+ * 关闭组建
+ *
+ * @return
+ */
+ boolean stop();
+
+ /**
+ * 获取组件对应的服务
+ *
+ * @return
+ */
+ T getService();
+
+ /**
+ * 覆盖init()属性中的内容，形式为key:value,可以多组
+ *
+ * @param kvs
+ * @return
+ */
+ @Deprecated
+ boolean initByPropertiesStr(String... kvs);
+
+ /**
+ * 加载文件初始化组件，如果是spring文件，同factory,如果是属性文件，同init，会对新创建的服务进行初始化。
+ *
+ * @param classPath
+ * @return
+ */
+ @Deprecated
+ boolean initByClassPath(String classPath);
+
+ @Deprecated
+ boolean initByFilePath(String filePath);
+
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/component/IgnoreNameSpace.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/component/IgnoreNameSpace.java
new file mode 100644
index 0000000..472aa0b
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/component/IgnoreNameSpace.java
@@ -0,0 +1,24 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.component;
+
+/**
+ * 表明此组件，不需要namespace
+ */
+public interface IgnoreNameSpace {
+
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/component/PropertyConfigureDescriptorManager.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/component/PropertyConfigureDescriptorManager.java
new file mode 100644
index 0000000..873f25e
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/component/PropertyConfigureDescriptorManager.java
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.component;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.rocketmq.streams.common.utils.ENVUtile;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+
+import java.util.*;
+
+public class PropertyConfigureDescriptorManager {
+
+ private static final Log LOG = LogFactory.getLog(PropertyConfigureDescriptorManager.class);
+
+ protected transient Map<String, List<ConfigureDescriptor>> groupByConfigures = new HashMap<>();
+
+ /**
+ * 多组配置文件，比如http是一组，db是一组。需要子类在启动时在数据中写如配置描述
+ */
+ protected transient List<ConfigureDescriptor> defaultGroupConfigureDescriptors = new ArrayList<>();
+
+ /**
+ * 增加一个属性配置
+ *
+ * @param configureDescriptor
+ */
+ public void addConfigureDescriptor(ConfigureDescriptor configureDescriptor) {
+ if (configureDescriptor == null) {
+ return;
+ }
+ String groupName = configureDescriptor.getGroupName();
+ if (StringUtil.isEmpty(groupName)) {
+ defaultGroupConfigureDescriptors.add(configureDescriptor);
+ return;
+ }
+ List<ConfigureDescriptor> configureDescriptors = groupByConfigures.get(groupName);
+ if (configureDescriptors == null) {
+ synchronized (this) {
+ configureDescriptors = groupByConfigures.computeIfAbsent(groupName, k -> new ArrayList<>());
+ }
+ }
+ configureDescriptors.add(configureDescriptor);
+ }
+
+ /**
+ * 对于一组配置进行检查，如果所有的必须参数都在环境变量中，则创建属性文件
+ *
+ * @param configureDescriptorList
+ * @return
+ */
+ public Properties createENVProperties(List<ConfigureDescriptor> configureDescriptorList) {
+ if (configureDescriptorList == null || configureDescriptorList.size() == 0) {
+ return null;
+ }
+ Properties properties = new Properties();
+ for (ConfigureDescriptor configureDescriptor : configureDescriptorList) {
+ String key = configureDescriptor.getEnvPropertyKey();
+ String value = ENVUtile.getENVParameter(key);
+ if (configureDescriptor.isRequiredSet() && value == null) {
+ return null;
+ }
+ if (value != null) {
+ properties.put(configureDescriptor.getPropertyKey(), value);
+ } else {
+ value = configureDescriptor.getDefaultValue();
+ if (StringUtil.isNotEmpty(value)) {
+ properties.put(configureDescriptor.getPropertyKey(), configureDescriptor.getDefaultValue());
+ }
+ }
+ }
+ LOG.info("env properties:" + properties.entrySet());
+ return properties;
+ }
+
+ public Map<String, List<ConfigureDescriptor>> getGroupByConfigures() {
+ Map<String, List<ConfigureDescriptor>> map = new HashMap<>();
+ Iterator<Map.Entry<String, List<ConfigureDescriptor>>> it = groupByConfigures.entrySet().iterator();
+ while (it.hasNext()) {
+ Map.Entry<String, List<ConfigureDescriptor>> entry = it.next();
+ String groupName = entry.getKey();
+ List<ConfigureDescriptor> value = entry.getValue();
+ List<ConfigureDescriptor> newValue = new ArrayList<>();
+ newValue.addAll(value);
+ newValue.addAll(defaultGroupConfigureDescriptors);
+ map.put(groupName, newValue);
+ }
+ return map;
+ }
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/component/PropertyConfigureDiscriptorManager.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/component/PropertyConfigureDiscriptorManager.java
new file mode 100644
index 0000000..6858bd4
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/component/PropertyConfigureDiscriptorManager.java
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.streams.common.component;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.rocketmq.streams.common.utils.ENVUtile;
+import org.apache.rocketmq.streams.common.utils.StringUtil;
+
+import java.util.*;
+
+public class PropertyConfigureDiscriptorManager {
+
+ private static final Log LOG = LogFactory.getLog(PropertyConfigureDiscriptorManager.class);
+
+ protected transient Map<String, List<ConfigureDescriptor>> groupbyConfigures = new HashMap<>();
+
+ /**
+ * 多组配置文件，比如http是一组，db是一组。需要子类在启动时在数据中写如配置描述
+ */
+ protected transient List<ConfigureDescriptor> defaultGroupConfigureDiscriptors = new ArrayList<>();
+
+ /**
+ * 增加一个属性配置
+ *
+ * @param configureDiscriptor
+ */
+ public void addConfigureDiscriptor(ConfigureDescriptor configureDiscriptor) {
+ if (configureDiscriptor == null) {
+ return;
+ }
+ String groupName = configureDiscriptor.getGroupName();
+ if (StringUtil.isEmpty(groupName)) {
+ defaultGroupConfigureDiscriptors.add(configureDiscriptor);
+ return;
+ }
+ List<ConfigureDescriptor> configureDiscriptors = groupbyConfigures.get(groupName);
+ if (configureDiscriptors == null) {
+ synchronized (this) {
+ configureDiscriptors = groupbyConfigures.get(groupName);
+ if (configureDiscriptors == null) {
+ configureDiscriptors = new ArrayList<>();
+ groupbyConfigures.put(groupName, configureDiscriptors);
+ }
+ }
+ }
+ configureDiscriptors.add(configureDiscriptor);
+ }
+
+ /**
+ * 对于一组配置进行检查，如果所有的必须参数都在环境变量中，则创建属性文件
+ *
+ * @param configureDiscriptorList
+ * @return
+ */
+ public Properties createENVProperties(List<ConfigureDescriptor> configureDiscriptorList) {
+ if (configureDiscriptorList == null || configureDiscriptorList.size() == 0) {
+ return null;
+ }
+ Properties properties = new Properties();
+ for (ConfigureDescriptor configureDiscriptor : configureDiscriptorList) {
+ String key = configureDiscriptor.getEnvPropertyKey();
+ String value = ENVUtile.getENVParameter(key);
+ // LOG.info("@@@envkey:" + key + ",envValue:" + value);
+ if (configureDiscriptor.isRequiredSet() && value == null) {
+ return null;
+ }
+ if (value != null) {
+ properties.put(configureDiscriptor.getPropertyKey(), value);
+ } else {
+ value = configureDiscriptor.getDefaultValue();
+ if (StringUtil.isNotEmpty(value)) {
+ properties.put(configureDiscriptor.getPropertyKey(), configureDiscriptor.getDefaultValue());
+ }
+ }
+ }
+ LOG.info("env properties:" + properties.entrySet());
+ return properties;
+ }
+
+ public Map<String, List<ConfigureDescriptor>> getGroupbyConfigures() {
+ Map<String, List<ConfigureDescriptor>> map = new HashMap<>();
+ Iterator<Map.Entry<String, List<ConfigureDescriptor>>> it = groupbyConfigures.entrySet().iterator();
+ while (it.hasNext()) {
+ Map.Entry<String, List<ConfigureDescriptor>> entry = it.next();
+ String groupName = entry.getKey();
+ List<ConfigureDescriptor> value = entry.getValue();
+ List<ConfigureDescriptor> newValue = new ArrayList<>();
+ newValue.addAll(value);
+ newValue.addAll(defaultGroupConfigureDiscriptors);
+ map.put(groupName, newValue);
+ }
+ return map;
+ }
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/configurable/AbstractConfigurable.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/configurable/AbstractConfigurable.java
new file mode 100644
index 0000000..5800644
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/configurable/AbstractConfigurable.java
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.configurable;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.rocketmq.streams.common.configure.ConfigureFileKey;
+import org.apache.rocketmq.streams.common.configurable.annotation.Changeable;
+import org.apache.rocketmq.streams.common.model.Entity;
+import org.apache.rocketmq.streams.common.utils.AESUtil;
+
+import java.util.HashMap;
+import java.util.Map;
+
+public abstract class AbstractConfigurable extends Entity implements IConfigurable {
+
+ private transient Log LOG = LogFactory.getLog(AbstractConfigurable.class);
+
+ /**
+ * 升级中心表
+ */
+ public static final String TABLE_NAME = "dipper_configure";
+
+ @Changeable
+ protected transient Map<String, Object> privateDatas = new HashMap<>();
+
+ protected transient IConfigurableService configurableService;
+
+ protected long updateFlag = 0;//通过它来触发更新，其他字段变更都不会触发更新
+
+ /**
+ * 是否完成初始化
+ */
+ private transient volatile boolean hasInit = false;
+
+ /**
+ * 是否初始化成功
+ */
+ protected transient boolean initSuccess = true;
+
+ /**
+ * 是否已经被销毁
+ */
+ protected transient boolean isDestroy = false;
+
+ /**
+ * 数据库的状态字段
+ */
+ private static final String STATUS = "status";
+
+ @Override
+ public boolean init() {
+ boolean initConfigurable = true;
+ if (!hasInit) {
+ try {
+ privateDatas = new HashMap<>();
+ hasInit = false;
+ initSuccess = true;
+ isDestroy = false;
+ initConfigurable = initConfigurable();
+ initSuccess = initConfigurable;
+ } catch (Exception e) {
+ initSuccess = false;
+ e.printStackTrace();
+ throw new RuntimeException("init configurable error " + this.toJson(), e);
+ }
+ hasInit = true;
+ }
+ return initConfigurable;
+ }
+
+ @Override
+ public void destroy() {
+ isDestroy = true;
+ }
+
+ /**
+ * 启用configurable 对象，可以被看到和应用
+ */
+ public void open() {
+ putPrivateData(STATUS, "1");
+ }
+
+ /**
+ * 关闭configuable 对象，对象失效
+ */
+ public void close() {
+ putPrivateData(STATUS, "0");
+ }
+
+ protected boolean initConfigurable() {
+ return true;
+ }
+
+ public String createSQL() {
+ return createSQL(this, TABLE_NAME);
+ }
+
+ public static String createSQL(IConfigurable configurable) {
+ return createSQL(configurable, "dipper_configure");
+ }
+
+ public static String createSQL(IConfigurable configurable, String tableName) {
+ String json = configurable.toJson();
+ Entity entity = null;
+ if (Entity.class.isInstance(configurable)) {
+ entity = (Entity)configurable;
+ } else {
+ entity = new Entity();
+ }
+ int status = 1;
+ if (configurable.getPrivateData("status") != null) {
+ status = Integer.valueOf(configurable.getPrivateData("status"));
+ }
+ String theSecretValue;
+ try {
+ theSecretValue = AESUtil.aesEncrypt(json, ConfigureFileKey.SECRECY);
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ String sql = "insert into " + tableName
+ + "(gmt_create,gmt_modified,namespace,type,name,json_value,status)" + "values(" + "now(),now(),'" + configurable.getNameSpace() + "','"
+ + configurable.getType() + "','" + configurable.getConfigureName() + "','" + theSecretValue + "'," + status + ")"
+ + "ON DUPLICATE KEY UPDATE status=" + status + ", gmt_modified = now()" + ",json_value='" + theSecretValue + "'";
+ return sql;
+ }
+
+ public void update() {
+ if (configurableService != null) {
+ configurableService.update(this);
+ } else {
+ LOG.warn("can not support configurable update configurable service is null");
+ }
+ }
+
+ protected String getDipperConfigureTableName() {
+ return TABLE_NAME;
+ }
+
+ public <T extends IConfigurableService> T getConfigurableService() {
+ return (T)configurableService;
+ }
+
+ public void setConfigurableService(IConfigurableService configurableService) {
+ this.configurableService = configurableService;
+ }
+
+ @Override
+ public <T> void putPrivateData(String key, T value) {
+ this.privateDatas.put(key, value);
+ }
+
+ @Override
+ public <T> T getPrivateData(String key) {
+ return (T)this.privateDatas.get(key);
+ }
+
+ public <T> T removePrivateData(String key) {
+ return (T)this.privateDatas.remove(key);
+ }
+
+ @Override
+ public Map<String, Object> getPrivateData() {
+ return this.privateDatas;
+ }
+
+ public Map<String, Object> getPrivateDatas() {
+ return privateDatas;
+ }
+
+ public void setPrivateDatas(Map<String, Object> privateDatas) {
+ this.privateDatas = privateDatas;
+ }
+
+ public boolean isInitSuccess() {
+ return initSuccess;
+ }
+
+ public void setInitSuccess(boolean initSuccess) {
+ this.initSuccess = initSuccess;
+ }
+
+ public boolean isDestroy() {
+ return isDestroy;
+ }
+
+ public boolean isHasInit() {
+ return hasInit;
+ }
+
+ public long getUpdateFlag() {
+ return updateFlag;
+ }
+
+ public void setHasInit(boolean hasInit) {
+ this.hasInit = hasInit;
+ }
+
+ public void setUpdateFlag(long updateFlag) {
+ this.updateFlag = updateFlag;
+ }
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/configurable/BasedConfigurable.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/configurable/BasedConfigurable.java
new file mode 100644
index 0000000..2865fa2
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/configurable/BasedConfigurable.java
@@ -0,0 +1,249 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.configurable;
+
+import com.alibaba.fastjson.JSON;
+import com.alibaba.fastjson.JSONObject;
+import org.apache.rocketmq.streams.common.component.ComponentCreator;
+import org.apache.rocketmq.streams.common.configurable.annotation.ENVDependence;
+import org.apache.rocketmq.streams.common.configurable.annotation.NoSerialized;
+import org.apache.rocketmq.streams.common.datatype.DataType;
+import org.apache.rocketmq.streams.common.utils.*;
+
+import java.lang.reflect.Field;
+import java.lang.reflect.Modifier;
+
+/**
+ * 这个类自动完成成员变量的序列化，反序列化，以及环境变量的替换 子类只要按pojo实现即可。 有几个要求： 1.需要序列化的类，必须实现getset方法，这块下个版本会优化，去掉这个限制 2.不需要序列化的字段必须加transient 关键字声明 3.成员变量是 DataType支持的类型
+ */
+public class BasedConfigurable extends AbstractConfigurable {
+
+ /**
+ * 扩展字段
+ */
+ public static final String EXTEND_FIELD_NAME = "extendField";
+
+ private String nameSpace;
+
+ private String configureName;
+
+ private String type;
+
+ protected String version = "1.0";
+
+ @Override
+ public String getNameSpace() {
+ return nameSpace;
+ }
+
+ @Override
+ public void setNameSpace(String nameSpace) {
+ this.nameSpace = nameSpace;
+ }
+
+ @Override
+ public String getConfigureName() {
+ return configureName;
+ }
+
+ @Override
+ public void setConfigureName(String configureName) {
+ this.configureName = configureName;
+ }
+
+ @Override
+ public String getType() {
+ return type;
+ }
+
+ @Override
+ public void setType(String type) {
+ this.type = type;
+ }
+
+ public JSONObject toJsonObject() {
+ JSONObject jsonObject = new JSONObject();
+ setJsonObject(jsonObject);
+ jsonObject.put(IConfigurableService.CLASS_NAME, this.getClass().getName());
+ return jsonObject;
+ }
+
+ @Override
+ public String toJson() {
+ JSONObject jsonObject = toJsonObject();
+ return jsonObject.toJSONString();
+ }
+
+ protected void setJsonObject(JSONObject jsonObject) {
+ Class<? extends BasedConfigurable> thisClass = this.getClass();
+ JSONObject tmp = jsonObject.getJSONObject(EXTEND_FIELD_NAME);
+ setJsonObject(thisClass, jsonObject);
+ }
+
+ protected void setJsonObject(Class clazz, JSONObject jsonObject) {
+ if (AbstractConfigurable.class.getName().equals(clazz.getName())) {
+ return;
+ }
+ Field[] fields = clazz.getDeclaredFields();
+ for (Field field : fields) {
+ if (field.isAnnotationPresent(NoSerialized.class)) {
+ continue;
+ }
+ if (Modifier.isStatic(field.getModifiers())) {
+ continue;
+ } else if (Modifier.isTransient(field.getModifiers())) {
+ continue;
+ } else if (Modifier.isNative(field.getModifiers())) {
+ continue;
+ } else if (field.getName().endsWith("this$0")) {
+ continue;
+ }
+
+ DataType dataType = DataTypeUtil.createFieldDataType(this, field.getName());
+ Object fieldValue = ReflectUtil.getBeanFieldValue(this, field.getName());
+ if (fieldValue != null) {
+ // 如果是空值则不再处理。入库也没意义
+ String fieldValueStr = dataType.toDataJson(fieldValue);
+ fieldValueStr = restoreFieldValue(field, fieldValueStr);
+ jsonObject.put(field.getName(), fieldValueStr);
+ }
+ }
+ Class parent = clazz.getSuperclass();
+ setJsonObject(parent, jsonObject);
+ }
+
+ public int getStatus() {
+ int status = 1;
+ if (this.getPrivateData("status") != null) {
+ status = Integer.valueOf(this.getPrivateData("status"));
+ }
+ return status;
+ }
+
+ protected void getJsonObject(JSONObject jsonObject) {
+ Class thisClass = this.getClass();
+ getJsonObject(thisClass, jsonObject);
+ }
+
+ protected void getJsonObject(Class clazz, JSONObject jsonObject) {
+ if (AbstractConfigurable.class.getName().equals(clazz.getName())) {
+ return;
+ }
+ if (Object.class.getName().equals(clazz.getName())) {
+ return;
+ }
+ Field[] fields = clazz.getDeclaredFields();
+ for (Field field : fields) {
+ if (field.isAnnotationPresent(NoSerialized.class)) {
+ continue;
+ }
+ if (Modifier.isStatic(field.getModifiers())) {
+ continue;
+ } else if (Modifier.isTransient(field.getModifiers())) {
+ continue;
+ } else if (Modifier.isNative(field.getModifiers())) {
+ continue;
+ }
+ DataType dataType = DataTypeUtil.createFieldDataType(this, field.getName());
+ String fieldJsonStr = jsonObject.getString(field.getName());
+ fieldJsonStr = getENVParamter(field, fieldJsonStr);
+ Object fieldValue = dataType.getData(fieldJsonStr);
+ if (fieldValue != null) {
+ ReflectUtil.setBeanFieldValue(this, field.getName(), fieldValue);
+ } else {
+ ReflectUtil.setFieldValue(this, field.getName(), null);
+ }
+ }
+ Class parent = clazz.getSuperclass();
+ getJsonObject(parent, jsonObject);
+ }
+
+ /**
+ * 支持存储env的key值，而具体的值存储在IENVParameter参数中
+ *
+ * @param fieldValue
+ * @return
+ */
+ protected String getENVParamter(Field field, String fieldValue) {
+ ENVDependence dependence = field.getAnnotation(ENVDependence.class);
+ if (dependence == null) {
+ return fieldValue;
+ }
+ String value = getENVVar(fieldValue);
+
+ if (StringUtil.isNotEmpty(value)) {
+ String key = MapKeyUtil.createKey(getNameSpace(), getType(), getConfigureName(), field.getName(), value);
+ this.putPrivateData(key, fieldValue);
+ return value;
+ }
+ return fieldValue;
+
+ }
+
+ protected String getENVVar(String fieldValue) {
+ if (StringUtil.isEmpty(fieldValue)) {
+ return null;
+ }
+ String value = ComponentCreator.getProperties().getProperty(fieldValue);
+ if (StringUtil.isNotEmpty(value)) {
+ return value;
+ }
+ return ENVUtile.getENVParameter(fieldValue);
+ }
+
+ private String restoreFieldValue(Field field, String fieldValueStr) {
+ return getOriFieldValue(field, fieldValueStr);
+ }
+
+ public String getOriFieldValue(Field field, String fieldValueStr) {
+ ENVDependence dependence = field.getAnnotation(ENVDependence.class);
+ if (dependence == null) {
+ return fieldValueStr;
+ }
+ String key =
+ MapKeyUtil.createKey(getNameSpace(), getType(), getConfigureName(), field.getName(), fieldValueStr);
+ String oriFieldValue = this.getPrivateData(key);
+ // if(needRemove){
+ // this.removePrivateData(key);
+ // }
+
+ if (StringUtil.isNotEmpty(oriFieldValue)) {
+ return oriFieldValue;
+ }
+ return fieldValueStr;
+ }
+
+ @Override
+ public void toObject(String jsonString) {
+ JSONObject jsonObject = JSON.parseObject(jsonString);
+ getJsonObject(jsonObject);
+ }
+
+ // @Override
+ // public String toString() {
+ // return toJson();
+ // }
+
+ public String getVersion() {
+ return version;
+ }
+
+ public void setVersion(String version) {
+ this.version = version;
+ }
+
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/configurable/IAfterConfiguableRefreshListerner.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/configurable/IAfterConfiguableRefreshListerner.java
new file mode 100644
index 0000000..7d78dff
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/configurable/IAfterConfiguableRefreshListerner.java
@@ -0,0 +1,25 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.configurable;
+
+public interface IAfterConfiguableRefreshListerner {
+
+ /**
+ * 当configurable数据全部加载完成时，调用实现这个接口的configurable对象
+ */
+ void doProcessAfterRefreshConfigurable(IConfigurableService configurableService);
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/configurable/IConfigurable.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/configurable/IConfigurable.java
new file mode 100644
index 0000000..79c5c05
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/configurable/IConfigurable.java
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.configurable;
+
+import org.apache.rocketmq.streams.common.datatype.IJsonable;
+
+import java.io.Serializable;
+import java.util.Map;
+
+/**
+ * 对所有可配置的对象，做统一抽象，可配置的对象是：数据量不大，可以加载到内存，整个框架除了数据外，多数框架对象都是configuahle对象 统一抽象的好处是：可以统一的序列化，反序列化，存储，查询。 通过namespace隔离业务，通过type区分类型，namespace+type+name唯一标识一个对象。所有其他字段会被序列化到json_value中 继承BasedConfigurable 基类，会自动实现序列化和反序列化。也可以自己实现
+ */
+public interface IConfigurable extends IJsonable, IConfigurableIdentification, Serializable {
+
+ /**
+ * 把toJson的结果当作一个特殊属性
+ */
+ static final String JSON_PROPERTY = "configurable_json";
+
+ /**
+ * 把status当作configurable 的一个特殊属性
+ */
+ static final String STATUS_PROPERTY = "configurable_status";
+
+ /**
+ * 每个配置有一个独立的名字
+ *
+ * @param configureName
+ */
+ void setConfigureName(String configureName);
+
+ /**
+ * 每个配置有独立的命名空间
+ *
+ * @param nameSpace
+ */
+ void setNameSpace(String nameSpace);
+
+ /**
+ * 区分配置类型
+ *
+ * @param type
+ */
+ void setType(String type);
+
+ boolean init();
+
+ void destroy();
+
+ /**
+ * 设置私有数据
+ *
+ * @param key
+ * @param value
+ * @param <T>
+ */
+ <T> void putPrivateData(String key, T value);
+
+ /**
+ * 获取私有数据
+ *
+ * @param key
+ * @param <T>
+ * @return
+ */
+ <T> T getPrivateData(String key);
+
+ Map<String, Object> getPrivateData();
+
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/configurable/IConfigurableIdentification.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/configurable/IConfigurableIdentification.java
new file mode 100644
index 0000000..e82ac10
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/configurable/IConfigurableIdentification.java
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.configurable;
+
+public interface IConfigurableIdentification {
+ /**
+ * 配置的名字，同一个配置空间名字必须唯一
+ *
+ * @return 配置项名称
+ */
+ String getConfigureName();
+
+ /**
+ * 用于区分不同的业务，用命名空间隔离业务
+ *
+ * @return 配置项命名空间
+ */
+ String getNameSpace();
+
+ /**
+ * 配置对应的type
+ *
+ * @return 配置项类型
+ */
+ String getType();
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/configurable/IConfigurableListerner.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/configurable/IConfigurableListerner.java
new file mode 100644
index 0000000..5911511
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/configurable/IConfigurableListerner.java
@@ -0,0 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.configurable;
+
+public interface IConfigurableListerner {
+}
diff --git a/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/configurable/IConfigurableService.java b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/configurable/IConfigurableService.java
new file mode 100644
index 0000000..13b7d12
--- /dev/null
+++ b/rocketmq-streams-commons/src/main/java/org/apache/rocketmq/streams/common/configurable/IConfigurableService.java
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.streams.common.configurable;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * 对Configurable对象，做统一的管理，统一查询，插入和更新。 insert/update 把configuabel对象写入存储，支持文件存储（file），内存存储（memory）和db存储（DB）。可以在配置通过这个ConfigureFileKey.CONNECT_TYPE key 配置 query 是基于内存的查询，对象定时load到内存，可以在属性文件通过这个ConfigureFileKey.POLLING_TIME key配置加载周期，单位是秒 新对象加载后生效，已经存在的对象只有updateFlag发生变化才会被替换
+ */
+public interface IConfigurableService {
+
+ String CLASS_NAME = "className";
+
+ String CLASS_TYPE = "classType";
+
+ /**
+ * 顶层的命名空间
+ */
+ String PARENT_CHANNEL_NAME_SPACE = "rocketmq.streams.root.namespace";
+
+ /**
+ * 超顶层的命名空间
+ */
+ String ROOT_CHANNEL_NAME_SPACE = "rocketmq_streams_root_namespace";
+
+ String DEFAULT_SERVICE_NAME = "DB";
+
+ String MEMORY_SERVICE_NAME = "memory";
+
+ String FILE_SERVICE_NAME = "file";
+
+ String HTTP_SERVICE_NAME = "http";
+
+ String FILE_PATH_NAME = "filePathAndName";
+
+ /**
+ * 启动定时任务，定期从存储加载对象到内存
+ *
+ * @param namespace
+ */
+ void initConfigurables(String namespace);
+
+ /**
+ * 从存储加载对象到内存
+ *
+ * @param namespace
+ * @return
+ */
+ boolean refreshConfigurable(String namespace);
+
... 47711 lines suppressed ...
[rocketmq-streams] 35/35: Merge pull request #8 from xstorm1/dev
Posted by du...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
duhengforever pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/rocketmq-streams.git
commit b3063f0f57fd9782c9781be4e24b3675bd638e98
Merge: 7981c23 3209044
Author: Heng Du <du...@apache.org>
AuthorDate: Tue Aug 3 14:49:47 2021 +0800
Merge pull request #8 from xstorm1/dev
add dbinit module
pom.xml | 2 +-
.../streams/common/configure/ConfigureFileKey.java | 1 +
.../rocketmq-streams-db-operator.iml | 16 --
.../rocketmq/streams/db/driver/orm/ORMUtil.java | 23 +++
rocketmq-streams-dbinit/pom.xml | 38 ++++
.../streams/dbinit/mysql/delegate/DBDelegate.java | 9 +
.../dbinit/mysql/delegate/DBDelegateFactory.java | 27 +++
.../streams/dbinit/mysql/delegate/DBType.java | 6 +
.../dbinit/mysql/delegate/MysqlDelegate.java | 48 +++++
.../src/main/resources/tables_mysql_innodb.sql | 199 +++++++++++++++++++++
10 files changed, 352 insertions(+), 17 deletions(-)
[rocketmq-streams] 16/35: modify the README.md
Posted by du...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
duhengforever pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/rocketmq-streams.git
commit ecb842a0a5930488d74bbc9cd906a1306571c0a8
Author: 刈刀 <je...@gmail.com>
AuthorDate: Mon Aug 2 12:35:26 2021 +0800
modify the README.md
---
README.md | 1 -
1 file changed, 1 deletion(-)
diff --git a/README.md b/README.md
index 3b163db..649319c 100644
--- a/README.md
+++ b/README.md
@@ -2,7 +2,6 @@
## Features
-
* 轻量级部署：可以单独部署，也支持集群部署
* 多种类型的数据输入以及输出，source支持 rocketmq ， sink支持db, rocketmq 等