You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by do...@apache.org on 2022/09/12 02:39:16 UTC
[inlong-website] branch master updated: [INLONG-523][Release] Add blog for the 1.3.0 release (#543)
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong-website.git
The following commit(s) were added to refs/heads/master by this push:
new 77ef7f7c1c [INLONG-523][Release] Add blog for the 1.3.0 release (#543)
77ef7f7c1c is described below
commit 77ef7f7c1cbd009d70678044412765d351e8ed9b
Author: Schnapps <zp...@connect.ust.hk>
AuthorDate: Mon Sep 12 10:39:10 2022 +0800
[INLONG-523][Release] Add blog for the 1.3.0 release (#543)
Co-authored-by: EMsnap <st...@tencent.com>
---
blog/2022-09-05-release-1.3.0.md | 99 ++++++++++++++++++++
.../2022-09-05-release-1.3.0.md | 100 +++++++++++++++++++++
2 files changed, 199 insertions(+)
diff --git a/blog/2022-09-05-release-1.3.0.md b/blog/2022-09-05-release-1.3.0.md
new file mode 100644
index 0000000000..f386ec5b0d
--- /dev/null
+++ b/blog/2022-09-05-release-1.3.0.md
@@ -0,0 +1,99 @@
+---
+title: Release InLong 1.3.0
+author: EMsnap
+author_url: https://github.com/EMsnap
+author_image_url: https://avatars.githubusercontent.com/u/26538404?v=4
+tags: [Apache InLong, Version]
+---
+
+Apache InLong is a one-stop integration framework for massive data that provides automatic, secure and reliable data transmission capabilities.
+InLong supports both batch and stream data processing at the same time, which offers great power to build data analysis, modeling and other real-time applications based on streaming data.
+
+<!--truncate-->
+
+## 1.3.0 Features Overview
+**The just-released 1.3.0-incubating version closes about 410+ issues, contains 110+ features and 170+ optimizations.**
+Mainly include the following:
+
+### Enhance management and control capabilities
+- Added permission authentication for Open Api
+- Added cluster heartbeat mechanism for Agent and DataProxy
+- Manager adapts to two roles such as user and approver
+- Abstract operations on Load nodes to support easy scaling of Load node resources
+- Supports creation of databases and tables for SQLServer, Oracle and MySQL
+- Enhanced functionality of the Manager client, including but not limited to user and data node management
+
+### Extended collection node
+- Support for collecting data in TubeMq
+- Support for collecting data in Redis
+- Support for collecting data in Doris
+- Support to collect data in Pulsar without AdminUrl
+
+### Optimize write node
+- Kafka Sink supports All Changelog Mode
+- JDBC Sink supports All Changelog Mode
+
+### Support data conversion
+- Support Union operator
+- Support encrypted Udf
+- Support Json Udf
+- Support Temporal Join
+- Support Lookup Join
+- Support Interval Join
+
+### Strengthen Agent function
+- Support regular expression custom line break: default "\n" line ending mark, custom regular matching line ending mark can realize multi-line merging and folding
+- Support K8s log collection and carry cluster information
+- Supports standard output, node log collection, and will carry container and cluster information for standard output
+- Support full and incremental collection of file content
+- Supports automatic heartbeat reporting and registration to Manager
+- Support custom IP and get IP automatically
+
+### Other optimizations
+- GitHub Action check, pipeline optimization
+- DataProxy improves monitoring capabilities such as auditing and indicator reporting
+- DataProxy adds c++ sdk data reporting capability
+- Sort Support metrics report and audit report
+
+## 1.3.0 Features Details
+
+### Abstracting Load node operations
+Manager abstracts Load nodes to support easy expansion of Load node resources and greatly reduce the development time of a Load node
+This part of the feature was contributed by @ciscozhou
+
+### Add permission authentication for Manager Open Api
+In the old version, the Manager Open Api can be accessed anonymously, and in the new version, it is implemented using the Apache Shiro framework.
+Login authentication method based on Basic Access Authentication, this part of the function was contributed by @woofyzhao
+
+### Enhanced collection of file data and k8s logs
+Version 1.3.0 enhances the collection of file data and k8s data, in which file collection supports regular expression custom line breaks, so that multiple lines can be merged and folded
+In addition, the new version of Agent supports full and incremental collection of file content. This part of the function was contributed by @ganfengtan
+
+### DataProxy adds c++ sdk capability
+In addition to the current java client, DataProxy has added c++ client capabilities, which are provided by @pocozh
+
+### Supports multiple udf and join operators
+The new version of Sort supports three kinds of Temporal Join\ Lookup Join \ Interval Join, this part of the function was contributed by @yunqingmoswu
+Most community users mentioned the need for encryption and decryption and Json Udf, this part of the function was contributed by @Emsnap and @Emhui
+
+### Sort connector supports indicator reporting function
+The new version of Sort Connector supports Flink built-in indicator reporting of various Connectors. External indicator systems such as Prometheus can directly obtain the number and rate of task data read and write.
+In addition, the new version also supports InLong Audit Audit data reporting, which is contributed by @pacigong, @Emsnap, @thesumery @Oneal65 @yunqingmoswu
+
+### Manager supports the creation of resources in multiple flow directions
+In version 1.3.0, Manager added the creation of some storage resources:
+
+- Create Topic for SQLServer
+- Create Oracle libraries and tables
+- Create MySQL namespaces and tables
+
+The above are all contributed by community member @haibo-duan, thanks
+
+### Other features and bug fixes
+For related content, please refer to the [Release Notes](https://github.com/apache/inlong/blob/master/CHANGES.md), which details the features, enhancements and bug fixes of this release.
+
+## Apache InLong follow-up planning
+In subsequent versions, we will expand more data sources and storages to cover more usage scenarios, and gradually improve the usability and robustness of the system, including:
+- Agent adds Redis, CloudEvents, MongoDB collection types
+- Unified DataProxy MQ framework
+- Full support for Apache Kafka
\ No newline at end of file
diff --git a/i18n/zh-CN/docusaurus-plugin-content-blog/2022-09-05-release-1.3.0.md b/i18n/zh-CN/docusaurus-plugin-content-blog/2022-09-05-release-1.3.0.md
new file mode 100644
index 0000000000..09f47e5e84
--- /dev/null
+++ b/i18n/zh-CN/docusaurus-plugin-content-blog/2022-09-05-release-1.3.0.md
@@ -0,0 +1,100 @@
+---
+title: 1.3.0 版本发布
+author: EMsnap
+author_url: https://github.com/EMsnap
+author_image_url: https://avatars.githubusercontent.com/u/26538404?v=4
+tags: [Apache InLong, Version]
+---
+
+Apache InLong（应龙）是一个一站式海量数据集成框架，提供自动、安全、可靠和高性能的数据传输能力，同时支持批和流，方便业务构建基于流式的数据分析、建模和应用。
+InLong 支持大数据领域的采集、汇聚、缓存和分拣功能，用户只需要简单的配置就可以把数据从数据源导入到实时计算引擎或者落地到离线存储。
+
+<!--truncate-->
+
+## 1.3.0 版本特性总览
+**刚刚发布的 1.3.0 版本关闭了约 410+ 个 issue，包含 110+ 个特性和 170+ 个优化。** 主要包括以下内容：
+
+### 增强管控能力
+- 增加 Open API 的权限认证
+- 支持 Agent 和 DataProxy 集群的心跳上报机制
+- Manager 支持对“用户”和“审批人”等角色的管理
+- 抽象 Load 节点的操作，以支持快速扩展 Load 节点的资源
+- 支持创建 SQLServer、Oracle 和 MySQL 的数据库和表
+- 增强 Manager 客户端的功能，包括但不限于用户和数据节点管理
+
+
+### 扩展采集节点
+- 支持采集 TubeMQ 中的数据
+- 支持采集 Redis 中的数据
+- 支持采集 Doris 中的数据
+- 支持采集无 AdminUrl 的 Pulsar 中的数据
+
+
+### 优化写入节点
+- Kafka Sink 支持 All Changelog Mode
+- JDBC Sink 支持 All Changelog Mode
+
+### 支持数据转换
+- 支持 Union 操作符
+- 支持 加密 UDF
+- 支持 JSON UDF
+- 支持 Temporal Join/Lookup Join 连接
+- 支持 Interval Join 连接
+
+### 强化Agent功能
+- 支持正则表达式自定义换行：默认 “\n” 行结束标识，自定义正则匹配行结束标识可以实现多行合并折叠
+- 支持 K8s 日志采集并携带集群信息
+- 支持标准输出、节点日志采集，针对标准输出会携带容器和集群信息
+- 支持文件内容全量和增量采集
+- 支持自动上报心跳和注册到 Manager
+- 支持自定义 IP 和自动获取 IP
+
+### 其他优化
+- GitHub Action 检查、流水线优化
+- DataProxy 完善审计、指标上报等监控能力
+- DataProxy 新增 C++ SDK 数据上报能力
+- Sort 支持指标上报以及 Audit 上报
+
+## 1.3.0 版本特性介绍
+
+### 抽象化 Load 节点操作
+Manager 对 Load 节点的抽象操作，以支持快速扩展 Load 节点的资源，大幅减少 Load 节点的开发时间。
+该部分功能由 @ciscozhou 贡献。
+
+### 增加 Manager Open API 的权限认证
+旧版本中 Manager Open API 可以匿名访问，新版本中使用 Apache Shiro 框架实现了基于 Basic Access Authentication 的登录认证方式，该部分功能由 @woofyzhao 贡献。
+
+### 增强对文件数据和 K8s 日志的采集
+1.3.0 版本增强了对文件数据以及 K8s 数据的采集，其中文件采集支持了正则表达式自定义换行符，从而可以实现多行合并折叠。
+另外新版本 Agent 支持文件内容全量和增量采集，这些功能均由 @ganfengtan 贡献。
+
+### DataProxy 新增 C++ SDK 能力
+DataProxy 在原 Java 客户端之外，新增了 C++ 的客户端能力，该部分功能由 @pocozh 贡献。
+
+### Sort 支持多种 UDF 以及 join 操作符
+Sort 支持 Temporal Join / Lookup Join / Interval Join，该部分功能由 @yunqingmoswu 贡献。
+
+另外，大量社区用户提出需要加解密和 JSON UDF 的功能，在此版本中，@Emsnap 和 @Emhui 贡献了相关特性。
+
+### Sort Connector 支持指标上报功能
+新版本的 Sort Connector 中支持多种 Connector 的 Flink 内置指标上报，外部指标系统能够比如 Prometheus 能够直接获任务数据读取以及写入的条数和速率。
+
+除此之外，新版本还支持 InLong Audit 审计数据上报，该部分功能由 @pacigong、@Emsnap、@thesumery，@Oneal65 和 @yunqingmoswu 等人贡献。
+
+### Manager 支持创建多种流向的资源
+1.3.0 版本中 Manager 增加了对部分存储资源（库表结构）的创建：
+
+- 创建 SQLServer 的 库和表
+- 创建 Oracle 的库和表
+- 创建 MySQL 的库和表
+
+以上特性均由社区爱好者 @haibo-duan 贡献。
+
+### 其他特性及问题修复
+更多内容，请参考 [版本说明](https://github.com/apache/inlong/blob/master/CHANGES.md) ，其中详细列出了此版本的特性、提升和 Bug 修复。
+
+## Apache InLong 后续规划
+后续版本，我们扩展更多的数据源端和存储端，覆盖更多的使用场景，并逐步提升系统的易用性和健壮性，主要包括：
+- Agent 新增 Redis、CloudEvents、MongoDB 采集类型
+- 统一 DataProxy 中 MQ 的处理方式
+- 全链路支持 Apache Kafka
\ No newline at end of file