You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by li...@apache.org on 2026/07/11 05:24:45 UTC
(seatunnel-website) branch main updated: [Blog] Add article on Modern Data Stack and Agentic Data Stack (#484)
This is an automated email from the ASF dual-hosted git repository.
davidzollo pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/seatunnel-website.git
The following commit(s) were added to refs/heads/main by this push:
new ef2324157184 [Blog] Add article on Modern Data Stack and Agentic Data Stack (#484)
ef2324157184 is described below
commit ef23241571844b27a9f24e6797d6caad5af0aed8
Author: Daniel <es...@outlook.com>
AuthorDate: Sat Jul 11 13:24:40 2026 +0800
[Blog] Add article on Modern Data Stack and Agentic Data Stack (#484)
---
...7-11-modern-data-stack-to-agentic-data-stack.md | 81 +++++++++++++++++++++
...7-11-modern-data-stack-to-agentic-data-stack.md | 81 +++++++++++++++++++++
.../01-overview.jpg | Bin 0 -> 521915 bytes
.../02-etl-elt-etlt.jpg | Bin 0 -> 346191 bytes
.../03-seatunnel-in-etlt.jpg | Bin 0 -> 446824 bytes
.../04-pure-elt-breakdown.jpg | Bin 0 -> 415192 bytes
.../05-agents-use-seatunnel.jpg | Bin 0 -> 415698 bytes
.../06-agentic-data-stack-layers.jpg | Bin 0 -> 471102 bytes
8 files changed, 162 insertions(+)
diff --git a/blog/2026-07-11-modern-data-stack-to-agentic-data-stack.md b/blog/2026-07-11-modern-data-stack-to-agentic-data-stack.md
new file mode 100644
index 000000000000..e15e3323fb5f
--- /dev/null
+++ b/blog/2026-07-11-modern-data-stack-to-agentic-data-stack.md
@@ -0,0 +1,81 @@
+---
+slug: modern-data-stack-to-agentic-data-stack
+title: "From Modern Data Stack to Agentic Data Stack: Why Apache SeaTunnel Matters as the Data Integration Runtime"
+tags: [Architecture, Data Integration, Lakehouse]
+---
+
+Over the past decade, the Modern Data Stack has changed how teams build data platforms. Data ingestion, lakehouses or cloud warehouses, dbt, orchestration, metrics layers, BI, and governance were split into modular building blocks that can be assembled more flexibly. That shift moved data platforms away from heavy, closed, tightly coupled systems and toward a more open and cloud-native engineering model.
+
+But once teams run that model in production, another truth becomes clear: the quality, speed, and recoverability of the whole platform still depend heavily on the first step, data integration.
+
+Because of that, the next key question in data engineering is no longer only "where data should live." It is also "how data enters the platform continuously, reliably, and in a governed way." From that perspective, Apache SeaTunnel is more than a connector catalog. It is better understood as a data integration runtime for modern platforms and future agent-driven systems.
+
+## 1. Modern Data Stack made platforms more modular, but it still underestimates data ingress
+
+The Modern Data Stack deserves real credit. It separates responsibilities more clearly: ingestion handles extraction, a lakehouse or warehouse provides unified storage and compute, dbt handles much of the transformation layer, orchestration manages lifecycle, and governance, quality, metrics, and BI deliver value to the business.
+
+That separation makes technology choices easier. But in real production systems, the ingestion layer is often the most underestimated part. "Just sync the data in" is rarely a connector-only problem. It quickly becomes an operational problem: CDC, incremental capture, schema drift, source pressure, recovery after failure, duplicate prevention, and downstream data quality.
+
+In other words, Modern Data Stack helped teams appreciate lakehouses and modeling more deeply, but the next step is to recognize the engineering value of the integration layer itself. It decides whether every downstream layer sees complete, timely, consistent, and recoverable data.
+
+
+
+## 2. From ETL to ELT to EtLT: the real shift is responsibility, not terminology
+
+Traditional ETL transforms data before loading it. That works well when rules are stable and business change is limited. The downside is a heavy front-loaded pipeline that becomes harder to maintain as change accelerates.
+
+ELT moved more responsibility to the destination. Teams extract first, load into the lakehouse or warehouse, and transform later. That takes advantage of stronger storage and compute at the platform layer, preserves more raw data, and gives analytics teams more flexibility.
+
+But pure ELT is not always the best answer. If everything is loaded first and cleaned up later, schema drift, dirty fields, CDC inconsistencies, sensitive columns, duplicate rows, and format differences all move into the platform unchanged. Lakehouses, dbt models, semantic layers, and analytics teams then pay the price for problems that should have been handled earlier.
+
+That is why EtLT matters. EtLT does not mean going back to heavyweight ETL. It means adding a necessary lightweight transformation stage before loading: standardization, filtering, mapping, routing, CDC event normalization, and structure adaptation before data lands in the unified foundation. The big semantic transformation still belongs to dbt, SQL models, and semantic layers later.
+
+
+
+## 3. SeaTunnel fits naturally in EtLT as a runtime, not just a list of connectors
+
+Once we look at EtLT, SeaTunnel's role becomes straightforward. It fits the E + lightweight t + L part of the stack: connecting heterogeneous sources, running CDC or incremental ingestion, applying lightweight transformation, and delivering data into lakehouses, engines, or downstream storage.
+
+That is the difference between SeaTunnel and a simple connector catalog. The real question is not just whether a system can connect. It is whether it can keep running, recover from failure, handle state, and prepare data so downstream systems can consume it safely.
+
+Seen that way, SeaTunnel is more accurately described as a data integration runtime for modern data platforms. Depending on the connector, engine, and configuration, it can support patterns such as checkpoint-based recovery, schema-aware handling, and reliable delivery. That is exactly the layer EtLT needs.
+
+
+
+## 4. Why pure ELT starts to break down
+
+Many teams naturally start with pure ELT: if the lakehouse is powerful enough, load everything first and clean it up later. That works in early or exploratory stages. But once the environment includes frequent change, CDC, heterogeneous sources, and always-on production pipelines, the cracks show quickly.
+
+Schema drift can land directly in the platform and make tables harder to manage. Dirty or sensitive columns can increase governance and compliance cost. CDC events from different systems can force downstream models to absorb unnecessary complexity. Duplicate or noisy records can make analytics depend on more and more compensating cleanup.
+
+EtLT addresses this by improving the landing step. It is not about moving all business logic back upstream. It is about making sure the platform receives healthier input: standardized data, schema-aware handling, normalized CDC events, and the filtering or routing that should happen before storage.
+
+
+
+## 5. In the agent era, SeaTunnel becomes more important as an execution layer
+
+If Modern Data Stack made tools modular, Agentic Data Stack makes tools callable. Future platforms will not only rely on people clicking through UIs, configuring jobs manually, and reading logs after the fact. They will increasingly include agents that interpret goals, reason over lineage and impact, choose actions, validate outcomes, and improve over time.
+
+In that loop, agents handle understanding and decision-making. They identify intent, analyze dependencies and blast radius, decide whether a problem needs ETL, ELT, or EtLT, and choose the right tools and risk boundaries.
+
+SeaTunnel's role is different. Through execution APIs, orchestration integrations, or higher-level control layers, it can serve as the data execution layer that turns plans into actual movement: synchronization, CDC ingestion, controlled reruns, selective reload workflows, and recoverable delivery. Agents are useful only when they can act through reliable systems, and that is where a runtime like SeaTunnel matters.
+
+
+
+## 6. SeaTunnel is not just a component of Modern Data Stack, but part of the runtime foundation of Agentic Data Stack
+
+Viewed as a layered system, the next generation of data engineering platforms may look something like this: data sources at the bottom, then a data integration runtime, then a unified data foundation, then transformation and semantics, then orchestration and governance, and finally an agent layer on top.
+
+In that structure, SeaTunnel sits in a critical position. It turns heterogeneous change from the real world into standardized, continuously ingestible input for the platform. Without that layer, the lakehouse is mostly a passive container. With it, the lakehouse has a better chance to become a reliable foundation of truth.
+
+That is why SeaTunnel should not be treated as only a "sync tool." In the Modern Data Stack, it carries real production-grade integration responsibilities. In the Agentic Data Stack, it becomes part of the runtime infrastructure that allows data engineering systems to move from analysis to action.
+
+
+
+## Conclusion
+
+The shift from Modern Data Stack to Agentic Data Stack is not just about adding an LLM interface. It is about changing the operating model of data engineering: people define goals, systems generate actions, tools execute those actions, and feedback improves the next round.
+
+In that world, lakehouses matter, dbt matters, orchestration matters, and governance matters. But the question of who brings real-world data into the platform in a reliable and recoverable way still points to the data integration runtime.
+
+That is where Apache SeaTunnel stands out. It is not merely a catalog of connectors. It is one of the layers that makes the rest of the platform operational.
diff --git a/i18n/zh-CN/docusaurus-plugin-content-blog/2026-07-11-modern-data-stack-to-agentic-data-stack.md b/i18n/zh-CN/docusaurus-plugin-content-blog/2026-07-11-modern-data-stack-to-agentic-data-stack.md
new file mode 100644
index 000000000000..b55734743bec
--- /dev/null
+++ b/i18n/zh-CN/docusaurus-plugin-content-blog/2026-07-11-modern-data-stack-to-agentic-data-stack.md
@@ -0,0 +1,81 @@
+---
+slug: modern-data-stack-to-agentic-data-stack
+title: "从 Modern Data Stack 到 Agentic Data Stack:为什么 Apache SeaTunnel 值得被理解为数据集成运行时"
+tags: [Architecture, Data Integration, Lakehouse]
+---
+
+过去十年,Modern Data Stack 改变了企业建设数据平台的方式。数据摄取、Lakehouse 或云数仓、dbt、编排系统、指标层、BI 与治理工具,被重新拆分成一组可以灵活组合的模块化能力。它让数据平台从过去那种重型、封闭、强耦合的体系,逐步演化成一个更开放、更云原生、也更容易扩展的现代工程系统。
+
+但当企业真正把这套体系跑到生产环境,另一个事实会越来越明显:整个平台的输入质量、变化速度和可恢复性,往往还是取决于最前面的那一层,也就是数据集成。
+
+也正因为如此,下一代数据工程体系的关键问题,已经不再只是“把数据放到哪里”,而是“数据如何持续、可靠、可治理地进入平台”。从这个角度看,Apache SeaTunnel 的价值,不应该只被理解成一个 connector 工具,而更应该被理解成面向现代数据平台和未来 Agent 系统的 Data Integration Runtime。
+
+## 一、Modern Data Stack 让平台更模块化了,但也低估了“数据进入平台”这件事
+
+Modern Data Stack 的贡献非常大。它把传统数据平台中混在一起的一大团能力,拆成了更清晰的层次:数据摄取负责 E,Lakehouse / Warehouse 负责统一存储与计算,dbt 负责大部分建模与转换,编排系统负责任务生命周期,治理、质量、指标与 BI 负责把数据价值推向业务侧。
+
+这种分层让企业更容易选型,也更容易把不同技术组合成适合自己的平台。但在真实生产环境里,最容易被低估的恰恰是最前面的数据摄取层。因为所谓“把数据同步进来”从来都不只是一个连接器问题,它很快会变成一整套持续运行的问题:要不要做 CDC、如何做增量捕获、如何处理 schema drift、如何控制源端压力、如何在失败后恢复、如何避免重复写入、如何让下游接收到更健康的数据。
+
+换句话说,Modern Data Stack 让企业重新认识了 Lakehouse 和建模层的重要性,但今天我们还需要继续往前走一步:重新认识数据集成层本身的工程价值。它决定了后面所有层看到的事实是否完整、及时、一致、可恢复。
+
+
+
+## 二、从 ETL 到 ELT,再到 EtLT:真正变化的是责任边界,不只是术语
+
+传统 ETL 的逻辑是先 Transform 再 Load。它适合规则明确、边界稳定、变化不快的场景。优势是进入目标系统之前就已经把数据处理得比较规整,但代价是前置链路重、灵活性弱,一旦业务变化频繁,维护成本就会持续上升。
+
+ELT 则把更多事情推到目标端去做:先 Extract,再 Load,最后在 Lakehouse 或云数仓里 Transform。它充分利用了现代底座更强的存储和计算能力,让企业能保留更多原始数据,也让分析建模更灵活。
+
+但 Pure ELT 并不总是最优解。如果什么都先加载进去,再统一处理,那么 schema drift、脏字段、CDC 事件差异、敏感数据、重复记录、格式不一致等问题,也会被原样推进平台。后面的 Lakehouse、dbt、语义层和分析团队,就不得不为前面没有处理好的工程问题买单。
+
+这也是 EtLT 变得重要的原因。EtLT 并不是重新回到重型 ETL,而是在 Load 之前增加一层必要的 lightweight transform:先做轻量标准化、过滤、映射、路由、CDC 事件整理和结构适配,再进入统一底座,最后再由 dbt、SQL 模型和语义层完成真正面向业务的大 Transform。它重新划分的不是字母顺序,而是工程责任边界。
+
+
+
+## 三、SeaTunnel 在 EtLT 里的位置:它更像运行时,而不只是连接器集合
+
+当我们把视角切到 EtLT,就会发现 SeaTunnel 的定位非常自然。它最适合承担的是 E + 小写 t + L:连接异构数据源、执行 CDC / 增量采集、完成轻量 transform,并把数据稳定送入 Lakehouse、分析引擎或下游存储系统。
+
+这也是 SeaTunnel 和单纯 connector catalog 的区别。企业真正需要的不是“能不能连上”,而是“连上之后能不能持续跑、稳定跑、出错后恢复、处理状态,并把数据准备成下游可消费的样子”。
+
+从工程本质看,更准确的表达不是“SeaTunnel 是一个数据同步工具”,而是“SeaTunnel 是面向现代数据平台的数据集成运行时”。结合不同 connector、执行引擎和配置方式,它可以承接 checkpoint 恢复、schema-aware 处理、可靠交付等模式,这恰恰是 EtLT 最需要的一层。
+
+
+
+## 四、为什么 Pure ELT 会开始失灵:好分析的前提,是先有一次“好落地”
+
+很多团队刚开始做现代数据平台时,会天然倾向于 Pure ELT:既然 Lakehouse 足够强,那就先全量装进去,后面再慢慢整理。这种思路在早期、简单或探索性场景里没有问题,但只要进入高频变更、CDC、多源异构和持续生产运行阶段,问题就会迅速冒出来。
+
+schema drift 会直接带进平台,让表结构越来越难治理;脏字段和敏感字段未经处理就落地,会抬高治理与合规成本;不同来源的 CDC 事件差异会把复杂度推给下游模型;重复和噪声记录增多后,分析结果会越来越依赖补救性清洗。
+
+EtLT 的意义就在这里。它不是为了把所有业务逻辑都搬回前置层,而是为了先把 landing 这一步做好:把输入标准化、把 schema-aware 处理做好、把 CDC 事件整理清楚、把必要的过滤和路由完成,再把数据送进 Lakehouse。只有这样,下游的 dbt、SQL 模型、语义层和指标层,才不会一开始就建立在混乱输入上。
+
+
+
+## 五、到了 Agent 时代,SeaTunnel 会更像“可被调用的数据执行层”
+
+如果说 Modern Data Stack 解决的是工具模块化,那么 Agentic Data Stack 解决的就是工具可调用。未来的数据平台不会只是让人手动点界面、配任务、看日志,而会越来越多地让 Agent 围绕目标去理解系统、分析上下文、规划动作、验证结果,并根据反馈不断优化。
+
+在这个闭环里,Agent 负责的是理解与决策:识别目标属于什么业务语境,分析 lineage、依赖和影响范围,判断该走 ETL、ELT 还是 EtLT,决定调用哪些工具、承担哪些风险。
+
+SeaTunnel 负责的则是把这些决策变成真正可执行的数据动作。更准确地说,它可以通过执行 API、编排集成或更高层控制层,承接同步、CDC 执行、受控重跑、选择性重载以及可恢复交付等动作。Agent 只有在能够调用可靠的数据执行系统时,才真正具备“行动力”,而这正是运行时层的重要性所在。
+
+
+
+## 六、最终形态:SeaTunnel 不只是 Modern Data Stack 的一个组件,而是 Agentic Data Stack 的运行时基础设施之一
+
+如果用分层视角来看下一代数据工程平台,我们大致会看到这样一套结构:最底层是 Data Sources;其上是 Data Integration Runtime;再往上是 Unified Data Foundation;然后是 Transformation & Semantics;再上面是 Orchestration & Governance;最上层才是 Agent Layer。
+
+在这个结构里,SeaTunnel 所在的位置非常关键。它处于“数据源”和“统一数据底座”之间,负责把原始世界里的异构变化,转化成平台能够持续承接的标准化输入。没有这一层,Lakehouse 更像一个被动接收数据的容器;有了这一层,Lakehouse 才更有机会成为稳定的统一事实底座。
+
+也正因为如此,SeaTunnel 不应该只被理解成某个时代里的“同步工具”。在 Modern Data Stack 里,它承担的是更真实、更生产级的数据集成职责;到了 Agentic Data Stack 里,它进一步成为让数据工程系统从“能分析”走向“能行动”的运行时基础设施之一。
+
+
+
+## 结语
+
+从 Modern Data Stack 走向 Agentic Data Stack,真正变化的并不是多了一个大模型界面,而是数据工程的操作范式正在改变:过去是人定义流程、系统执行流程;未来会越来越变成人定义目标、系统生成动作、工具执行动作、反馈再回流到系统。
+
+在这个变化里,Lakehouse 很重要,dbt 很重要,编排和治理也都很重要。但如果要问“谁负责把真实世界的数据变化,持续、可靠、可恢复地带入平台”,答案仍然会指向数据集成运行时。
+
+这正是 Apache SeaTunnel 的独特价值所在。它不是简单的连接器列表,而是让整个数据平台真正运转起来的一层关键基础设施。
diff --git a/static/image/20260711/modern-data-stack-to-agentic-data-stack/01-overview.jpg b/static/image/20260711/modern-data-stack-to-agentic-data-stack/01-overview.jpg
new file mode 100644
index 000000000000..a8261703ab2d
Binary files /dev/null and b/static/image/20260711/modern-data-stack-to-agentic-data-stack/01-overview.jpg differ
diff --git a/static/image/20260711/modern-data-stack-to-agentic-data-stack/02-etl-elt-etlt.jpg b/static/image/20260711/modern-data-stack-to-agentic-data-stack/02-etl-elt-etlt.jpg
new file mode 100644
index 000000000000..9c09c5d0f6ef
Binary files /dev/null and b/static/image/20260711/modern-data-stack-to-agentic-data-stack/02-etl-elt-etlt.jpg differ
diff --git a/static/image/20260711/modern-data-stack-to-agentic-data-stack/03-seatunnel-in-etlt.jpg b/static/image/20260711/modern-data-stack-to-agentic-data-stack/03-seatunnel-in-etlt.jpg
new file mode 100644
index 000000000000..84673a49f0e2
Binary files /dev/null and b/static/image/20260711/modern-data-stack-to-agentic-data-stack/03-seatunnel-in-etlt.jpg differ
diff --git a/static/image/20260711/modern-data-stack-to-agentic-data-stack/04-pure-elt-breakdown.jpg b/static/image/20260711/modern-data-stack-to-agentic-data-stack/04-pure-elt-breakdown.jpg
new file mode 100644
index 000000000000..cf0981987325
Binary files /dev/null and b/static/image/20260711/modern-data-stack-to-agentic-data-stack/04-pure-elt-breakdown.jpg differ
diff --git a/static/image/20260711/modern-data-stack-to-agentic-data-stack/05-agents-use-seatunnel.jpg b/static/image/20260711/modern-data-stack-to-agentic-data-stack/05-agents-use-seatunnel.jpg
new file mode 100644
index 000000000000..f843a5033b78
Binary files /dev/null and b/static/image/20260711/modern-data-stack-to-agentic-data-stack/05-agents-use-seatunnel.jpg differ
diff --git a/static/image/20260711/modern-data-stack-to-agentic-data-stack/06-agentic-data-stack-layers.jpg b/static/image/20260711/modern-data-stack-to-agentic-data-stack/06-agentic-data-stack-layers.jpg
new file mode 100644
index 000000000000..5b8a6932bcd1
Binary files /dev/null and b/static/image/20260711/modern-data-stack-to-agentic-data-stack/06-agentic-data-stack-layers.jpg differ