You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@seatunnel.apache.org by David Zollo <da...@gmail.com> on 2026/07/01 06:54:06 UTC
Re: [DISCUSS] Apache SeaTunnel 3.0 Roadmap
Hi guys,
I created an issue for the 3.0 roadmap to track progress[1]. Given that
most PRs have been merged so far, we plan to finish merging the remaining
PRs by the end of July and then kick off the release process!
1. https://github.com/apache/seatunnel/issues/11021
Best Regards,
David
Linkedin: https://www.linkedin.com/in/davidzollo
On Sat, May 16, 2026 at 12:32 PM David Zollo <da...@gmail.com>
wrote:
> Hi SeaTunnel community,
>
> Thanks everyone for the active discussion around the Apache SeaTunnel 3.0
> roadmap.
>
> To make the follow-up review easier, I summarized the proposals collected
> in this thread so far. This is only a discussion summary, not a final
> decision. Please feel free to correct me if I missed anything or
> misunderstood any proposal.
>
> 1.
>
> SeaTunnel 3.0 roadmap discussion
>
> The discussion was started to collect proposals for Apache SeaTunnel 3.0
> after PR #10510, which updated the version to 3.0.0, was merged into the
> dev branch on March 11, 2026.
>
> Several already-merged PRs were mentioned as possible parts of the 3.0
> scope:
>
> -
>
> PR #10586: [Feature][Zeta] Add datasource spi
> https://github.com/apache/seatunnel/pull/10586
> -
>
> PR #10585: [Feature][Zeta] Add /option-rules REST API for runtime
> connector metadata
> https://github.com/apache/seatunnel/pull/10585
> -
>
> PR #10489: [Feature][Transform] Add Zeta SQL UDF context and lifecycle
> hooks with e2e/docs updates
> https://github.com/apache/seatunnel/pull/10489
>
> The purpose of this thread is to collect PRs, features, improvements, and
> ideas that the community believes should be considered for SeaTunnel 3.0.
>
> 2.
>
> STIP-23: Engine-level timer flush for Sink connectors
>
> zhiwei niu proposed including STIP-23 in the SeaTunnel 3.0 roadmap. The
> proposal focuses on engine-level timer flush support for Sink connectors.
>
> Related PRs:
>
> -
>
> PR #10800: [Feature][Zeta] Phase 1: Engine timer flush core
> https://github.com/apache/seatunnel/pull/10800
>
> This PR adds sink.flush.interval and introduces a FlushSignal
> control-plane path through the Zeta engine. It allows sinks to flush based
> on a wall-clock interval without each sink managing its own timer thread.
> -
>
> PR #10801: [Feature][Connector-V2] Phase 2: JDBC connector adoption
> https://github.com/apache/seatunnel/pull/10801
>
> This PR wires registerFlushAction into the JDBC non-XA sink writer
> through a new enable_timer_flush option. It also includes multi-table
> aggregated flush and MySQL CDC E2E tests.
> -
>
> PR #10802: [Docs][Connector-V2] Phase 3: Timer flush feature matrix
> https://github.com/apache/seatunnel/pull/10802
>
> This PR adds a timer flush support marker to sink connector
> documentation and documents the sink.flush.interval and registerFlushAction
> contract.
>
> The main value is to make end-to-end latency more predictable for buffered
> sinks, especially in idle or low-throughput streaming jobs.
>
> 3.
>
> Independent seatunnel-shade project and packaging efficiency
> optimization
>
> Zhang Donghao proposed including PR #9993 in the SeaTunnel 3.0 roadmap.
>
> -
>
> PR #9993:
> https://github.com/apache/seatunnel/pull/9993
>
> This PR extracts the shade modules into an independent seatunnel-shade
> project, supports uploading shade jars to Maven Central, and optimizes the
> overall packaging efficiency of SeaTunnel.
>
> The proposal points out that current shade module builds, especially
> Hadoop3 Uber and Hadoop AWS, add significant build time. By publishing
> shade jars separately and reusing them as dependencies, SeaTunnel can
> reduce repeated build cost and improve release stability.
>
> 4.
>
> Table-level fault isolation for multi-table synchronization
>
> yi zeng proposed including PR #10600 in the SeaTunnel 3.0 roadmap.
>
> -
>
> PR #10600: [Feature][Core] Support table-level fault isolation for
> multi-table synchronization
> https://github.com/apache/seatunnel/pull/10600
>
> This PR introduces table-level fault isolation for multi-table
> synchronization through a new failure policy:
>
> env {
> multi_table {
> failure_policy = "CONTINUE_OTHER_TABLES"
> }
> }
>
> With this enabled, failures can be isolated at table granularity. Healthy
> tables can continue processing even if one table fails. Failed tables are
> collected and reported with metadata such as table path, failure phase,
> plugin, and error.
>
> The proposal is backward-compatible by default because the existing
> FAIL_FAST behavior remains unchanged unless users explicitly enable the new
> policy.
>
> 5.
>
> Checkpoint-gated post_sync_action and retention for file source
> continuous discovery
>
> yi zeng also proposed including PR #10563.
>
> -
>
> PR #10563: [Feature][Connector-V2] Add checkpoint-gated
> post_sync_action and retention for file source continuous discovery
> https://github.com/apache/seatunnel/pull/10563
>
> This PR adds checkpoint-aware source-side lifecycle management for file
> source continuous discovery, including:
>
> -
>
> post_sync_action = delete
> -
>
> post_sync_action = backup
> -
>
> retention_max_age
>
> The key idea is that source files should only be deleted or moved after a
> checkpoint has completed, so that continuous file ingestion remains safe
> and recoverable in production environments.
>
> 6.
>
> STIP-21 / StainTrace: End-to-end data lineage tracing and performance
> analysis for Zeta
>
> David proposed including PR #10491 in the SeaTunnel 3.0 roadmap.
>
> -
>
> PR #10491:
> https://github.com/apache/seatunnel/pull/10491
>
> StainTrace is the implementation of STIP-21. It provides end-to-end data
> lineage tracing and performance analysis for the Zeta engine.
>
> It adds lightweight timestamp sampling points at six key pipeline stages:
>
> -
>
> SOURCE_EMIT
> -
>
> QUEUE_IN
> -
>
> QUEUE_OUT
> -
>
> TRANSFORM_IN
> -
>
> TRANSFORM_OUT
> -
>
> SINK_WRITE_DONE
>
> The goal is to track the full journey of sampled rows from source emission
> to sink write completion, making it easier to measure end-to-end latency
> and identify bottlenecks.
>
> The proposal emphasizes production safety: the feature is disabled by
> default, supports configurable sampling, has a per-worker event budget,
> writes trace data locally as JSONL in OpenTelemetry OTLP JSON format, and
> includes an offline analyzer for P95/P99 latency and HTML visualization
> reports.
>
> 7.
>
> StateStore abstraction for Zeta engine
>
> Doyeon Kim proposed including PR #10812 in the SeaTunnel 3.0 roadmap.
>
> -
>
> PR #10812:
> https://github.com/apache/seatunnel/pull/10812
>
> This PR introduces an initial StateStore abstraction layer to reduce
> direct Hazelcast IMap dependencies in the Zeta engine. It makes state
> responsibilities more explicit and prepares the engine for future evolution
> around state storage, node communication, and coordination layers.
>
> 8.
>
> IMap external storage compaction
>
> Doyeon Kim also proposed including PR #10399.
>
> -
>
> PR #10399:
> https://github.com/apache/seatunnel/pull/10399
>
> This PR introduces LSM-style write and compaction logic for IMap external
> storage. It helps control storage growth by flushing sorted immutable files
> and compacting persisted files while keeping only the latest record per key.
>
> The proposal is intended to improve stability for long-running Zeta jobs
> by reducing duplicated records and tombstones in external storage.
>
> 9.
>
> AI-Native CLI and AI for Data Engineering
>
> sean zhang proposed including the AI-Native CLI capability in SeaTunnel
> 3.0.
>
> -
>
> PR #10789:
> https://github.com/apache/seatunnel/pull/10789
>
> This PR introduces seatunnel-cli, an AI-powered command-line tool that can
> generate production-ready pipeline configurations from natural language.
>
> Current capabilities include:
>
> -
>
> Multi-LLM provider support, including Anthropic, OpenAI, and AWS
> Bedrock
> -
>
> Runtime connector metadata extraction via Java OptionRule reflection
> -
>
> Source/Sink type-aware validation
> -
>
> Auto-fix loop for generated configurations
> -
>
> Sensitive value redaction before persistence and LLM transmission
> -
>
> Interactive mode with session memory
> -
>
> One-shot mode for CI/CD
>
> The proposal also describes a broader “AI for Data Engineering” roadmap,
> including intelligent schema mapping, natural language transforms, data
> profiling, failure diagnosis, self-healing pipelines, anomaly detection,
> performance tuning, cost-aware planning, intelligent scheduling, and
> eventually autonomous data infrastructure management.
>
> 10.
>
> Extension of the roadmap discussion window
>
> David proposed extending the SeaTunnel 3.0 roadmap discussion window to
> May 15, 2026, considering the upcoming holiday period and to give
> contributors more time to share ideas.
>
> This message also recognized the quality and breadth of the proposals so
> far, including StainTrace, AI-Native CLI, StateStore abstraction, and
> table-level fault isolation.
>
> 11.
>
> STIP-27: Apache Arrow as an optional internal data model
>
> xuepeng wang proposed adding STIP-27 to the SeaTunnel 3.0 roadmap
> discussion.
>
> -
>
> Issue #10806:
> https://github.com/apache/seatunnel/issues/10806
>
> This proposal introduces Apache Arrow as an optional internal data model,
> not as a replacement for the existing row model.
>
> The goal is to improve performance for columnar scenarios such as Parquet,
> ORC, ClickHouse, Doris, and batch transforms such as projection, filter,
> and cast.
>
> Expected benefits include better columnar I/O performance, vectorized
> execution, and lower JVM overhead.
>
> 12.
>
> STIP-24: Edge collection, EdgeSocket ingress, and SeaTunnel Edge Agent
>
> zhiwei niu proposed adding STIP-24 edge-collection work to the SeaTunnel
> 3.0 roadmap.
>
> The original PRs were:
>
> -
>
> PR #10871: [Feature][Zeta] STIP-24 Phase 1: Support lightweight
> EdgeSocket ingress for edge collector MVP
> https://github.com/apache/seatunnel/pull/10871
> -
>
> PR #10876: [Feature][Zeta] STIP-24 Phase 2: Introduce standalone
> SeaTunnel Edge Agent MVP and dist integration
> https://github.com/apache/seatunnel/pull/10876
>
> Phase 1 introduces the engine-side ingress foundation for edge collection
> through EdgeSocket, including authenticated ingress, source-side protocol
> handling, and task-group address discovery support in Zeta.
>
> Phase 2 introduces a standalone SeaTunnel Edge Agent runtime, including
> core, transport, connector, local file/log/event inputs, SQLite WAL
> durability, EdgeSocket transport interoperability, and distribution
> integration.
>
> The goal is to support an end-to-end edge collection workflow:
>
> edge host -> Zeta pipeline
>
> zhiwei later updated the PR references for STIP-24:
>
> -
>
> PR #10871 was updated to PR #10878
> https://github.com/apache/seatunnel/pull/10878
> -
>
> PR #10876 was updated to PR #10879
> https://github.com/apache/seatunnel/pull/10879
>
>
> 13.
>
> Prometheus metrics for Zeta Engine IMap state stores
>
> JeremyXin proposed including PR #10860 in the SeaTunnel 3.0 roadmap.
>
> -
>
> PR #10860:
> https://github.com/apache/seatunnel/pull/10860
>
> This PR improves Zeta Engine observability by exposing Prometheus metrics
> for core engine state stores backed by Hazelcast IMap.
>
> It covers internal stores such as running job information, job states,
> checkpoints, and slot ownership. It exposes metrics such as local entry
> counts, backup entry counts, and heap cost.
>
> The proposal aims to help users detect state growth, data skew, and memory
> pressure earlier, improving troubleshooting, capacity planning, and
> production stability.
>
> 14.
>
> Real-time observability for Zeta engine tasks
>
> zhangshenghang proposed improving real-time observability for Zeta engine
> tasks.
>
> This direction adds the ability to monitor the runtime busyness level of
> Zeta engine tasks, including real-time task metrics for Source, Transform,
> and Sink execution. It also includes queue and backpressure-related
> information exposed through REST API and Web UI.
>
> The goal is to help users understand whether a job is blocked by source
> reading, transform processing, sink writing, or intermediate queues.
>
> 15.
>
> Error data bypass and dead-letter queue capability for Transform and
> Sink
>
> zhangshenghang also proposed row-level error handling for Transform and
> Sink stages.
>
> When configured, SeaTunnel can collect failed records with useful context,
> including:
>
> -
>
> error stage
> -
>
> plugin type
> -
>
> plugin name
> -
>
> source table
> -
>
> error message
> -
>
> exception class
> -
>
> original data
> -
>
> occurrence time
>
> This allows jobs to bypass selected dirty or error records instead of
> always failing fast, while keeping the default behavior unchanged when the
> feature is not enabled.
>
> This is useful for long-running synchronization jobs where users need
> better fault tolerance and later inspection of bad data.
>
> 16.
>
> Improve cloud-native deployment experience
>
> zhangshenghang also suggested improving the cloud-native deployment
> experience for SeaTunnel 3.0.
>
> The proposal mentions Kubernetes and Helm support as important areas,
> including better guidance and behavior around:
>
> -
>
> deployment
> -
>
> restart
> -
>
> persistent checkpoint storage
> -
>
> configuration management
> -
>
> secret management
> -
>
> rolling upgrade
> -
>
> resource isolation
>
>
> 17.
>
> Clean up deprecated behavior carefully for 3.0
>
> zhangshenghang also suggested that SeaTunnel 3.0 is a good opportunity to
> clean up deprecated options and old APIs.
>
> At the same time, any incompatible change should be handled carefully,
> with clear migration documentation, examples, and release notes.
>
> Possible discussion groups
>
> Based on the above proposals, I think the roadmap discussion can be
> grouped into the following areas:
>
> 1.
>
> Zeta engine observability
>
>
> -
>
> STIP-21 / StainTrace
> -
>
> Prometheus metrics for IMap state stores
> -
>
> Real-time task metrics
> -
>
> Queue and backpressure visibility
> -
>
> REST API and Web UI exposure
>
>
> 2.
>
> Zeta state management and storage evolution
>
>
> -
>
> StateStore abstraction
> -
>
> IMap external storage compaction
> -
>
> IMap state metrics
> -
>
> Future storage backend evolution
>
>
> 3.
>
> Runtime and connector semantics
>
>
> -
>
> Engine-level timer flush for Sink connectors
> -
>
> JDBC adoption of timer flush
> -
>
> Checkpoint-gated file source lifecycle management
> -
>
> Table-level fault isolation
> -
>
> Row-level error bypass and dead-letter queue
>
>
> 4.
>
> AI for Data Engineering
>
>
> -
>
> AI-Native CLI
> -
>
> Natural language pipeline generation
> -
>
> Connector metadata reflection
> -
>
> Auto-fix loop
> -
>
> Security and sensitive value redaction
> -
>
> Future AI-assisted schema mapping, diagnosis, optimization, and
> autonomous data management
>
>
> 5.
>
> Edge collection
>
>
> -
>
> EdgeSocket ingress
> -
>
> SeaTunnel Edge Agent
> -
>
> Edge host to Zeta pipeline workflow
> -
>
> Authentication, local durability, and transport interoperability
>
>
> 6.
>
> Performance and internal data model
>
>
> -
>
> Apache Arrow as an optional internal data model
> -
>
> Columnar I/O
> -
>
> Vectorized execution
> -
>
> Lower JVM overhead
>
>
> 7.
>
> Build, release, and project structure
>
>
> -
>
> Independent seatunnel-shade project
> -
>
> Maven Central distribution of shade jars
> -
>
> Reduced build and release time
>
>
> 8.
>
> Cloud-native and compatibility
>
>
> -
>
> Kubernetes and Helm deployment experience
> -
>
> Persistent checkpoint storage
> -
>
> Secret management
> -
>
> Rolling upgrade
> -
>
> Resource isolation
> -
>
> Deprecated behavior cleanup
> -
>
> Migration documentation and release notes
>
> Next steps
>
> I suggest we continue to review these proposals and decide:
>
> 1.
>
> Which items should be included in the SeaTunnel 3.0 roadmap.
> 2.
>
> Which items should be marked as experimental.
> 3.
>
> Which items require an STIP or further design discussion.
> 4.
>
> Which items can be merged independently before the 3.0 release.
> 5.
>
> Which items need migration documentation, compatibility notes, or
> release notes.
>
> Again, this is only a summary of the current discussion thread. Please
> help correct or supplement anything I missed.
>
>
>
> Best Regards,
> David
> Linkedin: https://www.linkedin.com/in/davidzollo
>
>
>
> On Thu, May 14, 2026 at 2:59 PM zhangshenghang <sh...@apache.org>
> wrote:
>
>> Hi community,
>>
>> For the SeaTunnel 3.0 roadmap
>>
>> I have already submitted two related PRs, and besides those, I would like
>> to propose the following directions for 3.0:
>>
>> 1. Real-time observability for Zeta engine tasks
>>
>> This work adds the ability to monitor the runtime busyness level of Zeta
>> engine tasks. It introduces real-time task metrics for Source, Transform,
>> and Sink execution, including queue/backpressure-related information, and
>> exposes these metrics through REST API and Web UI. This can help users
>> understand whether a job is blocked by source reading, transform
>> processing, sink writing, or intermediate queues, which is very important
>> for production troubleshooting and performance tuning.
>>
>> 2. Error data bypass and dead-letter queue capability for Transform and
>> Sink
>>
>> This work introduces row-level error handling for Transform and Sink
>> stages. When configured, SeaTunnel can collect failed records with useful
>> context such as error stage, plugin type, plugin name, source table, error
>> message, exception class, original data, and occurrence time. This allows
>> jobs to bypass selected dirty/error records instead of always failing
>> fast,
>> while keeping the default behavior unchanged when the feature is not
>> enabled. This is useful for long-running synchronization jobs where users
>> need better fault tolerance and later inspection of bad data.
>>
>> 3. Improve cloud-native deployment experience
>>
>> Kubernetes and Helm support are important for new users. We should improve
>> guidance and behavior around deployment, restart, persistent checkpoint
>> storage, configuration management, secret management, rolling upgrade, and
>> resource isolation in cloud-native environments.
>>
>> 4. Clean up deprecated behavior carefully for 3.0
>>
>> Since 3.0 is a major version, it is a good opportunity to clean up
>> deprecated options and old APIs, but we should do this carefully. Any
>> incompatible change should have clear migration documentation, examples,
>> and release notes.
>>
>>
>> Regards,
>> Jast (Shenghang)
>>
>>
>> JeremyXin <73...@qq.com> 于2026年5月14日周四 14:27写道：
>>
>> > Hi everyone, I’d like to propose adding this PR to the SeaTunnel 3.0
>> > roadmap.
>> >
>> >
>> > - PR#10860
>> > - link: https://github.com/apache/seatunnel/pull/10860
>> >
>> >
>> > This PR improves Zeta Engine observability by exposing Prometheus
>> metrics
>> > for core engine state stores backed by Hazelcast IMap.
>> >
>> >
>> > It covers important internal stores such as running job information, job
>> > states, checkpoints, slot ownership and provides visibility into their
>> > local entry counts, backup entry counts, and heap cost.
>> >
>> >
>> > Including this in SeaTunnel 3.0 would make the engine easier to monitor
>> > and operate in production. It helps users detect state growth, data
>> skew,
>> > and memory pressure earlier, which improves troubleshooting efficiency,
>> > cluster capacity planning, and overall operational stability.
>> >
>> >
>> > Best regards,
>> > JeremyXin
>> >
>> >
>> > 原始邮件
>> >
>> >
>> > 发件人：zhiwei niu <zniu70696@gmail.com>
>> > 发件时间：2026年5月12日 10:27
>> > 收件人：dev <dev@seatunnel.apache.org>
>> > 主题：Re: [DISCUSS] Apache SeaTunnel 3.0 Roadmap
>> >
>> >
>> >
>> > Hi everyone,
>> >
>> >
>> >
>> As part of Roadmap 3.0 progress tracking, the STIP-24 PR links have been
>> >
>> >
>> updated to keep references consistent and easier to follow.
>> >
>> > Please use the updated PR links below:
>> >
>> >
>> >
>> [Feature][Zeta] STIP-24 Phase 1: Support lightweight EdgeSocket ingress
>> > for edge collector MVP
>> > - PR #10871(
>> > https://github.com/apache/seatunnel/pull/10871)
>> > —> PR#10878(
>> > https://github.com/apache/seatunnel/pull/10878)
>> >
>> >
>> >
>> [Feature][Zeta] STIP-24 Phase 2: Introduce standalone SeaTunnel Edge Agent
>> > MVP and dist integration
>> > - PR #10876(
>> > https://github.com/apache/seatunnel/pull/10876)
>> > — PR#10879(
>> > https://github.com/apache/seatunnel/pull/10879)
>> >
>> >
>> > zhiwei niu <zniu70696@gmail.com
>> > > 于2026年5月12日周二 01:31写道：
>> >
>> > > Hi everyone,
>> > >
>> >
>> >
>> > In addition to STIP-23, I’d like to propose the STIP-24 edge-collection
>> > > work for SeaTunnel 3.0.
>> > >
>> >
>> >
>> > - PR #10871 — [Feature][Zeta] STIP-24 Phase 1: Support lightweight
>> >
>> >
>> > EdgeSocket ingress for edge collector MVP
>> > > https://github.com/apache/seatunnel/pull/10871
>> > > What it adds:
>> >
>> >
>> > Introduces the engine-side ingress foundation for edge collection via
>> >
>> >
>> > EdgeSocket, including authenticated ingress, source-side protocol handling,
>> >
>> >
>> > and task-group address discovery support in Zeta.
>> > > Why 3.0:
>> >
>> >
>> > This is the foundational engine capability for remote edge data collection
>> >
>> >
>> > scenarios. It is backward-compatible and establishes the protocol/runtime
>> >
>> > path required for the edge-side agent.
>> > >
>> >
>> >
>> > - PR #10876 — [Feature][Zeta] STIP-24 Phase 2: Introduce standalone
>> >
>> >
>> > SeaTunnel Edge Agent MVP and dist integration
>> > > https://github.com/apache/seatunnel/pull/10876
>> > > What it adds:
>> >
>> >
>> > Introduces a standalone SeaTunnel Edge Agent runtime
>> >
>> >
>> > (core/transport/connector), local file/log/event inputs, SQLite WAL
>> >
>> >
>> > durability, EdgeSocket transport interop, and distribution integration.
>> > > Why 3.0:
>> >
>> >
>> > This delivers the first practical end-to-end edge collection workflow
>> >
>> >
>> > (edge host -> Zeta pipeline), which significantly expands SeaTunnel’s
>> >
>> >
>> > deployment topology and real-world ingestion coverage.
>> > >
>> >
>> >
>> > PR #10876 depends on #10871; both are under active review.
>> > >
>> > > Best regards,
>> > > zhiwei.niu
>> > >
>> > > xuepeng wang <wxuepeng5@gmail.com
>> > > 于2026年5月8日周五 14:03写道：
>> > >
>> > >> Hi team,
>> > >>
>> > >>
>> > >>
>> >
>> >
>> >> I'd like to add STIP-27 [1] to the SeaTunnel 3.0 Roadmap discussion.
>> > >>
>> >
>> >
>> >> This proposal introduces Apache Arrow as an optional internal data model
>> >
>> >
>> >> (not replacing the existing row model). It aims to improve performance for
>> >
>> >
>> >> columnar scenarios (Parquet/ORC, ClickHouse/Doris) and batch transforms
>> >
>> >
>> >> (projection/filter/cast), while keeping the current architecture stable.
>> > >>
>> >
>> >
>> >> Key benefits: better columnar I/O performance, vectorized execution,
>> > >> lower JVM overhead.
>> > >>
>> > >> [1]
>> > https://github.com/apache/seatunnel/issues/10806
>> > >> <https://github.com/apache/seatunnel/issues/10806>>>;
>> > >> Thoughts?
>> > >>
>> > >> Regards,
>> > >> xuepeng(github id:chl-wxp)
>> > >>
>> > >> David Zollo <davidzollo365@gmail.com
>> > > 于2026年4月30日周四 21:17写道：
>> > >>
>> > >> > Hi everyone,
>> > >> >
>> >
>> >
>> >> > First of all, thank you all for the fantastic proposals so far — the
>> >
>> >
>> >> > quality and breadth of ideas in this thread have been genuinely
>> >
>> >
>> >> > exciting to read. From StainTrace to the AI-Native CLI, from
>> >
>> >
>> >> > StateStore abstraction to table-level fault isolation, it's clear that
>> >
>> >
>> >> > this community has a bold and thoughtful vision for what SeaTunnel 3.0
>> > >> > can become.
>> > >> >
>> >
>> >
>> >> > Given that the Golden Week holiday is just around the corner, and many
>> >
>> >
>> >> > of our contributors are based in China or taking time off during this
>> >
>> >
>> >> > period, I'd like to extend the discussion window to May 15th. This
>> >
>> >
>> >> > gives everyone enough breathing room to enjoy the holiday, come back
>> >
>> >
>> >> > refreshed, and still have time to share their thoughts without feeling
>> > >> > rushed.
>> > >> >
>> > >> > Happy Labor Day to everyone!
>> > >> >
>> >
>> >
>> >> > We highly welcome everyone's suggestions for version 3.0! Feel free to
>> >
>> >
>> >> > leave a message if you have any ideas.
>> > >> >
>> > >> >
>> > >> >
>> > >> > Best Regards,
>> > >> > David
>> > >> > Linkedin:
>> > https://www.linkedin.com/in/davidzollo
>> > >> >
>> > <https://www.linkedin.com/in/davidzollo>> >>;
>> >
>> >
>> >> > On Wed, Apr 29, 2026 at 8:50 PM sean zhang <
>> > seanzhang92@gmail.com>
>> > >> wrote:
>> > >> > >
>> > >> > > Hi SeaTunnel community,
>> > >> > >
>> >
>> >
>> >> > > I'd like to propose including the AI-Native CLI capability in the
>> >
>> >
>> >> > > SeaTunnel 3.0 release, and share a broader vision for the "AI for Data
>> >
>> >
>> >> > > Engineering" direction. I believe this aligns well with the 3.0
>> > >> milestone
>> >
>> >
>> >> > > as a significant user-experience leap.
>> > >> > > This PR
>> > https://github.com/apache/seatunnel/pull/10789 introduces
>> >
>> >
>> >> > > `seatunnel-cli`, an AI-powered command-line tool that
>> >
>> >
>> >> > > generates production-ready pipeline configurations from natural
>> > >> language
>> > <
>> https://github.com/apache/seatunnel/pull/10789 introduces>> > > seatunnel-cli, an AI-powered command-line tool that>> > > generates production-ready pipeline configurations from natural>> language
>> >
>> > :
>> > >> > >
>> >
>> >
>> >> > > $ seatunnel "Sync the users table from MySQL to S3 in parquet
>> > >> format"
>> >
>> >
>> >> > > ✅ Generated config: mysql_users_to_s3.conf
>> > >> > >
>> >
>> >
>> >> > > Key capabilities in the current implementation:
>> > >> > >
>> >
>> >
>> >> > > - Multi-LLM provider support (Anthropic / OpenAI / AWS Bedrock)
>> >
>> >
>> >> > > with auto-detection
>> >
>> >
>> >> > > - Runtime connector metadata extraction via Java OptionRule
>> > >> reflection,
>> >
>> >
>> >> > > ensuring 100% accurate parameter knowledge for all 100+ connectors
>> >
>> >
>> >> > > - Source/Sink type-aware validation (Jdbc source options ≠ Jdbc sink
>> > >> > > options)
>> >
>> >
>> >> > > - Auto-fix loop: generated configs are validated and self-corrected
>> >
>> >
>> >> > > (up to 3 iterations)
>> >
>> >
>> >> > > - Security hardening: sensitive values (passwords, tokens, keys) are
>> >
>> >
>> >> > > automatically redacted to ${ENV_VAR} placeholders before
>> > >> persistence
>> >
>> >
>> >> > > and LLM transmission
>> >
>> >
>> >> > > - Interactive mode with session memory and one-shot mode for CI/CD
>> > >> > >
>> >
>> >
>> >> > > Future Roadmap — AI for Data Engineering
>> > >> > >
>> >
>> >
>> >> > > Looking beyond config generation, I see a multi-level roadmap where
>> >
>> >
>> >> > > AI becomes deeply integrated into SeaTunnel's core value proposition:
>> > >> > >
>> >
>> >
>> >> > > ### Level 2: AI Understands Data (Next phase)
>> > >> > >
>> >
>> >
>> >> > > - Intelligent Schema Mapping: AI reads source schema, infers target
>> >
>> >
>> >> > > schema, and auto-generates type conversions and DDL
>> >
>> >
>> >> > > - Natural Language Transforms: "mask phone numbers, merge first and
>> >
>> >
>> >> > > last name, filter last 7 days" → SQL Transform auto-generation
>> >
>> >
>> >> > > - Data Profiling: AI samples source data, detects anomalies (NULLs,
>> >
>> >
>> >> > > outliers, format violations), and suggests data quality transforms
>> > >> > >
>> >
>> >
>> >> > > ### Level 3: AI Operates Pipelines
>> > >> > >
>> >
>> >
>> >> > > - Intelligent Failure Diagnosis: AI analyzes error logs, pinpoints
>> >
>> >
>> >> > > root cause (connection refused → MySQL down vs firewall), suggests
>> > >> > fix
>> >
>> >
>> >> > > - Self-Healing Pipelines: auto-detect DDL changes in CDC sources,
>> >
>> >
>> >> > > auto-evolve sink schema, zero-downtime recovery
>> >
>> >
>> >> > > - Anomaly Detection: monitor data flow volume/latency, alert when
>> >
>> >
>> >> > > throughput drops 99% with correlated root cause
>> > >> > >
>> >
>> >
>> >> > > ### Level 4: AI Optimizes Everything
>> > >> > >
>> >
>> >
>> >> > > - Performance Tuning: AI analyzes bottlenecks (batch_size too small,
>> >
>> >
>> >> > > wrong file format, insufficient parallelism), suggests
>> > >> optimization
>> >
>> >
>> >> > > - Cost-Aware Planning: given N tables to sync, AI proposes pipeline
>> >
>> >
>> >> > > topology that balances cost vs latency vs resource usage
>> >
>> >
>> >> > > - Intelligent Scheduling: learn from historical patterns to optimize
>> >
>> >
>> >> > > sync frequency per table
>> > >> > >
>> >
>> >
>> >> > > ### Level 5: Autonomous Data Platform (Long-term vision)
>> > >> > >
>> >
>> >
>> >> > > - Conversational data infrastructure management: "onboard the
>> > >> payment
>> >
>> >
>> >> > > system to the data lake, ensure GDPR compliance" → AI
>> > >> auto-discovers
>> >
>> >
>> >> > > sources, identifies PII, generates pipelines with masking
>> > >> transforms,
>> >
>> >
>> >> > > deploys with monitoring and alerting
>> > >> > >
>> > >> > >
>> > >> > > Doyeon Kim <
>> > ekagnlek2@gmail.com> 于2026年4月29日周三 18:17写道：
>> > >> > >
>> > >> > > > Hi everyone,
>> > >> > > >
>> >
>> >
>> >> > > > I’d like to propose the following Zeta engine improvements for
>> > >> > SeaTunnel
>> > >> > > > 3.0.
>> > >> > > >
>> >
>> >
>> >> > > > 1. StateStore abstraction
>> > >> > > > PR:
>> > https://github.com/apache/seatunnel/pull/10812
>> > >> > > >
>> > <
>> https://github.com/apache/seatunnel/pull/10812>> > > >
>> >
>> > ;
>> >
>> >
>> >> > > > This introduces an initial StateStore abstraction layer to reduce
>> > >> > direct
>> >
>> >
>> >> > > > Hazelcast IMap dependencies in the engine. It makes different state
>> >
>> >
>> >> > > > responsibilities more explicit and prepares the engine for future
>> > >> > backend
>> > >> > > > evolution.
>> > >> > > >
>> >
>> >
>> >> > > > I think this is relevant for 3.0 because it is a foundational step
>> > >> > toward
>> >
>> >
>> >> > > > decoupling the engine from Hazelcast-specific APIs and enabling
>> > >> future
>> >
>> >
>> >> > > > improvements around state storage, node communication, and
>> > >> coordination
>> > >> > > > layers.
>> > >> > > >
>> >
>> >
>> >> > > > 2. IMap external storage compaction
>> > >> > > > PR:
>> > https://github.com/apache/seatunnel/pull/10399
>> > >> > > >
>> > <
>> https://github.com/apache/seatunnel/pull/10399>> > > >
>> >
>> > ;
>> >
>> >
>> >> > > > This introduces LSM-style write and compaction logic for IMap
>> > >> external
>> >
>> >
>> >> > > > storage. It helps control storage growth by flushing sorted
>> > >> immutable
>> > >> > files
>> >
>> >
>> >> > > > and compacting persisted files while keeping only the latest record
>> > >> per
>> > >> > > > key.
>> > >> > > >
>> >
>> >
>> >> > > > I think this should be considered for 3.0 because long-running Zeta
>> > >> > jobs
>> >
>> >
>> >> > > > may accumulate duplicated records and tombstones in external storage
>> > >> > over
>> >
>> >
>> >> > > > time. Compaction can improve stability and reduce the risk of
>> > >> unbounded
>> > >> > > > storage growth.
>> > >> > > >
>> > >> > > > Best Regards,
>> > >> > > > Doyeon
>> >
>> >> > > > (Github ID: dybyte)
>> > >> > > >
>> >
>> >
>> >> > > > On Wed, Apr 29, 2026 at 7:13 PM David Zollo <
>> > >> davidzollo365@gmail.com>
>> > >> > > > wrote:
>> > >> > > >
>> > >> > > > > Hi guys,
>> > >> > > > >
>> >
>> >
>> >> > > > > I'd like to propose including PR #10491 in the SeaTunnel 3.0
>> > >> roadmap,
>> >
>> >
>> >> > > > > here are the details with additional packaging time analysis:
>> > >> > > > >
>> >
>> >> > > > > ### PR link
>> > >> > > > >
>> > https://github.com/apache/seatunnel/pull/10491
>> > >> > > > >
>> > <
>> https://github.com/apache/seatunnel/pull/10491>> > > > >
>> >
>> > ;
>> >
>> >
>> >> > > > > StainTrace (PR #10491) is the concrete implementation of STIP-21,
>> >
>> >
>> >> > > > > delivering end-to-end data lineage tracing and performance
>> > >> analysis
>> >
>> >
>> >> > > > > capabilities for the Apache SeaTunnel Zeta engine. It injects
>> >
>> >
>> >> > > > > lightweight timestamp sampling points at 6 key pipeline stages —
>> >
>> >
>> >> > > > > SOURCE_EMIT → QUEUE_IN → QUEUE_OUT → TRANSFORM_IN → TRANSFORM_OUT
>> > >> →
>> >
>> >
>> >> > > > > SINK_WRITE_DONE — tracking each sampled row's complete journey
>> > >> from
>> >
>> >
>> >> > > > > source emission to sink write completion, enabling precise
>> > >> end-to-end
>> >
>> >
>> >> > > > > latency measurement and bottleneck identification.
>> > >> > > > >
>> >
>> >
>> >> > > > > The system is designed with production safety as a first-class
>> >
>> >
>> >> > > > > concern: it is disabled by default with very low overhead, and
>> > >> when
>> >
>> >
>> >> > > > > enabled it operates at a configurable sampling rate with a strict
>> >
>> >
>> >> > > > > per-worker per-second event budget, achieving approximately 0.1~1%
>> >
>> >
>> >> > > > > overhead at 1% sampling. Trace data is written locally as JSONL
>> > >> files
>> >
>> >
>> >> > > > > in OpenTelemetry OTLP JSON format with no external dependencies.
>> > >> An
>> >
>> >
>> >> > > > > accompanying offline analyzer reads these files to compute P95/P99
>> >
>> >
>> >> > > > > latency statistics, identify bottleneck stages, and generate an
>> > >> HTML
>> >
>> >
>> >> > > > > visualization report with Chinese/English language toggle support.
>> > >> > > > >
>> >
>> >
>> >> > > > > The value to SeaTunnel lies in closing a critical observability
>> > >> gap
>> > >> > at
>> >
>> >
>> >> > > > > the engine level. Before this feature, users facing latency
>> > >> spikes or
>> >
>> >
>> >> > > > > throughput bottlenecks had no structured way to pinpoint whether
>> > >> the
>> >
>> >
>> >> > > > > problem resided in queue backpressure, transform processing, or
>> > >> sink
>> >
>> >
>> >> > > > > write time — they could only rely on logs and guesswork.
>> > >> StainTrace
>> >
>> >
>> >> > > > > makes per-stage timing transparent and actionable, which is
>> > >> > especially
>> >
>> >
>> >> > > > > impactful for large-scale, multi-table, long-chain production
>> > >> > > > > pipelines.
>> > >> > > > >
>> >
>> >
>> >> > > > > The case for including it in the 3.0 roadmap is equally strong. As
>> > >> > the
>> >
>> >
>> >> > > > > implementation of STIP-21, StainTrace is a framework-level
>> >
>> >
>> >> > > > > observability capability — connector-agnostic and immediately
>> >
>> >
>> >> > > > > effective across all jobs once merged. One of SeaTunnel 3.0's
>> > >> central
>> >
>> >
>> >> > > > > goals is to become a more mature, production-ready data
>> > >> integration
>> >
>> >
>> >> > > > > engine, and observability is an indispensable foundation for that
>> >
>> >
>> >> > > > > goal. Establishing it at a major version boundary like 3.0
>> > >> delivers
>> >
>> >
>> >> > > > > far greater long-term value to users.
>> > >> > > > >
>> > >> > > > >
>> > >> > > > >
>> > >> > > > >
>> > >> > > > >
>> > >> > > > >
>> > >> > > > > Best Regards,
>> > >> > > > > David
>> > >> > > > > Linkedin:
>> > https://www.linkedin.com/in/davidzollo
>> > >> > > > >
>> > <
>> https://www.linkedin.com/in/davidzollo>> > > > >
>> >
>> > ;
>> >
>> >
>> >> > > > > On Wed, Apr 29, 2026 at 2:48 PM yi zeng <
>> > yzeng1618@gmail.com>
>> > >> wrote:
>> > >> > > > > >
>> >
>> >
>> >> > > > > > Hi everyone,
>> > >> > > > > >
>> >
>> >
>> >> > > > > > I’d like to propose including the following two PRs in the
>> > >> > *SeaTunnel
>> > >> > > > 3.0
>> >
>> >
>> >> > > > > > roadmap*. Both focus on strengthening *correctness, reliability,
>> > >> > and
>> >
>> >
>> >> > > > > > operational safety* for core multi-table and continuous
>> > >> ingestion
>> >
>> >
>> >> > > > > > scenarios, which I believe are key pillars for 3.0.
>> >
>> >
>> >> > > > > > ------------------------------
>> >
>> >
>> >> > > > > > PR #10600 — [Feature][Core] Support table-level fault isolation
>> > >> for
>> >
>> >
>> >> > > > > > multi-table synchronization
>> > >> > > > > >
>> > >> > > > > >
>> > https://github.com/apache/seatunnel/pull/10600
>> > >> > > > > >
>> > <
>> https://github.com/apache/seatunnel/pull/10600>> > > > > >
>> >
>> > ;
>> >
>> >
>> >> > > > > > *What it adds / solves*
>> > >> > > > > >
>> >
>> >
>> >> > > > > > This PR introduces *table-level fault isolation* for multi-table
>> >
>> >
>> >> > > > > > synchronization by adding a new failure policy:
>> > >> > > > > > code
>> > >> > > > > > 复制
>> > >> > > > > >
>> >
>> >> > > > > > env {
>> >
>> >
>> >> > > > > > multi_table {
>> >
>> >
>> >> > > > > > failure_policy = "CONTINUE_OTHER_TABLES"
>> >
>> >
>> >> > > > > > }
>> > >> > > > > > }
>> > >> > > > > >
>> >
>> >
>> >> > > > > > With this enabled:
>> > >> > > > > >
>> >
>> >
>> >> > > > > > - Failures are isolated at *table granularity*, instead of
>> > >> > failing
>> > >> > > > the
>> >
>> >
>> >> > > > > > entire job.
>> >
>> >
>> >> > > > > > - Healthy tables continue processing even if one table fails.
>> >
>> >
>> >> > > > > > - Failed tables are collected and reported with clear
>> > >> metadata
>> > >> > > > (table
>> >
>> >
>> >> > > > > > path, failure phase, plugin, error).
>> >
>> >
>> >> > > > > > - Batch jobs report failed tables in final results; streaming
>> > >> > jobs
>> >
>> >
>> >> > > > > > continue running with healthy tables.
>> > >> > > > > >
>> >
>> >
>> >> > > > > > This addresses a long-standing pain point where a single
>> > >> > problematic
>> > >> > > > > table
>> >
>> >
>> >> > > > > > can block progress for all others.
>> > >> > > > > >
>> >
>> >
>> >> > > > > > *Why it should be included in 3.0*
>> > >> > > > > >
>> >
>> >
>> >> > > > > > - *Improved fault tolerance model*: 3.0 aims to be more
>> > >> > resilient
>> > >> > > > and
>> >
>> >
>> >> > > > > > production-ready; table-level isolation is a foundational
>> > >> > capability
>> > >> > > > > for
>> >
>> >
>> >> > > > > > large-scale, multi-table pipelines.
>> >
>> >
>> >> > > > > > - *Backward-compatible by default*: The default behavior
>> > >> remains
>> >
>> >
>> >> > > > > > FAIL_FAST, so existing jobs are unaffected unless users
>> > >> > explicitly
>> > >> > > > opt
>> >
>> >
>> >> > > > > > in.
>> >
>> >
>> >> > > > > > - *Framework-level improvement*: This is not
>> > >> > connector-specific; it
>> >
>> >
>> >> > > > > > enhances the core execution and failure-handling semantics of
>> > >> > > > > SeaTunnel.
>> >
>> >
>> >> > > > > > - *Mature design & coverage*: Includes core logic changes,
>> > >> > runtime
>> >
>> >
>> >> > > > > > handling, reporting, and E2E tests, making it suitable for a
>> > >> > major
>> > >> > > > > version
>> >
>> >
>> >> > > > > > milestone.
>> > >> > > > > >
>> >
>> >
>> >> > > > > > ------------------------------
>> >
>> >
>> >> > > > > > PR #10563 — [Feature][Connector-V2] Add checkpoint-gated
>> > >> > > > post_sync_action
>> >
>> >
>> >> > > > > > and retention for file source continuous discovery
>> > >> > > > > >
>> > >> > > > > >
>> > https://github.com/apache/seatunnel/pull/10563
>> > >> > > > > >
>> > <
>> https://github.com/apache/seatunnel/pull/10563>> > > > > >
>> >
>> > ;
>> >
>> >
>> >> > > > > > *What it adds / solves*
>> > >> > > > > >
>> >
>> >
>> >> > > > > > This PR adds *checkpoint-gated source-side lifecycle management*
>> > >> > for
>> > >> > > > file
>> >
>> >
>> >> > > > > > source continuous discovery, including:
>> > >> > > > > >
>> >
>> >
>> >> > > > > > - post_sync_action = delete
>> >
>> >
>> >> > > > > > Delete source files only *after a checkpoint has completed*
>> >
>> >
>> >> > > > > > - post_sync_action = backup
>> >
>> >
>> >> > > > > > Move source files to a backup_path after checkpoint
>> > >> completion
>> >
>> >
>> >> > > > > > - Optional backup retention via retention_max_age
>> > >> > > > > >
>> >
>> >
>> >> > > > > > All operations are *checkpoint-aware*, ensuring source files are
>> > >> > never
>> >
>> >
>> >> > > > > > deleted or moved before the data is durably processed.
>> > >> > > > > >
>> >
>> >
>> >> > > > > > *Why it should be included in 3.0*
>> > >> > > > > >
>> >
>> >
>> >> > > > > > - *Correctness-first continuous ingestion*: Safe source
>> > >> cleanup
>> > >> > tied
>> > >> > > > > to
>> >
>> >
>> >> > > > > > checkpoint completion is essential for exactly-once and
>> > >> > recoverable
>> > >> > > > > file
>> >
>> >
>> >> > > > > > ingestion, which aligns strongly with 3.0 quality goals.
>> >
>> >
>> >> > > > > > - *Production-grade operational semantics*: Without
>> > >> > checkpoint-gated
>> >
>> >
>> >> > > > > > cleanup, continuous file ingestion is risky in real
>> > >> production
>> > >> > > > > environments.
>> >
>> >
>> >> > > > > > - *Extends Connector-V2 maturity*: This PR deepens the
>> > >> > operational
>> >
>> >
>> >> > > > > > completeness of Connector-V2, especially for file-based
>> > >> sources.
>> >
>> >
>> >> > > > > > - *Well-tested and documented*: Includes unit tests, E2E
>> > >> tests
>> > >> > > > (Local
>> > >> > > > > +
>> >
>> >
>> >> > > > > > HDFS), and documentation updates (EN + ZH).
>> > >> > > > > >
>> >
>> >
>> >> > > > > > ------------------------------
>> > >> > > > > >
>> >
>> >
>> >> > > > > > Both PRs are *opt-in*, backward-compatible, and focused on
>> > >> making
>> > >> > > > > SeaTunnel
>> >
>> >
>> >> > > > > > more robust, predictable, and safe in real-world deployments. I
>> > >> > believe
>> >
>> >
>> >> > > > > > they fit naturally into the SeaTunnel 3.0 scope, especially
>> > >> given
>> > >> > the
>> >
>> >
>> >> > > > > > emphasis on stability, correctness, and large-scale usage.
>> > >> > > > > >
>> >
>> >
>> >> > > > > > Happy to discuss details or answer questions.
>> > >> > > > > >
>> >
>> >
>> >> > > > > > Zhang Donghao <
>> > zhangdonghao@apache.org> 于2026年4月29日周三 14:38写道：
>> > >> > > > > >
>> >
>> >
>> >> > > > > > > Hi everyone,
>> > >> > > > > > >
>> >
>> >
>> >> > > > > > > I'd like to propose including PR #9993 in the SeaTunnel 3.0
>> > >> > roadmap,
>> > >> > > > > here
>> >
>> >
>> >> > > > > > > are the details with additional packaging time analysis:
>> > >> > > > > > >
>> >
>> >
>> >> > > > > > > ### PR link
>> >
>> >> > > > > > >
>> > https://github.com/apache/seatunnel/pull/9993
>> > >> > > > > > >
>> > <
>> https://github.com/apache/seatunnel/pull/9993>> > > > > > >
>> >
>> > ;
>> >
>> >
>> >> > > > > > > ### Short description
>> >
>> >
>> >> > > > > > > This PR extracts the shade modules into an independent
>> > >> > > > seatunnel-shade
>> >
>> >
>> >> > > > > > > project, enables uploading shade-jars to the central
>> > >> repository,
>> > >> > and
>> >
>> >
>> >> > > > > > > optimizes the overall packaging efficiency of SeaTunnel.
>> > >> > > > > > >
>> >
>> >
>> >> > > > > > > ### Packaging time analysis (current state before PR
>> > >> > optimization)
>> >
>> >
>> >> > > > > > > The current build time breakdown of individual shade modules
>> > >> is
>> > >> > as
>> >
>> >
>> >> > > > > > > follows, highlighting the inefficiency of the integrated build
>> > >> > > > > approach:
>> >
>> >
>> >> > > > > > > ```
>> >
>> >
>> >> > > > > > > [INFO] SeaTunnel : Shade : 3.0.0 ..........................
>> > >> > SUCCESS [
>> >
>> >
>> >> > > > > > > 12.661 s]
>> >
>> >
>> >> > > > > > > [INFO] SeaTunnel : Shade : Hazelcast 5.1-3.0.0 ............
>> > >> > SUCCESS [
>> >
>> >
>> >> > > > > > > 8.192 s]
>> >
>> >
>> >> > > > > > > [INFO] SeaTunnel : Shade : Hadoop3 Uber 3.1.4-3.0.0 .......
>> > >> > SUCCESS
>> > >> > > > > [02:29
>> >
>> >
>> >> > > > > > > min]
>> >
>> >
>> >> > > > > > > [INFO] SeaTunnel : Shade : Jackson 2.13.3-3.0.0 ...........
>> > >> > SUCCESS [
>> >
>> >
>> >> > > > > > > 3.289 s]
>> >
>> >
>> >> > > > > > > [INFO] SeaTunnel : Shade : Guava 27.0-jre-3.0.0 ...........
>> > >> > SUCCESS [
>> >
>> >
>> >> > > > > > > 2.960 s]
>> >
>> >
>> >> > > > > > > [INFO] SeaTunnel : Shade : Hadoop : AWS 3.1.4-3.0.0 .......
>> > >> > SUCCESS
>> > >> > > > > [05:05
>> >
>> >
>> >> > > > > > > min]
>> >
>> >
>> >> > > > > > > [INFO] SeaTunnel : Shade : Arrow 15.0.1-3.0.0 .............
>> > >> > SUCCESS [
>> >
>> >
>> >> > > > > > > 5.880 s]
>> >
>> >
>> >> > > > > > > [INFO] SeaTunnel : Shade : Hikari 4.0.3-3.0.0 .............
>> > >> > SUCCESS [
>> >
>> >
>> >> > > > > > > 1.085 s]
>> >
>> >
>> >> > > > > > > [INFO] SeaTunnel : Shade : Thrift-Service 1.0.0-3.0.0 .....
>> > >> > SUCCESS [
>> >
>> >
>> >> > > > > > > 1.572 s]
>> >
>> >
>> >> > > > > > > [INFO] SeaTunnel : Shade : Janino 3.0.11-3.0.0 ............
>> > >> > SUCCESS [
>> >
>> >
>> >> > > > > > > 1.235 s]
>> >
>> >
>> >> > > > > > > [INFO] SeaTunnel : Shade : Scala 2.13.11-3.0.0 ............
>> > >> > SUCCESS [
>> >
>> >
>> >> > > > > > > 6.586 s]
>> >
>> >
>> >> > > > > > > [INFO] SeaTunnel : Shade : Jetty 9.4.56.v20240826-3.0.0 ...
>> > >> > SUCCESS [
>> >
>> >
>> >> > > > > > > 2.421 s]
>> >
>> >
>> >> > > > > > > [INFO] SeaTunnel : Shade : Commons Lang3 3.18.0-3.0.0 .....
>> > >> > SUCCESS [
>> >
>> >
>> >> > > > > > > 2.044 s]
>> >
>> >
>> >> > > > > > > ```
>> >
>> >
>> >> > > > > > > Key observations from the time breakdown:
>> >
>> >
>> >> > > > > > > 1. **Long-tail modules dominate build time**: Two modules
>> > >> > account for
>> > >> > > > > the
>> >
>> >
>> >> > > > > > > vast majority of shade build time — `Hadoop3 Uber` (2 minutes
>> > >> 29
>> > >> > > > > seconds)
>> >
>> >
>> >> > > > > > > and `Hadoop : AWS` (5 minutes 5 seconds), totaling ~7.5
>> > >> minutes,
>> > >> > > > while
>> >
>> >
>> >> > > > > > > other modules take only a few seconds each.
>> >
>> >
>> >> > > > > > > 2. **Integrated build waste**: In the current structure, these
>> >
>> >
>> >> > > > > > > time-consuming shade modules are built alongside core modules
>> > >> > every
>> > >> > > > > time
>> >
>> >
>> >> > > > > > > SeaTunnel is packaged, even when there are no changes to shade
>> >
>> >
>> >> > > > > > > dependencies. This results in repeated long build cycles for
>> >
>> >> > > > > developers and
>> >
>> >
>> >> > > > > > > release workflows.
>> >
>> >
>> >> > > > > > > 3. **Total shade build time**: The cumulative build time of
>> > >> all
>> > >> > shade
>> >
>> >
>> >> > > > > > > modules is approximately 8 minutes (excluding
>> > >> > overlapping/parallel
>> > >> > > > > build
>> >
>> >
>> >> > > > > > > optimizations), which is a significant portion of the overall
>> > >> > > > SeaTunnel
>> >
>> >
>> >> > > > > > > packaging process.
>> > >> > > > > > >
>> >
>> >
>> >> > > > > > > ### Reason for including in 3.0
>> >
>> >
>> >> > > > > > > 1. **Build efficiency optimization (quantifiable gains)**:
>> >
>> >
>> >> > > > > > > - By isolating shade modules into an independent project
>> > >> and
>> > >> > > > > uploading
>> >
>> >
>> >> > > > > > > shade-jars to the central repository, we eliminate the need to
>> > >> > > > rebuild
>> >
>> >
>> >> > > > > > > these modules in every SeaTunnel packaging cycle. For daily
>> > >> > > > > development,
>> >
>> >
>> >> > > > > > > this reduces the total build time by ~8 minutes per build (for
>> > >> > > > > developers
>> >
>> >
>> >> > > > > > > not modifying shade code).
>> >
>> >
>> >> > > > > > > - For the 3.0 release process, we can pre-build and publish
>> > >> > > > > shade-jars
>> >
>> >
>> >> > > > > > > to the central repository once, then reference them as
>> > >> > dependencies —
>> >
>> >
>> >> > > > > > > cutting the release packaging time by a substantial margin and
>> > >> > > > > reducing the
>> >
>> >
>> >> > > > > > > risk of build failures caused by long-running shade module
>> > >> > builds.
>> >
>> >
>> >> > > > > > > 2. **Dependency management enhancement**: Uploading shade-jars
>> > >> > to the
>> >
>> >
>> >> > > > > > > central repository standardizes dependency distribution,
>> > >> avoiding
>> > >> > > > > conflicts
>> >
>> >
>> >> > > > > > > caused by internal shade module duplication across
>> > >> subprojects,
>> > >> > and
>> >
>> >
>> >> > > > > > > improving the stability of the 3.0 version's dependency
>> > >> system.
>> >
>> >
>> >> > > > > > > 3. **Project structure rationalization**: Isolating shade
>> > >> modules
>> > >> > > > > aligns
>> >
>> >
>> >> > > > > > > with the modular design concept of 3.0, making the project
>> > >> > structure
>> >
>> >
>> >> > > > > > > clearer, easier to maintain, and facilitating future
>> > >> expansion of
>> >
>> >
>> >> > > > > > > shade-related capabilities (e.g., adding new shade modules for
>> > >> > > > > additional
>> >
>> >
>> >> > > > > > > dependencies).
>> > >> > > > > > >
>> >
>> >
>> >> > > > > > > This optimization directly addresses the identified packaging
>> > >> > > > > bottlenecks
>> >
>> >
>> >> > > > > > > (long-tail shade modules) and delivers measurable time savings
>> > >> > for
>> > >> > > > both
>> >
>> >
>> >> > > > > > > development and release workflows — a critical improvement for
>> > >> > the
>> > >> > > > > smooth
>> >
>> >
>> >> > > > > > > delivery and maintainability of SeaTunnel 3.0.
>> > >> > > > > > >
>> >
>> >
>> >> > > > > > > Best regards,
>> >
>> >
>> >> > > > > > > zhangdonghao
>> > >> > > > > > >
>> > >> > > > >
>> > >> > > >
>> > >> >
>> > >>
>> > >
>>
>
Re: [DISCUSS] Apache SeaTunnel 3.0 Roadmap
Posted by David Zollo <da...@gmail.com>.
Hi guys,
Thanks for all your effort.Now we have merged all the feature PRs as
planned in the 3.0.0 Roadmap.
and there are 7 bug fix PRs left to be merged before the 3.0.0 release:
- https://github.com/apache/seatunnel/pull/11503 @nielifeng
- https://github.com/apache/seatunnel/pull/11757 @waterWang
- https://github.com/apache/seatunnel/pull/11458 @zhangshenghang
- https://github.com/apache/seatunnel/pull/11727 @abdessalems
- https://github.com/apache/seatunnel/pull/11077 @hesam-oxe
- https://github.com/apache/seatunnel/pull/11569 @DanielLeens
- https://github.com/apache/seatunnel/pull/11271 @hutiefang76
Best Regards,
David
Linkedin: https://www.linkedin.com/in/davidzollo
On Wed, Jul 22, 2026 at 10:56 PM David Zollo <da...@gmail.com>
wrote:
> Hi guys,
>
> Guys, please leave a message if you have any PR that needs to be merged in
> the 3.0.0 version.
>
> Firstly, thanks for your hard work and making great progress. Now we only
> have 3 PRs left to update and review in the 3.0.0 Roadmap list[1].
> Below is the list of remaining PRs pending merge in the Roadmap:
>
> 1. [#10563](https://github.com/apache/seatunnel/pull/10563)
> [Feature][Connector-V2] Add checkpoint-gated post_sync_action and retention
> for file source continuous discovery @yzeng1618
> 2. [#9993](https://github.com/apache/seatunnel/pull/9993)
> [Feature][shade]Refactor the seatunnel-shade module @hawk9821
> 3. [#10306](https://github.com/apache/seatunnel/pull/10306)
> [Feature][Core] Error Data Bypass Capability for Transform & Sink (Dead
> Letter Queue) @zhangshenghang
>
>
> Again, please leave a message if you have any PR need to be merged in
> 3.0.0 version.
>
>
> [1]. https://github.com/apache/seatunnel/issues/11021
>
>
>
> Best Regards,
> David
> Linkedin: https://www.linkedin.com/in/davidzollo
>
>
>
> On Wed, Jul 15, 2026 at 12:36 PM David Zollo <da...@gmail.com>
> wrote:
>
>> Hi guys,
>>
>> Thanks for your hard work and make a great progress. Now we only have
>> four PRs left to be reviewed and merged.
>> Below is the list of remaining PRs pending merge in the Roadmap:
>>
>> 1. [#10563](https://github.com/apache/seatunnel/pull/10563)
>> [Feature][Connector-V2] Add checkpoint-gated post_sync_action and retention
>> for file source continuous discovery @yzeng1618
>> 2. [#9993](https://github.com/apache/seatunnel/pull/9993)
>> [Feature][shade]Refactor the seatunnel-shade module. @hawk9821
>> 3. [#10879](https://github.com/apache/seatunnel/pull/10879)
>> [Feature][Zeta] STIP-24 Phase 2: Introduce standalone SeaTunnel Edge Agent
>> MVP and dist integration @nzw921rx
>> 4. [#10306](https://github.com/apache/seatunnel/pull/10306)
>> [Feature][Core] Error Data Bypass Capability for Transform & Sink (Dead
>> Letter Queue) @zhangshenghang
>>
>>
>> Guys, please help review the PR for the 3.0 Roadmap when you have time.
>>
>>
>> Best Regards,
>> David
>> Linkedin: https://www.linkedin.com/in/davidzollo
>>
>>
>>
>>
>> On Thu, Jul 9, 2026 at 10:56 PM zhiwei niu <zn...@gmail.com> wrote:
>>
>>> Hi community,
>>>
>>> I would like to propose including STIP-28 in the SeaTunnel 3.0 roadmap.
>>>
>>> STIP-28 enhances SeaTunnel's OptionRule framework from basic
>>> presence/equality validation to richer declarative configuration
>>> validation. This allows connector and transform configuration constraints
>>> to be expressed as structured metadata instead of being hidden in
>>> scattered
>>> runtime if/throw logic.
>>>
>>> Related PRs:
>>>
>>> - Main PR: apache/seatunnel#10977
>>> https://github.com/apache/seatunnel/pull/10977
>>>
>>> Completes the rich OptionRule validation framework.
>>>
>>> - Follow-up PR: apache/seatunnel#11228
>>> https://github.com/apache/seatunnel/pull/11228
>>>
>>> Preserves extension validation error context.
>>>
>>> - Follow-up PR: apache/seatunnel#11121
>>> https://github.com/apache/seatunnel/pull/11121
>>>
>>> Fixes validation error aggregation.
>>>
>>> - Follow-up PR: apache/seatunnel#11010
>>> https://github.com/apache/seatunnel/pull/11010
>>>
>>> Adds Map-type condition validation.
>>>
>>> - Follow-up PR: apache/seatunnel#11022
>>> https://github.com/apache/seatunnel/pull/11022
>>>
>>> Supports combining structural and value rules.
>>>
>>> - Follow-up PR: apache/seatunnel#11048
>>> https://github.com/apache/seatunnel/pull/11048
>>>
>>> Adds custom extension validation support.
>>>
>>> Why this should be included in 3.0:
>>>
>>> - It makes SeaTunnel's configuration contract clearer and more explicit.
>>> - Users can catch more configuration errors earlier through --check.
>>> - REST API, Web UI, CLI metadata export, and future AI-assisted config
>>> generation can consume richer validation metadata.
>>> - Connector and transform developers can reduce duplicated imperative
>>> validation logic.
>>> - It provides a stronger foundation for future configuration generation,
>>> validation, explanation, and repair capabilities.
>>>
>>> STIP-28 framework capabilities have already been merged, and some
>>> connectors have already completed migration. Therefore, I believe STIP-28
>>> is ready to be included in the SeaTunnel 3.0 roadmap.
>>>
>>> Best Regards,
>>> zhiwei.niu
>>>
>>> David Zollo <da...@gmail.com> 于2026年7月9日周四 21:19写道：
>>>
>>> > Hi guys,
>>> >
>>> > Thanks for your hard work and make a great progress. Now we only have
>>> the
>>> > following PRs left to be reviewed and merged.
>>> > Below is the list of remaining PRs pending merge in the Roadmap:
>>> >
>>> > 1. [#10563](https://github.com/apache/seatunnel/pull/10563)
>>> > [Feature][Connector-V2] Add checkpoint-gated post_sync_action and
>>> retention
>>> > for file source continuous discovery @yzeng1618 PR conflicts and need
>>> to
>>> > solve
>>> > 2. [#9993](https://github.com/apache/seatunnel/pull/9993)
>>> > [Feature][shade]Refactor the seatunnel-shade module. @hawk9821 PR
>>> > conflicts and need to solve
>>> > 3. [#10879](https://github.com/apache/seatunnel/pull/10879)
>>> > [Feature][Zeta]
>>> > STIP-24 Phase 2: Introduce standalone SeaTunnel Edge Agent MVP and dist
>>> > integration @nzw921rx
>>> > 4. [#10800](https://github.com/apache/seatunnel/pull/10800)
>>> > [Feature][Zeta]
>>> > STIP-23 Phase 1: Add engine timer flush core flow and FlushSignal
>>> handling
>>> > @nzw921rx
>>> > 5. [#10801](https://github.com/apache/seatunnel/pull/10801)
>>> > [Feature][Connector-V2] STIP-23 Phase 2: JDBC connector adoption for
>>> engine
>>> > timer flush @nzw921rx
>>> > 6. [#10306](https://github.com/apache/seatunnel/pull/10306)
>>> > [Feature][Core]
>>> > Error Data Bypass Capability for Transform & Sink (Dead Letter Queue)
>>> > @zhangshenghang
>>> >
>>> >
>>> > Guys, please help review the PR for the 3.0 Roadmap when you have time.
>>> >
>>> >
>>> > Best Regards,
>>> > David
>>> > Linkedin: https://www.linkedin.com/in/davidzollo
>>> >
>>> >
>>> >
>>> > On Wed, Jul 1, 2026 at 2:54 PM David Zollo <da...@gmail.com>
>>> > wrote:
>>> >
>>> > > Hi guys,
>>> > >
>>> > > I created an issue for the 3.0 roadmap to track progress[1]. Given
>>> that
>>> > > most PRs have been merged so far, we plan to finish merging the
>>> remaining
>>> > > PRs by the end of July and then kick off the release process!
>>> > >
>>> > > 1. https://github.com/apache/seatunnel/issues/11021
>>> > >
>>> > >
>>> > >
>>> > >
>>> > > Best Regards,
>>> > > David
>>> > > Linkedin: https://www.linkedin.com/in/davidzollo
>>> > >
>>> > >
>>> > >
>>> > > On Sat, May 16, 2026 at 12:32 PM David Zollo <
>>> davidzollo365@gmail.com>
>>> > > wrote:
>>> > >
>>> > >> Hi SeaTunnel community,
>>> > >>
>>> > >> Thanks everyone for the active discussion around the Apache
>>> SeaTunnel
>>> > 3.0
>>> > >> roadmap.
>>> > >>
>>> > >> To make the follow-up review easier, I summarized the proposals
>>> > collected
>>> > >> in this thread so far. This is only a discussion summary, not a
>>> final
>>> > >> decision. Please feel free to correct me if I missed anything or
>>> > >> misunderstood any proposal.
>>> > >>
>>> > >> 1.
>>> > >>
>>> > >> SeaTunnel 3.0 roadmap discussion
>>> > >>
>>> > >> The discussion was started to collect proposals for Apache
>>> SeaTunnel 3.0
>>> > >> after PR #10510, which updated the version to 3.0.0, was merged
>>> into the
>>> > >> dev branch on March 11, 2026.
>>> > >>
>>> > >> Several already-merged PRs were mentioned as possible parts of the
>>> 3.0
>>> > >> scope:
>>> > >>
>>> > >> -
>>> > >>
>>> > >> PR #10586: [Feature][Zeta] Add datasource spi
>>> > >> https://github.com/apache/seatunnel/pull/10586
>>> > >> -
>>> > >>
>>> > >> PR #10585: [Feature][Zeta] Add /option-rules REST API for runtime
>>> > >> connector metadata
>>> > >> https://github.com/apache/seatunnel/pull/10585
>>> > >> -
>>> > >>
>>> > >> PR #10489: [Feature][Transform] Add Zeta SQL UDF context and
>>> > >> lifecycle hooks with e2e/docs updates
>>> > >> https://github.com/apache/seatunnel/pull/10489
>>> > >>
>>> > >> The purpose of this thread is to collect PRs, features,
>>> improvements,
>>> > and
>>> > >> ideas that the community believes should be considered for SeaTunnel
>>> > 3.0.
>>> > >>
>>> > >> 2.
>>> > >>
>>> > >> STIP-23: Engine-level timer flush for Sink connectors
>>> > >>
>>> > >> zhiwei niu proposed including STIP-23 in the SeaTunnel 3.0 roadmap.
>>> The
>>> > >> proposal focuses on engine-level timer flush support for Sink
>>> > connectors.
>>> > >>
>>> > >> Related PRs:
>>> > >>
>>> > >> -
>>> > >>
>>> > >> PR #10800: [Feature][Zeta] Phase 1: Engine timer flush core
>>> > >> https://github.com/apache/seatunnel/pull/10800
>>> > >>
>>> > >> This PR adds sink.flush.interval and introduces a FlushSignal
>>> > >> control-plane path through the Zeta engine. It allows sinks to
>>> flush
>>> > based
>>> > >> on a wall-clock interval without each sink managing its own timer
>>> > thread.
>>> > >> -
>>> > >>
>>> > >> PR #10801: [Feature][Connector-V2] Phase 2: JDBC connector
>>> adoption
>>> > >> https://github.com/apache/seatunnel/pull/10801
>>> > >>
>>> > >> This PR wires registerFlushAction into the JDBC non-XA sink
>>> writer
>>> > >> through a new enable_timer_flush option. It also includes
>>> multi-table
>>> > >> aggregated flush and MySQL CDC E2E tests.
>>> > >> -
>>> > >>
>>> > >> PR #10802: [Docs][Connector-V2] Phase 3: Timer flush feature
>>> matrix
>>> > >> https://github.com/apache/seatunnel/pull/10802
>>> > >>
>>> > >> This PR adds a timer flush support marker to sink connector
>>> > >> documentation and documents the sink.flush.interval and
>>> > registerFlushAction
>>> > >> contract.
>>> > >>
>>> > >> The main value is to make end-to-end latency more predictable for
>>> > >> buffered sinks, especially in idle or low-throughput streaming jobs.
>>> > >>
>>> > >> 3.
>>> > >>
>>> > >> Independent seatunnel-shade project and packaging efficiency
>>> > >> optimization
>>> > >>
>>> > >> Zhang Donghao proposed including PR #9993 in the SeaTunnel 3.0
>>> roadmap.
>>> > >>
>>> > >> -
>>> > >>
>>> > >> PR #9993:
>>> > >> https://github.com/apache/seatunnel/pull/9993
>>> > >>
>>> > >> This PR extracts the shade modules into an independent
>>> seatunnel-shade
>>> > >> project, supports uploading shade jars to Maven Central, and
>>> optimizes
>>> > the
>>> > >> overall packaging efficiency of SeaTunnel.
>>> > >>
>>> > >> The proposal points out that current shade module builds, especially
>>> > >> Hadoop3 Uber and Hadoop AWS, add significant build time. By
>>> publishing
>>> > >> shade jars separately and reusing them as dependencies, SeaTunnel
>>> can
>>> > >> reduce repeated build cost and improve release stability.
>>> > >>
>>> > >> 4.
>>> > >>
>>> > >> Table-level fault isolation for multi-table synchronization
>>> > >>
>>> > >> yi zeng proposed including PR #10600 in the SeaTunnel 3.0 roadmap.
>>> > >>
>>> > >> -
>>> > >>
>>> > >> PR #10600: [Feature][Core] Support table-level fault isolation
>>> for
>>> > >> multi-table synchronization
>>> > >> https://github.com/apache/seatunnel/pull/10600
>>> > >>
>>> > >> This PR introduces table-level fault isolation for multi-table
>>> > >> synchronization through a new failure policy:
>>> > >>
>>> > >> env {
>>> > >> multi_table {
>>> > >> failure_policy = "CONTINUE_OTHER_TABLES"
>>> > >> }
>>> > >> }
>>> > >>
>>> > >> With this enabled, failures can be isolated at table granularity.
>>> > Healthy
>>> > >> tables can continue processing even if one table fails. Failed
>>> tables
>>> > are
>>> > >> collected and reported with metadata such as table path, failure
>>> phase,
>>> > >> plugin, and error.
>>> > >>
>>> > >> The proposal is backward-compatible by default because the existing
>>> > >> FAIL_FAST behavior remains unchanged unless users explicitly enable
>>> the
>>> > new
>>> > >> policy.
>>> > >>
>>> > >> 5.
>>> > >>
>>> > >> Checkpoint-gated post_sync_action and retention for file source
>>> > >> continuous discovery
>>> > >>
>>> > >> yi zeng also proposed including PR #10563.
>>> > >>
>>> > >> -
>>> > >>
>>> > >> PR #10563: [Feature][Connector-V2] Add checkpoint-gated
>>> > >> post_sync_action and retention for file source continuous
>>> discovery
>>> > >> https://github.com/apache/seatunnel/pull/10563
>>> > >>
>>> > >> This PR adds checkpoint-aware source-side lifecycle management for
>>> file
>>> > >> source continuous discovery, including:
>>> > >>
>>> > >> -
>>> > >>
>>> > >> post_sync_action = delete
>>> > >> -
>>> > >>
>>> > >> post_sync_action = backup
>>> > >> -
>>> > >>
>>> > >> retention_max_age
>>> > >>
>>> > >> The key idea is that source files should only be deleted or moved
>>> after
>>> > a
>>> > >> checkpoint has completed, so that continuous file ingestion remains
>>> safe
>>> > >> and recoverable in production environments.
>>> > >>
>>> > >> 6.
>>> > >>
>>> > >> STIP-21 / StainTrace: End-to-end data lineage tracing and
>>> performance
>>> > >> analysis for Zeta
>>> > >>
>>> > >> David proposed including PR #10491 in the SeaTunnel 3.0 roadmap.
>>> > >>
>>> > >> -
>>> > >>
>>> > >> PR #10491:
>>> > >> https://github.com/apache/seatunnel/pull/10491
>>> > >>
>>> > >> StainTrace is the implementation of STIP-21. It provides end-to-end
>>> data
>>> > >> lineage tracing and performance analysis for the Zeta engine.
>>> > >>
>>> > >> It adds lightweight timestamp sampling points at six key pipeline
>>> > stages:
>>> > >>
>>> > >> -
>>> > >>
>>> > >> SOURCE_EMIT
>>> > >> -
>>> > >>
>>> > >> QUEUE_IN
>>> > >> -
>>> > >>
>>> > >> QUEUE_OUT
>>> > >> -
>>> > >>
>>> > >> TRANSFORM_IN
>>> > >> -
>>> > >>
>>> > >> TRANSFORM_OUT
>>> > >> -
>>> > >>
>>> > >> SINK_WRITE_DONE
>>> > >>
>>> > >> The goal is to track the full journey of sampled rows from source
>>> > >> emission to sink write completion, making it easier to measure
>>> > end-to-end
>>> > >> latency and identify bottlenecks.
>>> > >>
>>> > >> The proposal emphasizes production safety: the feature is disabled
>>> by
>>> > >> default, supports configurable sampling, has a per-worker event
>>> budget,
>>> > >> writes trace data locally as JSONL in OpenTelemetry OTLP JSON
>>> format,
>>> > and
>>> > >> includes an offline analyzer for P95/P99 latency and HTML
>>> visualization
>>> > >> reports.
>>> > >>
>>> > >> 7.
>>> > >>
>>> > >> StateStore abstraction for Zeta engine
>>> > >>
>>> > >> Doyeon Kim proposed including PR #10812 in the SeaTunnel 3.0
>>> roadmap.
>>> > >>
>>> > >> -
>>> > >>
>>> > >> PR #10812:
>>> > >> https://github.com/apache/seatunnel/pull/10812
>>> > >>
>>> > >> This PR introduces an initial StateStore abstraction layer to reduce
>>> > >> direct Hazelcast IMap dependencies in the Zeta engine. It makes
>>> state
>>> > >> responsibilities more explicit and prepares the engine for future
>>> > evolution
>>> > >> around state storage, node communication, and coordination layers.
>>> > >>
>>> > >> 8.
>>> > >>
>>> > >> IMap external storage compaction
>>> > >>
>>> > >> Doyeon Kim also proposed including PR #10399.
>>> > >>
>>> > >> -
>>> > >>
>>> > >> PR #10399:
>>> > >> https://github.com/apache/seatunnel/pull/10399
>>> > >>
>>> > >> This PR introduces LSM-style write and compaction logic for IMap
>>> > external
>>> > >> storage. It helps control storage growth by flushing sorted
>>> immutable
>>> > files
>>> > >> and compacting persisted files while keeping only the latest record
>>> per
>>> > key.
>>> > >>
>>> > >> The proposal is intended to improve stability for long-running Zeta
>>> jobs
>>> > >> by reducing duplicated records and tombstones in external storage.
>>> > >>
>>> > >> 9.
>>> > >>
>>> > >> AI-Native CLI and AI for Data Engineering
>>> > >>
>>> > >> sean zhang proposed including the AI-Native CLI capability in
>>> SeaTunnel
>>> > >> 3.0.
>>> > >>
>>> > >> -
>>> > >>
>>> > >> PR #10789:
>>> > >> https://github.com/apache/seatunnel/pull/10789
>>> > >>
>>> > >> This PR introduces seatunnel-cli, an AI-powered command-line tool
>>> that
>>> > >> can generate production-ready pipeline configurations from natural
>>> > language.
>>> > >>
>>> > >> Current capabilities include:
>>> > >>
>>> > >> -
>>> > >>
>>> > >> Multi-LLM provider support, including Anthropic, OpenAI, and AWS
>>> > >> Bedrock
>>> > >> -
>>> > >>
>>> > >> Runtime connector metadata extraction via Java OptionRule
>>> reflection
>>> > >> -
>>> > >>
>>> > >> Source/Sink type-aware validation
>>> > >> -
>>> > >>
>>> > >> Auto-fix loop for generated configurations
>>> > >> -
>>> > >>
>>> > >> Sensitive value redaction before persistence and LLM transmission
>>> > >> -
>>> > >>
>>> > >> Interactive mode with session memory
>>> > >> -
>>> > >>
>>> > >> One-shot mode for CI/CD
>>> > >>
>>> > >> The proposal also describes a broader “AI for Data Engineering”
>>> roadmap,
>>> > >> including intelligent schema mapping, natural language transforms,
>>> data
>>> > >> profiling, failure diagnosis, self-healing pipelines, anomaly
>>> detection,
>>> > >> performance tuning, cost-aware planning, intelligent scheduling, and
>>> > >> eventually autonomous data infrastructure management.
>>> > >>
>>> > >> 10.
>>> > >>
>>> > >> Extension of the roadmap discussion window
>>> > >>
>>> > >> David proposed extending the SeaTunnel 3.0 roadmap discussion
>>> window to
>>> > >> May 15, 2026, considering the upcoming holiday period and to give
>>> > >> contributors more time to share ideas.
>>> > >>
>>> > >> This message also recognized the quality and breadth of the
>>> proposals so
>>> > >> far, including StainTrace, AI-Native CLI, StateStore abstraction,
>>> and
>>> > >> table-level fault isolation.
>>> > >>
>>> > >> 11.
>>> > >>
>>> > >> STIP-27: Apache Arrow as an optional internal data model
>>> > >>
>>> > >> xuepeng wang proposed adding STIP-27 to the SeaTunnel 3.0 roadmap
>>> > >> discussion.
>>> > >>
>>> > >> -
>>> > >>
>>> > >> Issue #10806:
>>> > >> https://github.com/apache/seatunnel/issues/10806
>>> > >>
>>> > >> This proposal introduces Apache Arrow as an optional internal data
>>> > model,
>>> > >> not as a replacement for the existing row model.
>>> > >>
>>> > >> The goal is to improve performance for columnar scenarios such as
>>> > >> Parquet, ORC, ClickHouse, Doris, and batch transforms such as
>>> > projection,
>>> > >> filter, and cast.
>>> > >>
>>> > >> Expected benefits include better columnar I/O performance,
>>> vectorized
>>> > >> execution, and lower JVM overhead.
>>> > >>
>>> > >> 12.
>>> > >>
>>> > >> STIP-24: Edge collection, EdgeSocket ingress, and SeaTunnel Edge
>>> > Agent
>>> > >>
>>> > >> zhiwei niu proposed adding STIP-24 edge-collection work to the
>>> SeaTunnel
>>> > >> 3.0 roadmap.
>>> > >>
>>> > >> The original PRs were:
>>> > >>
>>> > >> -
>>> > >>
>>> > >> PR #10871: [Feature][Zeta] STIP-24 Phase 1: Support lightweight
>>> > >> EdgeSocket ingress for edge collector MVP
>>> > >> https://github.com/apache/seatunnel/pull/10871
>>> > >> -
>>> > >>
>>> > >> PR #10876: [Feature][Zeta] STIP-24 Phase 2: Introduce standalone
>>> > >> SeaTunnel Edge Agent MVP and dist integration
>>> > >> https://github.com/apache/seatunnel/pull/10876
>>> > >>
>>> > >> Phase 1 introduces the engine-side ingress foundation for edge
>>> > collection
>>> > >> through EdgeSocket, including authenticated ingress, source-side
>>> > protocol
>>> > >> handling, and task-group address discovery support in Zeta.
>>> > >>
>>> > >> Phase 2 introduces a standalone SeaTunnel Edge Agent runtime,
>>> including
>>> > >> core, transport, connector, local file/log/event inputs, SQLite WAL
>>> > >> durability, EdgeSocket transport interoperability, and distribution
>>> > >> integration.
>>> > >>
>>> > >> The goal is to support an end-to-end edge collection workflow:
>>> > >>
>>> > >> edge host -> Zeta pipeline
>>> > >>
>>> > >> zhiwei later updated the PR references for STIP-24:
>>> > >>
>>> > >> -
>>> > >>
>>> > >> PR #10871 was updated to PR #10878
>>> > >> https://github.com/apache/seatunnel/pull/10878
>>> > >> -
>>> > >>
>>> > >> PR #10876 was updated to PR #10879
>>> > >> https://github.com/apache/seatunnel/pull/10879
>>> > >>
>>> > >>
>>> > >> 13.
>>> > >>
>>> > >> Prometheus metrics for Zeta Engine IMap state stores
>>> > >>
>>> > >> JeremyXin proposed including PR #10860 in the SeaTunnel 3.0 roadmap.
>>> > >>
>>> > >> -
>>> > >>
>>> > >> PR #10860:
>>> > >> https://github.com/apache/seatunnel/pull/10860
>>> > >>
>>> > >> This PR improves Zeta Engine observability by exposing Prometheus
>>> > metrics
>>> > >> for core engine state stores backed by Hazelcast IMap.
>>> > >>
>>> > >> It covers internal stores such as running job information, job
>>> states,
>>> > >> checkpoints, and slot ownership. It exposes metrics such as local
>>> entry
>>> > >> counts, backup entry counts, and heap cost.
>>> > >>
>>> > >> The proposal aims to help users detect state growth, data skew, and
>>> > >> memory pressure earlier, improving troubleshooting, capacity
>>> planning,
>>> > and
>>> > >> production stability.
>>> > >>
>>> > >> 14.
>>> > >>
>>> > >> Real-time observability for Zeta engine tasks
>>> > >>
>>> > >> zhangshenghang proposed improving real-time observability for Zeta
>>> > engine
>>> > >> tasks.
>>> > >>
>>> > >> This direction adds the ability to monitor the runtime busyness
>>> level of
>>> > >> Zeta engine tasks, including real-time task metrics for Source,
>>> > Transform,
>>> > >> and Sink execution. It also includes queue and backpressure-related
>>> > >> information exposed through REST API and Web UI.
>>> > >>
>>> > >> The goal is to help users understand whether a job is blocked by
>>> source
>>> > >> reading, transform processing, sink writing, or intermediate queues.
>>> > >>
>>> > >> 15.
>>> > >>
>>> > >> Error data bypass and dead-letter queue capability for Transform
>>> and
>>> > >> Sink
>>> > >>
>>> > >> zhangshenghang also proposed row-level error handling for Transform
>>> and
>>> > >> Sink stages.
>>> > >>
>>> > >> When configured, SeaTunnel can collect failed records with useful
>>> > >> context, including:
>>> > >>
>>> > >> -
>>> > >>
>>> > >> error stage
>>> > >> -
>>> > >>
>>> > >> plugin type
>>> > >> -
>>> > >>
>>> > >> plugin name
>>> > >> -
>>> > >>
>>> > >> source table
>>> > >> -
>>> > >>
>>> > >> error message
>>> > >> -
>>> > >>
>>> > >> exception class
>>> > >> -
>>> > >>
>>> > >> original data
>>> > >> -
>>> > >>
>>> > >> occurrence time
>>> > >>
>>> > >> This allows jobs to bypass selected dirty or error records instead
>>> of
>>> > >> always failing fast, while keeping the default behavior unchanged
>>> when
>>> > the
>>> > >> feature is not enabled.
>>> > >>
>>> > >> This is useful for long-running synchronization jobs where users
>>> need
>>> > >> better fault tolerance and later inspection of bad data.
>>> > >>
>>> > >> 16.
>>> > >>
>>> > >> Improve cloud-native deployment experience
>>> > >>
>>> > >> zhangshenghang also suggested improving the cloud-native deployment
>>> > >> experience for SeaTunnel 3.0.
>>> > >>
>>> > >> The proposal mentions Kubernetes and Helm support as important
>>> areas,
>>> > >> including better guidance and behavior around:
>>> > >>
>>> > >> -
>>> > >>
>>> > >> deployment
>>> > >> -
>>> > >>
>>> > >> restart
>>> > >> -
>>> > >>
>>> > >> persistent checkpoint storage
>>> > >> -
>>> > >>
>>> > >> configuration management
>>> > >> -
>>> > >>
>>> > >> secret management
>>> > >> -
>>> > >>
>>> > >> rolling upgrade
>>> > >> -
>>> > >>
>>> > >> resource isolation
>>> > >>
>>> > >>
>>> > >> 17.
>>> > >>
>>> > >> Clean up deprecated behavior carefully for 3.0
>>> > >>
>>> > >> zhangshenghang also suggested that SeaTunnel 3.0 is a good
>>> opportunity
>>> > to
>>> > >> clean up deprecated options and old APIs.
>>> > >>
>>> > >> At the same time, any incompatible change should be handled
>>> carefully,
>>> > >> with clear migration documentation, examples, and release notes.
>>> > >>
>>> > >> Possible discussion groups
>>> > >>
>>> > >> Based on the above proposals, I think the roadmap discussion can be
>>> > >> grouped into the following areas:
>>> > >>
>>> > >> 1.
>>> > >>
>>> > >> Zeta engine observability
>>> > >>
>>> > >>
>>> > >> -
>>> > >>
>>> > >> STIP-21 / StainTrace
>>> > >> -
>>> > >>
>>> > >> Prometheus metrics for IMap state stores
>>> > >> -
>>> > >>
>>> > >> Real-time task metrics
>>> > >> -
>>> > >>
>>> > >> Queue and backpressure visibility
>>> > >> -
>>> > >>
>>> > >> REST API and Web UI exposure
>>> > >>
>>> > >>
>>> > >> 2.
>>> > >>
>>> > >> Zeta state management and storage evolution
>>> > >>
>>> > >>
>>> > >> -
>>> > >>
>>> > >> StateStore abstraction
>>> > >> -
>>> > >>
>>> > >> IMap external storage compaction
>>> > >> -
>>> > >>
>>> > >> IMap state metrics
>>> > >> -
>>> > >>
>>> > >> Future storage backend evolution
>>> > >>
>>> > >>
>>> > >> 3.
>>> > >>
>>> > >> Runtime and connector semantics
>>> > >>
>>> > >>
>>> > >> -
>>> > >>
>>> > >> Engine-level timer flush for Sink connectors
>>> > >> -
>>> > >>
>>> > >> JDBC adoption of timer flush
>>> > >> -
>>> > >>
>>> > >> Checkpoint-gated file source lifecycle management
>>> > >> -
>>> > >>
>>> > >> Table-level fault isolation
>>> > >> -
>>> > >>
>>> > >> Row-level error bypass and dead-letter queue
>>> > >>
>>> > >>
>>> > >> 4.
>>> > >>
>>> > >> AI for Data Engineering
>>> > >>
>>> > >>
>>> > >> -
>>> > >>
>>> > >> AI-Native CLI
>>> > >> -
>>> > >>
>>> > >> Natural language pipeline generation
>>> > >> -
>>> > >>
>>> > >> Connector metadata reflection
>>> > >> -
>>> > >>
>>> > >> Auto-fix loop
>>> > >> -
>>> > >>
>>> > >> Security and sensitive value redaction
>>> > >> -
>>> > >>
>>> > >> Future AI-assisted schema mapping, diagnosis, optimization, and
>>> > >> autonomous data management
>>> > >>
>>> > >>
>>> > >> 5.
>>> > >>
>>> > >> Edge collection
>>> > >>
>>> > >>
>>> > >> -
>>> > >>
>>> > >> EdgeSocket ingress
>>> > >> -
>>> > >>
>>> > >> SeaTunnel Edge Agent
>>> > >> -
>>> > >>
>>> > >> Edge host to Zeta pipeline workflow
>>> > >> -
>>> > >>
>>> > >> Authentication, local durability, and transport interoperability
>>> > >>
>>> > >>
>>> > >> 6.
>>> > >>
>>> > >> Performance and internal data model
>>> > >>
>>> > >>
>>> > >> -
>>> > >>
>>> > >> Apache Arrow as an optional internal data model
>>> > >> -
>>> > >>
>>> > >> Columnar I/O
>>> > >> -
>>> > >>
>>> > >> Vectorized execution
>>> > >> -
>>> > >>
>>> > >> Lower JVM overhead
>>> > >>
>>> > >>
>>> > >> 7.
>>> > >>
>>> > >> Build, release, and project structure
>>> > >>
>>> > >>
>>> > >> -
>>> > >>
>>> > >> Independent seatunnel-shade project
>>> > >> -
>>> > >>
>>> > >> Maven Central distribution of shade jars
>>> > >> -
>>> > >>
>>> > >> Reduced build and release time
>>> > >>
>>> > >>
>>> > >> 8.
>>> > >>
>>> > >> Cloud-native and compatibility
>>> > >>
>>> > >>
>>> > >> -
>>> > >>
>>> > >> Kubernetes and Helm deployment experience
>>> > >> -
>>> > >>
>>> > >> Persistent checkpoint storage
>>> > >> -
>>> > >>
>>> > >> Secret management
>>> > >> -
>>> > >>
>>> > >> Rolling upgrade
>>> > >> -
>>> > >>
>>> > >> Resource isolation
>>> > >> -
>>> > >>
>>> > >> Deprecated behavior cleanup
>>> > >> -
>>> > >>
>>> > >> Migration documentation and release notes
>>> > >>
>>> > >> Next steps
>>> > >>
>>> > >> I suggest we continue to review these proposals and decide:
>>> > >>
>>> > >> 1.
>>> > >>
>>> > >> Which items should be included in the SeaTunnel 3.0 roadmap.
>>> > >> 2.
>>> > >>
>>> > >> Which items should be marked as experimental.
>>> > >> 3.
>>> > >>
>>> > >> Which items require an STIP or further design discussion.
>>> > >> 4.
>>> > >>
>>> > >> Which items can be merged independently before the 3.0 release.
>>> > >> 5.
>>> > >>
>>> > >> Which items need migration documentation, compatibility notes, or
>>> > >> release notes.
>>> > >>
>>> > >> Again, this is only a summary of the current discussion thread.
>>> Please
>>> > >> help correct or supplement anything I missed.
>>> > >>
>>> > >>
>>> > >>
>>> > >> Best Regards,
>>> > >> David
>>> > >> Linkedin: https://www.linkedin.com/in/davidzollo
>>> > >>
>>> > >>
>>> > >>
>>> > >> On Thu, May 14, 2026 at 2:59 PM zhangshenghang <
>>> shenghang@apache.org>
>>> > >> wrote:
>>> > >>
>>> > >>> Hi community,
>>> > >>>
>>> > >>> For the SeaTunnel 3.0 roadmap
>>> > >>>
>>> > >>> I have already submitted two related PRs, and besides those, I
>>> would
>>> > like
>>> > >>> to propose the following directions for 3.0:
>>> > >>>
>>> > >>> 1. Real-time observability for Zeta engine tasks
>>> > >>>
>>> > >>> This work adds the ability to monitor the runtime busyness level of
>>> > Zeta
>>> > >>> engine tasks. It introduces real-time task metrics for Source,
>>> > Transform,
>>> > >>> and Sink execution, including queue/backpressure-related
>>> information,
>>> > and
>>> > >>> exposes these metrics through REST API and Web UI. This can help
>>> users
>>> > >>> understand whether a job is blocked by source reading, transform
>>> > >>> processing, sink writing, or intermediate queues, which is very
>>> > important
>>> > >>> for production troubleshooting and performance tuning.
>>> > >>>
>>> > >>> 2. Error data bypass and dead-letter queue capability for
>>> Transform and
>>> > >>> Sink
>>> > >>>
>>> > >>> This work introduces row-level error handling for Transform and
>>> Sink
>>> > >>> stages. When configured, SeaTunnel can collect failed records with
>>> > useful
>>> > >>> context such as error stage, plugin type, plugin name, source
>>> table,
>>> > >>> error
>>> > >>> message, exception class, original data, and occurrence time. This
>>> > allows
>>> > >>> jobs to bypass selected dirty/error records instead of always
>>> failing
>>> > >>> fast,
>>> > >>> while keeping the default behavior unchanged when the feature is
>>> not
>>> > >>> enabled. This is useful for long-running synchronization jobs where
>>> > users
>>> > >>> need better fault tolerance and later inspection of bad data.
>>> > >>>
>>> > >>> 3. Improve cloud-native deployment experience
>>> > >>>
>>> > >>> Kubernetes and Helm support are important for new users. We should
>>> > >>> improve
>>> > >>> guidance and behavior around deployment, restart, persistent
>>> checkpoint
>>> > >>> storage, configuration management, secret management, rolling
>>> upgrade,
>>> > >>> and
>>> > >>> resource isolation in cloud-native environments.
>>> > >>>
>>> > >>> 4. Clean up deprecated behavior carefully for 3.0
>>> > >>>
>>> > >>> Since 3.0 is a major version, it is a good opportunity to clean up
>>> > >>> deprecated options and old APIs, but we should do this carefully.
>>> Any
>>> > >>> incompatible change should have clear migration documentation,
>>> > examples,
>>> > >>> and release notes.
>>> > >>>
>>> > >>>
>>> > >>> Regards,
>>> > >>> Jast (Shenghang)
>>> > >>>
>>> > >>>
>>> > >>> JeremyXin <73...@qq.com> 于2026年5月14日周四 14:27写道：
>>> > >>>
>>> > >>> > Hi everyone, I’d like to propose adding this PR to the SeaTunnel
>>> 3.0
>>> > >>> > roadmap.
>>> > >>> >
>>> > >>> >
>>> > >>> > - PR#10860
>>> > >>> > - link: https://github.com/apache/seatunnel/pull/10860
>>> > >>> >
>>> > >>> >
>>> > >>> > This PR improves Zeta Engine observability by exposing Prometheus
>>> > >>> metrics
>>> > >>> > for core engine state stores backed by Hazelcast IMap.
>>> > >>> >
>>> > >>> >
>>> > >>> > It covers important internal stores such as running job
>>> information,
>>> > >>> job
>>> > >>> > states, checkpoints, slot ownership and provides visibility into
>>> > their
>>> > >>> > local entry counts, backup entry counts, and heap cost.
>>> > >>> >
>>> > >>> >
>>> > >>> > Including this in SeaTunnel 3.0 would make the engine easier to
>>> > monitor
>>> > >>> > and operate in production. It helps users detect state growth,
>>> data
>>> > >>> skew,
>>> > >>> > and memory pressure earlier, which improves troubleshooting
>>> > efficiency,
>>> > >>> > cluster capacity planning, and overall operational stability.
>>> > >>> >
>>> > >>> >
>>> > >>> > Best regards,
>>> > >>> > JeremyXin
>>> > >>> >
>>> > >>> >
>>> > >>> > 原始邮件
>>> > >>> >
>>> > >>> >
>>> > >>> > 发件人：zhiwei niu <zniu70696@gmail.com>
>>> > >>> > 发件时间：2026年5月12日 10:27
>>> > >>> > 收件人：dev <dev@seatunnel.apache.org>
>>> > >>> > 主题：Re: [DISCUSS] Apache SeaTunnel 3.0 Roadmap
>>> > >>> >
>>> > >>> >
>>> > >>> >
>>> > >>> > Hi everyone,
>>> > >>> >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> As part of Roadmap 3.0 progress tracking, the STIP-24 PR links have been
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> updated to keep references consistent and easier to follow.
>>> > >>> >
>>> > >>> >
>>> Please use the updated PR links below:
>>> > >>> >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> [Feature][Zeta] STIP-24 Phase 1: Support lightweight EdgeSocket ingress
>>> > >>> > for edge collector MVP
>>> > >>> > - PR #10871(
>>> > >>> > https://github.com/apache/seatunnel/pull/10871)
>>> > >>> > —> PR#10878(
>>> > >>> > https://github.com/apache/seatunnel/pull/10878)
>>> > >>> >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> [Feature][Zeta] STIP-24 Phase 2: Introduce standalone SeaTunnel Edge Agent
>>> > >>> > MVP and dist integration
>>> > >>> > - PR #10876(
>>> > >>> > https://github.com/apache/seatunnel/pull/10876)
>>> > >>> > — PR#10879(
>>> > >>> > https://github.com/apache/seatunnel/pull/10879)
>>> > >>> >
>>> > >>> >
>>> > >>> > zhiwei niu <zniu70696@gmail.com
>>> > >>> > > 于2026年5月12日周二 01:31写道：
>>> > >>> >
>>> > >>> > > Hi everyone,
>>> > >>> > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> > In addition to STIP-23, I’d like to propose the STIP-24 edge-collection
>>> > >>> > > work for SeaTunnel 3.0.
>>> > >>> > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> > - PR #10871 — [Feature][Zeta] STIP-24 Phase 1: Support lightweight
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> > EdgeSocket ingress for edge collector MVP
>>> > >>> > > https://github.com/apache/seatunnel/pull/10871
>>> > >>> > > What it adds:
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> > Introduces the engine-side ingress foundation for edge collection via
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> > EdgeSocket, including authenticated ingress, source-side protocol handling,
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> > and task-group address discovery support in Zeta.
>>> > >>> > > Why 3.0:
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> > This is the foundational engine capability for remote edge data collection
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> > scenarios. It is backward-compatible and establishes the protocol/runtime
>>> > >>> >
>>> > >>>
>>> >
>>> > path required for the edge-side agent.
>>> > >>> > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> > - PR #10876 — [Feature][Zeta] STIP-24 Phase 2: Introduce standalone
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> > SeaTunnel Edge Agent MVP and dist integration
>>> > >>> > > https://github.com/apache/seatunnel/pull/10876
>>> > >>> > > What it adds:
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> > Introduces a standalone SeaTunnel Edge Agent runtime
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> > (core/transport/connector), local file/log/event inputs, SQLite WAL
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> > durability, EdgeSocket transport interop, and distribution integration.
>>> > >>> > > Why 3.0:
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> > This delivers the first practical end-to-end edge collection workflow
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> > (edge host -> Zeta pipeline), which significantly expands SeaTunnel’s
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> > deployment topology and real-world ingestion coverage.
>>> > >>> > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> > PR #10876 depends on #10871; both are under active review.
>>> > >>> > >
>>> > >>> > > Best regards,
>>> > >>> > > zhiwei.niu
>>> > >>> > >
>>> > >>> > > xuepeng wang <wxuepeng5@gmail.com
>>> > >>> > > 于2026年5月8日周五 14:03写道：
>>> > >>> > >
>>> > >>> > >> Hi team,
>>> > >>> > >>
>>> > >>> > >>
>>> > >>> > >>
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> I'd like to add STIP-27 [1] to the SeaTunnel 3.0 Roadmap discussion.
>>> > >>> > >>
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> This proposal introduces Apache Arrow as an optional internal data model
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> (not replacing the existing row model). It aims to improve performance for
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> columnar scenarios (Parquet/ORC, ClickHouse/Doris) and batch transforms
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> (projection/filter/cast), while keeping the current architecture stable.
>>> > >>> > >>
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> Key benefits: better columnar I/O performance, vectorized execution,
>>> > >>> > >> lower JVM overhead.
>>> > >>> > >>
>>> > >>> > >> [1]
>>> > >>> > https://github.com/apache/seatunnel/issues/10806
>>> > >>> > >> <https://github.com/apache/seatunnel/issues/10806>>
>>> >;
>>> > >>> > >> Thoughts?
>>> > >>> > >>
>>> > >>> > >> Regards,
>>> > >>> > >> xuepeng(github id:chl-wxp)
>>> > >>> > >>
>>> > >>> > >> David Zollo <davidzollo365@gmail.com
>>> > >>> > > 于2026年4月30日周四 21:17写道：
>>> > >>> > >>
>>> > >>> > >> > Hi everyone,
>>> > >>> > >> >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > First of all, thank you all for the fantastic proposals so far — the
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > quality and breadth of ideas in this thread have been genuinely
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > exciting to read. From StainTrace to the AI-Native CLI, from
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > StateStore abstraction to table-level fault isolation, it's clear that
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > this community has a bold and thoughtful vision for what SeaTunnel 3.0
>>> > >>> > >> > can become.
>>> > >>> > >> >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > Given that the Golden Week holiday is just around the corner, and many
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > of our contributors are based in China or taking time off during this
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > period, I'd like to extend the discussion window to May 15th. This
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > gives everyone enough breathing room to enjoy the holiday, come back
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > refreshed, and still have time to share their thoughts without feeling
>>> > >>> > >> > rushed.
>>> > >>> > >> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > Happy Labor Day to everyone!
>>> > >>> > >> >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > We highly welcome everyone's suggestions for version 3.0! Feel free to
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > leave a message if you have any ideas.
>>> > >>> > >> >
>>> > >>> > >> >
>>> > >>> > >> >
>>> > >>> > >> > Best Regards,
>>> > >>> > >> > David
>>> > >>> > >> > Linkedin:
>>> > >>> > https://www.linkedin.com/in/davidzollo
>>> > >>> > >> >
>>> > >>> > <https://www.linkedin.com/in/davidzollo>> >>;
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > On Wed, Apr 29, 2026 at 8:50 PM sean zhang <
>>> > >>> > seanzhang92@gmail.com>
>>> > >>> > >> wrote:
>>> > >>> > >> > >
>>> > >>> >
>>> >> > > Hi SeaTunnel community,
>>> > >>> > >> > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > I'd like to propose including the AI-Native CLI capability in the
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > SeaTunnel 3.0 release, and share a broader vision for the "AI for Data
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > Engineering" direction. I believe this aligns well with the 3.0
>>> > >>> > >> milestone
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > as a significant user-experience leap.
>>> > >>> > >> > > This PR
>>> > >>> > https://github.com/apache/seatunnel/pull/10789 introduces
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > `seatunnel-cli`, an AI-powered command-line tool that
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > generates production-ready pipeline configurations from natural
>>> > >>> > >> language
>>> > >>> > <
>>> > >>>
>>> >
>>> https://github.com/apache/seatunnel/pull/10789 introduces>> > > seatunnel-cli, an AI-powered command-line tool that>> > > generates production-ready pipeline configurations from natural>> language
>>> > >>> >
>>> > >>> > :
>>> > >>> > >> > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > $ seatunnel "Sync the users table from MySQL to S3 in parquet
>>> > >>> > >> format"
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > ✅ Generated config: mysql_users_to_s3.conf
>>> > >>> > >> > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > Key capabilities in the current implementation:
>>> > >>> > >> > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > - Multi-LLM provider support (Anthropic / OpenAI / AWS Bedrock)
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > with auto-detection
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > - Runtime connector metadata extraction via Java OptionRule
>>> > >>> > >> reflection,
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > ensuring 100% accurate parameter knowledge for all 100+ connectors
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > - Source/Sink type-aware validation (Jdbc source options ≠ Jdbc sink
>>> > >>> > >> > > options)
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > - Auto-fix loop: generated configs are validated and self-corrected
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > (up to 3 iterations)
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > - Security hardening: sensitive values (passwords, tokens, keys) are
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > automatically redacted to ${ENV_VAR} placeholders before
>>> > >>> > >> persistence
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > and LLM transmission
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > - Interactive mode with session memory and one-shot mode for CI/CD
>>> > >>> > >> > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > Future Roadmap — AI for Data Engineering
>>> > >>> > >> > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > Looking beyond config generation, I see a multi-level roadmap where
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > AI becomes deeply integrated into SeaTunnel's core value proposition:
>>> > >>> > >> > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > ### Level 2: AI Understands Data (Next phase)
>>> > >>> > >> > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > - Intelligent Schema Mapping: AI reads source schema, infers target
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > schema, and auto-generates type conversions and DDL
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > - Natural Language Transforms: "mask phone numbers, merge first and
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > last name, filter last 7 days" → SQL Transform auto-generation
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > - Data Profiling: AI samples source data, detects anomalies (NULLs,
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > outliers, format violations), and suggests data quality transforms
>>> > >>> > >> > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > ### Level 3: AI Operates Pipelines
>>> > >>> > >> > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > - Intelligent Failure Diagnosis: AI analyzes error logs, pinpoints
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > root cause (connection refused → MySQL down vs firewall), suggests
>>> > >>> > >> > fix
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > - Self-Healing Pipelines: auto-detect DDL changes in CDC sources,
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > auto-evolve sink schema, zero-downtime recovery
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > - Anomaly Detection: monitor data flow volume/latency, alert when
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > throughput drops 99% with correlated root cause
>>> > >>> > >> > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > ### Level 4: AI Optimizes Everything
>>> > >>> > >> > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > - Performance Tuning: AI analyzes bottlenecks (batch_size too small,
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > wrong file format, insufficient parallelism), suggests
>>> > >>> > >> optimization
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > - Cost-Aware Planning: given N tables to sync, AI proposes pipeline
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > topology that balances cost vs latency vs resource usage
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > - Intelligent Scheduling: learn from historical patterns to optimize
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > sync frequency per table
>>> > >>> > >> > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > ### Level 5: Autonomous Data Platform (Long-term vision)
>>> > >>> > >> > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > - Conversational data infrastructure management: "onboard the
>>> > >>> > >> payment
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > system to the data lake, ensure GDPR compliance" → AI
>>> > >>> > >> auto-discovers
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > sources, identifies PII, generates pipelines with masking
>>> > >>> > >> transforms,
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > deploys with monitoring and alerting
>>> > >>> > >> > >
>>> > >>> > >> > >
>>> > >>> > >> > > Doyeon Kim <
>>> > >>> > ekagnlek2@gmail.com> 于2026年4月29日周三 18:17写道：
>>> > >>> > >> > >
>>> > >>> > >> > > > Hi everyone,
>>> > >>> > >> > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > I’d like to propose the following Zeta engine improvements for
>>> > >>> > >> > SeaTunnel
>>> > >>> > >> > > > 3.0.
>>> > >>> > >> > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > 1. StateStore abstraction
>>> > >>> > >> > > > PR:
>>> > >>> > https://github.com/apache/seatunnel/pull/10812
>>> > >>> > >> > > >
>>> > >>> > <
>>> > >>>
>>> >
>>> https://github.com/apache/seatunnel/pull/10812>> > > >
>>> > >>> >
>>> > >>> > ;
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > This introduces an initial StateStore abstraction layer to reduce
>>> > >>> > >> > direct
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > Hazelcast IMap dependencies in the engine. It makes different state
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > responsibilities more explicit and prepares the engine for future
>>> > >>> > >> > backend
>>> > >>> > >> > > > evolution.
>>> > >>> > >> > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > I think this is relevant for 3.0 because it is a foundational step
>>> > >>> > >> > toward
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > decoupling the engine from Hazelcast-specific APIs and enabling
>>> > >>> > >> future
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > improvements around state storage, node communication, and
>>> > >>> > >> coordination
>>> > >>> > >> > > > layers.
>>> > >>> > >> > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > 2. IMap external storage compaction
>>> > >>> > >> > > > PR:
>>> > >>> > https://github.com/apache/seatunnel/pull/10399
>>> > >>> > >> > > >
>>> > >>> > <
>>> > >>>
>>> >
>>> https://github.com/apache/seatunnel/pull/10399>> > > >
>>> > >>> >
>>> > >>> > ;
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > This introduces LSM-style write and compaction logic for IMap
>>> > >>> > >> external
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > storage. It helps control storage growth by flushing sorted
>>> > >>> > >> immutable
>>> > >>> > >> > files
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > and compacting persisted files while keeping only the latest record
>>> > >>> > >> per
>>> > >>> > >> > > > key.
>>> > >>> > >> > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > I think this should be considered for 3.0 because long-running Zeta
>>> > >>> > >> > jobs
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > may accumulate duplicated records and tombstones in external storage
>>> > >>> > >> > over
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > time. Compaction can improve stability and reduce the risk of
>>> > >>> > >> unbounded
>>> > >>> > >> > > > storage growth.
>>> > >>> > >> > > >
>>> > >>> > >> > > > Best Regards,
>>> > >>> > >> > > > Doyeon
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > (Github ID: dybyte)
>>> > >>> > >> > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > On Wed, Apr 29, 2026 at 7:13 PM David Zollo <
>>> > >>> > >> davidzollo365@gmail.com>
>>> > >>> > >> > > > wrote:
>>> > >>> > >> > > >
>>> > >>> >
>>> >> > > > > Hi guys,
>>> > >>> > >> > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > I'd like to propose including PR #10491 in the SeaTunnel 3.0
>>> > >>> > >> roadmap,
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > here are the details with additional packaging time analysis:
>>> > >>> > >> > > > >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > ### PR link
>>> > >>> > >> > > > >
>>> > >>> > https://github.com/apache/seatunnel/pull/10491
>>> > >>> > >> > > > >
>>> > >>> > <
>>> > >>>
>>> >
>>> https://github.com/apache/seatunnel/pull/10491>> > > > >
>>> > >>> >
>>> > >>> > ;
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > StainTrace (PR #10491) is the concrete implementation of STIP-21,
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > delivering end-to-end data lineage tracing and performance
>>> > >>> > >> analysis
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > capabilities for the Apache SeaTunnel Zeta engine. It injects
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > lightweight timestamp sampling points at 6 key pipeline stages —
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > SOURCE_EMIT → QUEUE_IN → QUEUE_OUT → TRANSFORM_IN → TRANSFORM_OUT
>>> > >>> > >> →
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > SINK_WRITE_DONE — tracking each sampled row's complete journey
>>> > >>> > >> from
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > source emission to sink write completion, enabling precise
>>> > >>> > >> end-to-end
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > latency measurement and bottleneck identification.
>>> > >>> > >> > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > The system is designed with production safety as a first-class
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > concern: it is disabled by default with very low overhead, and
>>> > >>> > >> when
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > enabled it operates at a configurable sampling rate with a strict
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > per-worker per-second event budget, achieving approximately 0.1~1%
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > overhead at 1% sampling. Trace data is written locally as JSONL
>>> > >>> > >> files
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > in OpenTelemetry OTLP JSON format with no external dependencies.
>>> > >>> > >> An
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > accompanying offline analyzer reads these files to compute P95/P99
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > latency statistics, identify bottleneck stages, and generate an
>>> > >>> > >> HTML
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > visualization report with Chinese/English language toggle support.
>>> > >>> > >> > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > The value to SeaTunnel lies in closing a critical observability
>>> > >>> > >> gap
>>> > >>> > >> > at
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > the engine level. Before this feature, users facing latency
>>> > >>> > >> spikes or
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > throughput bottlenecks had no structured way to pinpoint whether
>>> > >>> > >> the
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > problem resided in queue backpressure, transform processing, or
>>> > >>> > >> sink
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > write time — they could only rely on logs and guesswork.
>>> > >>> > >> StainTrace
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > makes per-stage timing transparent and actionable, which is
>>> > >>> > >> > especially
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > impactful for large-scale, multi-table, long-chain production
>>> > >>> > >> > > > > pipelines.
>>> > >>> > >> > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > The case for including it in the 3.0 roadmap is equally strong. As
>>> > >>> > >> > the
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > implementation of STIP-21, StainTrace is a framework-level
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > observability capability — connector-agnostic and immediately
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > effective across all jobs once merged. One of SeaTunnel 3.0's
>>> > >>> > >> central
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > goals is to become a more mature, production-ready data
>>> > >>> > >> integration
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > engine, and observability is an indispensable foundation for that
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > goal. Establishing it at a major version boundary like 3.0
>>> > >>> > >> delivers
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > far greater long-term value to users.
>>> > >>> > >> > > > >
>>> > >>> > >> > > > >
>>> > >>> > >> > > > >
>>> > >>> > >> > > > >
>>> > >>> > >> > > > >
>>> > >>> > >> > > > >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > Best Regards,
>>> > >>> > >> > > > > David
>>> > >>> >
>>> >> > > > > Linkedin:
>>> > >>> > https://www.linkedin.com/in/davidzollo
>>> > >>> > >> > > > >
>>> > >>> > <
>>> > >>>
>>> >
>>> https://www.linkedin.com/in/davidzollo>> > > > >
>>> > >>> >
>>> > >>> > ;
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > On Wed, Apr 29, 2026 at 2:48 PM yi zeng <
>>> > >>> > yzeng1618@gmail.com>
>>> > >>> > >> wrote:
>>> > >>> > >> > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > Hi everyone,
>>> > >>> > >> > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > I’d like to propose including the following two PRs in the
>>> > >>> > >> > *SeaTunnel
>>> > >>> > >> > > > 3.0
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > roadmap*. Both focus on strengthening *correctness, reliability,
>>> > >>> > >> > and
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > operational safety* for core multi-table and continuous
>>> > >>> > >> ingestion
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > scenarios, which I believe are key pillars for 3.0.
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > ------------------------------
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > PR #10600 — [Feature][Core] Support table-level fault isolation
>>> > >>> > >> for
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > multi-table synchronization
>>> > >>> > >> > > > > >
>>> > >>> > >> > > > > >
>>> > >>> > https://github.com/apache/seatunnel/pull/10600
>>> > >>> > >> > > > > >
>>> > >>> > <
>>> > >>>
>>> >
>>> https://github.com/apache/seatunnel/pull/10600>> > > > > >
>>> > >>> >
>>> > >>> > ;
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > *What it adds / solves*
>>> > >>> > >> > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > This PR introduces *table-level fault isolation* for multi-table
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > synchronization by adding a new failure policy:
>>> > >>> >
>>> >> > > > > > code
>>> > >>> >
>>> >> > > > > > 复制
>>> > >>> > >> > > > > >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > env {
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > multi_table {
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > failure_policy = "CONTINUE_OTHER_TABLES"
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > }
>>> > >>> > >> > > > > > }
>>> > >>> > >> > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > With this enabled:
>>> > >>> > >> > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > - Failures are isolated at *table granularity*, instead of
>>> > >>> > >> > failing
>>> > >>> > >> > > > the
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > entire job.
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > - Healthy tables continue processing even if one table fails.
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > - Failed tables are collected and reported with clear
>>> > >>> > >> metadata
>>> > >>> > >> > > > (table
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > path, failure phase, plugin, error).
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > - Batch jobs report failed tables in final results; streaming
>>> > >>> > >> > jobs
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > continue running with healthy tables.
>>> > >>> > >> > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > This addresses a long-standing pain point where a single
>>> > >>> > >> > problematic
>>> > >>> > >> > > > > table
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > can block progress for all others.
>>> > >>> > >> > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > *Why it should be included in 3.0*
>>> > >>> > >> > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > - *Improved fault tolerance model*: 3.0 aims to be more
>>> > >>> > >> > resilient
>>> > >>> > >> > > > and
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > production-ready; table-level isolation is a foundational
>>> > >>> > >> > capability
>>> > >>> > >> > > > > for
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > large-scale, multi-table pipelines.
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > - *Backward-compatible by default*: The default behavior
>>> > >>> > >> remains
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > FAIL_FAST, so existing jobs are unaffected unless users
>>> > >>> > >> > explicitly
>>> > >>> > >> > > > opt
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > in.
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > - *Framework-level improvement*: This is not
>>> > >>> > >> > connector-specific; it
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > enhances the core execution and failure-handling semantics of
>>> > >>> > >> > > > > SeaTunnel.
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > - *Mature design & coverage*: Includes core logic changes,
>>> > >>> > >> > runtime
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > handling, reporting, and E2E tests, making it suitable for a
>>> > >>> > >> > major
>>> > >>> > >> > > > > version
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > milestone.
>>> > >>> > >> > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > ------------------------------
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > PR #10563 — [Feature][Connector-V2] Add checkpoint-gated
>>> > >>> > >> > > > post_sync_action
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > and retention for file source continuous discovery
>>> > >>> > >> > > > > >
>>> > >>> > >> > > > > >
>>> > >>> > https://github.com/apache/seatunnel/pull/10563
>>> > >>> > >> > > > > >
>>> > >>> > <
>>> > >>>
>>> >
>>> https://github.com/apache/seatunnel/pull/10563>> > > > > >
>>> > >>> >
>>> > >>> > ;
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > *What it adds / solves*
>>> > >>> > >> > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > This PR adds *checkpoint-gated source-side lifecycle management*
>>> > >>> > >> > for
>>> > >>> > >> > > > file
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > source continuous discovery, including:
>>> > >>> > >> > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > - post_sync_action = delete
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > Delete source files only *after a checkpoint has completed*
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > - post_sync_action = backup
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > Move source files to a backup_path after checkpoint
>>> > >>> > >> completion
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > - Optional backup retention via retention_max_age
>>> > >>> > >> > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > All operations are *checkpoint-aware*, ensuring source files are
>>> > >>> > >> > never
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > deleted or moved before the data is durably processed.
>>> > >>> > >> > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > *Why it should be included in 3.0*
>>> > >>> > >> > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > - *Correctness-first continuous ingestion*: Safe source
>>> > >>> > >> cleanup
>>> > >>> > >> > tied
>>> > >>> > >> > > > > to
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > checkpoint completion is essential for exactly-once and
>>> > >>> > >> > recoverable
>>> > >>> > >> > > > > file
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > ingestion, which aligns strongly with 3.0 quality goals.
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > - *Production-grade operational semantics*: Without
>>> > >>> > >> > checkpoint-gated
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > cleanup, continuous file ingestion is risky in real
>>> > >>> > >> production
>>> > >>> >
>>> >> > > > > environments.
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > - *Extends Connector-V2 maturity*: This PR deepens the
>>> > >>> > >> > operational
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > completeness of Connector-V2, especially for file-based
>>> > >>> > >> sources.
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > - *Well-tested and documented*: Includes unit tests, E2E
>>> > >>> > >> tests
>>> > >>> > >> > > > (Local
>>> > >>> > >> > > > > +
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > HDFS), and documentation updates (EN + ZH).
>>> > >>> > >> > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > ------------------------------
>>> > >>> > >> > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > Both PRs are *opt-in*, backward-compatible, and focused on
>>> > >>> > >> making
>>> > >>> > >> > > > > SeaTunnel
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > more robust, predictable, and safe in real-world deployments. I
>>> > >>> > >> > believe
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > they fit naturally into the SeaTunnel 3.0 scope, especially
>>> > >>> > >> given
>>> > >>> > >> > the
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > emphasis on stability, correctness, and large-scale usage.
>>> > >>> > >> > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > Happy to discuss details or answer questions.
>>> > >>> > >> > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > Zhang Donghao <
>>> > >>> > zhangdonghao@apache.org> 于2026年4月29日周三 14:38写道：
>>> > >>> > >> > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > Hi everyone,
>>> > >>> >
>>> >> > > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > I'd like to propose including PR #9993 in the SeaTunnel 3.0
>>> > >>> > >> > roadmap,
>>> > >>> > >> > > > > here
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > are the details with additional packaging time analysis:
>>> > >>> >
>>> >> > > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > ### PR link
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > >
>>> > >>> > https://github.com/apache/seatunnel/pull/9993
>>> > >>> >
>>> >> > > > > > >
>>> > >>> > <
>>> > >>>
>>> >
>>> https://github.com/apache/seatunnel/pull/9993>> > > > > > >
>>> > >>> >
>>> > >>> > ;
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > ### Short description
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > This PR extracts the shade modules into an independent
>>> > >>> > >> > > > seatunnel-shade
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > project, enables uploading shade-jars to the central
>>> > >>> > >> repository,
>>> > >>> > >> > and
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > optimizes the overall packaging efficiency of SeaTunnel.
>>> > >>> >
>>> >> > > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > ### Packaging time analysis (current state before PR
>>> > >>> > >> > optimization)
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > The current build time breakdown of individual shade modules
>>> > >>> > >> is
>>> > >>> > >> > as
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > follows, highlighting the inefficiency of the integrated build
>>> > >>> > >> > > > > approach:
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > ```
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > [INFO] SeaTunnel : Shade : 3.0.0 ..........................
>>> > >>> > >> > SUCCESS [
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > 12.661 s]
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > [INFO] SeaTunnel : Shade : Hazelcast 5.1-3.0.0 ............
>>> > >>> > >> > SUCCESS [
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > 8.192 s]
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > [INFO] SeaTunnel : Shade : Hadoop3 Uber 3.1.4-3.0.0 .......
>>> > >>> > >> > SUCCESS
>>> > >>> > >> > > > > [02:29
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > min]
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > [INFO] SeaTunnel : Shade : Jackson 2.13.3-3.0.0 ...........
>>> > >>> > >> > SUCCESS [
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > 3.289 s]
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > [INFO] SeaTunnel : Shade : Guava 27.0-jre-3.0.0 ...........
>>> > >>> > >> > SUCCESS [
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > 2.960 s]
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > [INFO] SeaTunnel : Shade : Hadoop : AWS 3.1.4-3.0.0 .......
>>> > >>> > >> > SUCCESS
>>> > >>> > >> > > > > [05:05
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > min]
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > [INFO] SeaTunnel : Shade : Arrow 15.0.1-3.0.0 .............
>>> > >>> > >> > SUCCESS [
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > 5.880 s]
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > [INFO] SeaTunnel : Shade : Hikari 4.0.3-3.0.0 .............
>>> > >>> > >> > SUCCESS [
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > 1.085 s]
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > [INFO] SeaTunnel : Shade : Thrift-Service 1.0.0-3.0.0 .....
>>> > >>> > >> > SUCCESS [
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > 1.572 s]
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > [INFO] SeaTunnel : Shade : Janino 3.0.11-3.0.0 ............
>>> > >>> > >> > SUCCESS [
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > 1.235 s]
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > [INFO] SeaTunnel : Shade : Scala 2.13.11-3.0.0 ............
>>> > >>> > >> > SUCCESS [
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > 6.586 s]
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > [INFO] SeaTunnel : Shade : Jetty 9.4.56.v20240826-3.0.0 ...
>>> > >>> > >> > SUCCESS [
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > 2.421 s]
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > [INFO] SeaTunnel : Shade : Commons Lang3 3.18.0-3.0.0 .....
>>> > >>> > >> > SUCCESS [
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > 2.044 s]
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > ```
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > Key observations from the time breakdown:
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > 1. **Long-tail modules dominate build time**: Two modules
>>> > >>> > >> > account for
>>> > >>> > >> > > > > the
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > vast majority of shade build time — `Hadoop3 Uber` (2 minutes
>>> > >>> > >> 29
>>> > >>> > >> > > > > seconds)
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > and `Hadoop : AWS` (5 minutes 5 seconds), totaling ~7.5
>>> > >>> > >> minutes,
>>> > >>> > >> > > > while
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > other modules take only a few seconds each.
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > 2. **Integrated build waste**: In the current structure, these
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > time-consuming shade modules are built alongside core modules
>>> > >>> > >> > every
>>> > >>> > >> > > > > time
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > SeaTunnel is packaged, even when there are no changes to shade
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > dependencies. This results in repeated long build cycles for
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > developers and
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > release workflows.
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > 3. **Total shade build time**: The cumulative build time of
>>> > >>> > >> all
>>> > >>> > >> > shade
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > modules is approximately 8 minutes (excluding
>>> > >>> > >> > overlapping/parallel
>>> > >>> > >> > > > > build
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > optimizations), which is a significant portion of the overall
>>> > >>> > >> > > > SeaTunnel
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > packaging process.
>>> > >>> >
>>> >> > > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > ### Reason for including in 3.0
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > 1. **Build efficiency optimization (quantifiable gains)**:
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > - By isolating shade modules into an independent project
>>> > >>> > >> and
>>> > >>> > >> > > > > uploading
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > shade-jars to the central repository, we eliminate the need to
>>> > >>> > >> > > > rebuild
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > these modules in every SeaTunnel packaging cycle. For daily
>>> > >>> >
>>> >> > > > > development,
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > this reduces the total build time by ~8 minutes per build (for
>>> > >>> > >> > > > > developers
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > not modifying shade code).
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > - For the 3.0 release process, we can pre-build and publish
>>> > >>> > >> > > > > shade-jars
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > to the central repository once, then reference them as
>>> > >>> > >> > dependencies —
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > cutting the release packaging time by a substantial margin and
>>> > >>> >
>>> > >> > > > > reducing the
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > risk of build failures caused by long-running shade module
>>> > >>> > >> > builds.
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > 2. **Dependency management enhancement**: Uploading shade-jars
>>> > >>> > >> > to the
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > central repository standardizes dependency distribution,
>>> > >>> > >> avoiding
>>> > >>> > >> > > > > conflicts
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > caused by internal shade module duplication across
>>> > >>> > >> subprojects,
>>> > >>> > >> > and
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > improving the stability of the 3.0 version's dependency
>>> > >>> > >> system.
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > 3. **Project structure rationalization**: Isolating shade
>>> > >>> > >> modules
>>> > >>> > >> > > > > aligns
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > with the modular design concept of 3.0, making the project
>>> > >>> > >> > structure
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > clearer, easier to maintain, and facilitating future
>>> > >>> > >> expansion of
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > shade-related capabilities (e.g., adding new shade modules for
>>> > >>> > >> > > > > additional
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > dependencies).
>>> > >>> >
>>> >> > > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > This optimization directly addresses the identified packaging
>>> > >>> > >> > > > > bottlenecks
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > (long-tail shade modules) and delivers measurable time savings
>>> > >>> > >> > for
>>> > >>> > >> > > > both
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > development and release workflows — a critical improvement for
>>> > >>> > >> > the
>>> > >>> > >> > > > > smooth
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > delivery and maintainability of SeaTunnel 3.0.
>>> > >>> >
>>> >> > > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > Best regards,
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > zhangdonghao
>>> > >>> >
>>> >> > > > > > >
>>> > >>> > >> > > > >
>>> > >>> > >> > > >
>>> > >>> > >> >
>>> > >>> > >>
>>> > >>> > >
>>> > >>>
>>> > >>
>>> >
>>>
>>
Re: [DISCUSS] Apache SeaTunnel 3.0 Roadmap
Posted by David Zollo <da...@gmail.com>.
Hi guys,
Guys, please leave a message if you have any PR that needs to be merged in
the 3.0.0 version.
Firstly, thanks for your hard work and making great progress. Now we only
have 3 PRs left to update and review in the 3.0.0 Roadmap list[1].
Below is the list of remaining PRs pending merge in the Roadmap:
1. [#10563](https://github.com/apache/seatunnel/pull/10563)
[Feature][Connector-V2] Add checkpoint-gated post_sync_action and retention
for file source continuous discovery @yzeng1618
2. [#9993](https://github.com/apache/seatunnel/pull/9993)
[Feature][shade]Refactor the seatunnel-shade module @hawk9821
3. [#10306](https://github.com/apache/seatunnel/pull/10306) [Feature][Core]
Error Data Bypass Capability for Transform & Sink (Dead Letter Queue)
@zhangshenghang
Again, please leave a message if you have any PR need to be merged in 3.0.0
version.
[1]. https://github.com/apache/seatunnel/issues/11021
Best Regards,
David
Linkedin: https://www.linkedin.com/in/davidzollo
On Wed, Jul 15, 2026 at 12:36 PM David Zollo <da...@gmail.com>
wrote:
> Hi guys,
>
> Thanks for your hard work and make a great progress. Now we only have four
> PRs left to be reviewed and merged.
> Below is the list of remaining PRs pending merge in the Roadmap:
>
> 1. [#10563](https://github.com/apache/seatunnel/pull/10563)
> [Feature][Connector-V2] Add checkpoint-gated post_sync_action and retention
> for file source continuous discovery @yzeng1618
> 2. [#9993](https://github.com/apache/seatunnel/pull/9993)
> [Feature][shade]Refactor the seatunnel-shade module. @hawk9821
> 3. [#10879](https://github.com/apache/seatunnel/pull/10879)
> [Feature][Zeta] STIP-24 Phase 2: Introduce standalone SeaTunnel Edge Agent
> MVP and dist integration @nzw921rx
> 4. [#10306](https://github.com/apache/seatunnel/pull/10306)
> [Feature][Core] Error Data Bypass Capability for Transform & Sink (Dead
> Letter Queue) @zhangshenghang
>
>
> Guys, please help review the PR for the 3.0 Roadmap when you have time.
>
>
> Best Regards,
> David
> Linkedin: https://www.linkedin.com/in/davidzollo
>
>
>
>
> On Thu, Jul 9, 2026 at 10:56 PM zhiwei niu <zn...@gmail.com> wrote:
>
>> Hi community,
>>
>> I would like to propose including STIP-28 in the SeaTunnel 3.0 roadmap.
>>
>> STIP-28 enhances SeaTunnel's OptionRule framework from basic
>> presence/equality validation to richer declarative configuration
>> validation. This allows connector and transform configuration constraints
>> to be expressed as structured metadata instead of being hidden in
>> scattered
>> runtime if/throw logic.
>>
>> Related PRs:
>>
>> - Main PR: apache/seatunnel#10977
>> https://github.com/apache/seatunnel/pull/10977
>>
>> Completes the rich OptionRule validation framework.
>>
>> - Follow-up PR: apache/seatunnel#11228
>> https://github.com/apache/seatunnel/pull/11228
>>
>> Preserves extension validation error context.
>>
>> - Follow-up PR: apache/seatunnel#11121
>> https://github.com/apache/seatunnel/pull/11121
>>
>> Fixes validation error aggregation.
>>
>> - Follow-up PR: apache/seatunnel#11010
>> https://github.com/apache/seatunnel/pull/11010
>>
>> Adds Map-type condition validation.
>>
>> - Follow-up PR: apache/seatunnel#11022
>> https://github.com/apache/seatunnel/pull/11022
>>
>> Supports combining structural and value rules.
>>
>> - Follow-up PR: apache/seatunnel#11048
>> https://github.com/apache/seatunnel/pull/11048
>>
>> Adds custom extension validation support.
>>
>> Why this should be included in 3.0:
>>
>> - It makes SeaTunnel's configuration contract clearer and more explicit.
>> - Users can catch more configuration errors earlier through --check.
>> - REST API, Web UI, CLI metadata export, and future AI-assisted config
>> generation can consume richer validation metadata.
>> - Connector and transform developers can reduce duplicated imperative
>> validation logic.
>> - It provides a stronger foundation for future configuration generation,
>> validation, explanation, and repair capabilities.
>>
>> STIP-28 framework capabilities have already been merged, and some
>> connectors have already completed migration. Therefore, I believe STIP-28
>> is ready to be included in the SeaTunnel 3.0 roadmap.
>>
>> Best Regards,
>> zhiwei.niu
>>
>> David Zollo <da...@gmail.com> 于2026年7月9日周四 21:19写道：
>>
>> > Hi guys,
>> >
>> > Thanks for your hard work and make a great progress. Now we only have
>> the
>> > following PRs left to be reviewed and merged.
>> > Below is the list of remaining PRs pending merge in the Roadmap:
>> >
>> > 1. [#10563](https://github.com/apache/seatunnel/pull/10563)
>> > [Feature][Connector-V2] Add checkpoint-gated post_sync_action and
>> retention
>> > for file source continuous discovery @yzeng1618 PR conflicts and need
>> to
>> > solve
>> > 2. [#9993](https://github.com/apache/seatunnel/pull/9993)
>> > [Feature][shade]Refactor the seatunnel-shade module. @hawk9821 PR
>> > conflicts and need to solve
>> > 3. [#10879](https://github.com/apache/seatunnel/pull/10879)
>> > [Feature][Zeta]
>> > STIP-24 Phase 2: Introduce standalone SeaTunnel Edge Agent MVP and dist
>> > integration @nzw921rx
>> > 4. [#10800](https://github.com/apache/seatunnel/pull/10800)
>> > [Feature][Zeta]
>> > STIP-23 Phase 1: Add engine timer flush core flow and FlushSignal
>> handling
>> > @nzw921rx
>> > 5. [#10801](https://github.com/apache/seatunnel/pull/10801)
>> > [Feature][Connector-V2] STIP-23 Phase 2: JDBC connector adoption for
>> engine
>> > timer flush @nzw921rx
>> > 6. [#10306](https://github.com/apache/seatunnel/pull/10306)
>> > [Feature][Core]
>> > Error Data Bypass Capability for Transform & Sink (Dead Letter Queue)
>> > @zhangshenghang
>> >
>> >
>> > Guys, please help review the PR for the 3.0 Roadmap when you have time.
>> >
>> >
>> > Best Regards,
>> > David
>> > Linkedin: https://www.linkedin.com/in/davidzollo
>> >
>> >
>> >
>> > On Wed, Jul 1, 2026 at 2:54 PM David Zollo <da...@gmail.com>
>> > wrote:
>> >
>> > > Hi guys,
>> > >
>> > > I created an issue for the 3.0 roadmap to track progress[1]. Given
>> that
>> > > most PRs have been merged so far, we plan to finish merging the
>> remaining
>> > > PRs by the end of July and then kick off the release process!
>> > >
>> > > 1. https://github.com/apache/seatunnel/issues/11021
>> > >
>> > >
>> > >
>> > >
>> > > Best Regards,
>> > > David
>> > > Linkedin: https://www.linkedin.com/in/davidzollo
>> > >
>> > >
>> > >
>> > > On Sat, May 16, 2026 at 12:32 PM David Zollo <davidzollo365@gmail.com
>> >
>> > > wrote:
>> > >
>> > >> Hi SeaTunnel community,
>> > >>
>> > >> Thanks everyone for the active discussion around the Apache SeaTunnel
>> > 3.0
>> > >> roadmap.
>> > >>
>> > >> To make the follow-up review easier, I summarized the proposals
>> > collected
>> > >> in this thread so far. This is only a discussion summary, not a final
>> > >> decision. Please feel free to correct me if I missed anything or
>> > >> misunderstood any proposal.
>> > >>
>> > >> 1.
>> > >>
>> > >> SeaTunnel 3.0 roadmap discussion
>> > >>
>> > >> The discussion was started to collect proposals for Apache SeaTunnel
>> 3.0
>> > >> after PR #10510, which updated the version to 3.0.0, was merged into
>> the
>> > >> dev branch on March 11, 2026.
>> > >>
>> > >> Several already-merged PRs were mentioned as possible parts of the
>> 3.0
>> > >> scope:
>> > >>
>> > >> -
>> > >>
>> > >> PR #10586: [Feature][Zeta] Add datasource spi
>> > >> https://github.com/apache/seatunnel/pull/10586
>> > >> -
>> > >>
>> > >> PR #10585: [Feature][Zeta] Add /option-rules REST API for runtime
>> > >> connector metadata
>> > >> https://github.com/apache/seatunnel/pull/10585
>> > >> -
>> > >>
>> > >> PR #10489: [Feature][Transform] Add Zeta SQL UDF context and
>> > >> lifecycle hooks with e2e/docs updates
>> > >> https://github.com/apache/seatunnel/pull/10489
>> > >>
>> > >> The purpose of this thread is to collect PRs, features, improvements,
>> > and
>> > >> ideas that the community believes should be considered for SeaTunnel
>> > 3.0.
>> > >>
>> > >> 2.
>> > >>
>> > >> STIP-23: Engine-level timer flush for Sink connectors
>> > >>
>> > >> zhiwei niu proposed including STIP-23 in the SeaTunnel 3.0 roadmap.
>> The
>> > >> proposal focuses on engine-level timer flush support for Sink
>> > connectors.
>> > >>
>> > >> Related PRs:
>> > >>
>> > >> -
>> > >>
>> > >> PR #10800: [Feature][Zeta] Phase 1: Engine timer flush core
>> > >> https://github.com/apache/seatunnel/pull/10800
>> > >>
>> > >> This PR adds sink.flush.interval and introduces a FlushSignal
>> > >> control-plane path through the Zeta engine. It allows sinks to
>> flush
>> > based
>> > >> on a wall-clock interval without each sink managing its own timer
>> > thread.
>> > >> -
>> > >>
>> > >> PR #10801: [Feature][Connector-V2] Phase 2: JDBC connector
>> adoption
>> > >> https://github.com/apache/seatunnel/pull/10801
>> > >>
>> > >> This PR wires registerFlushAction into the JDBC non-XA sink writer
>> > >> through a new enable_timer_flush option. It also includes
>> multi-table
>> > >> aggregated flush and MySQL CDC E2E tests.
>> > >> -
>> > >>
>> > >> PR #10802: [Docs][Connector-V2] Phase 3: Timer flush feature
>> matrix
>> > >> https://github.com/apache/seatunnel/pull/10802
>> > >>
>> > >> This PR adds a timer flush support marker to sink connector
>> > >> documentation and documents the sink.flush.interval and
>> > registerFlushAction
>> > >> contract.
>> > >>
>> > >> The main value is to make end-to-end latency more predictable for
>> > >> buffered sinks, especially in idle or low-throughput streaming jobs.
>> > >>
>> > >> 3.
>> > >>
>> > >> Independent seatunnel-shade project and packaging efficiency
>> > >> optimization
>> > >>
>> > >> Zhang Donghao proposed including PR #9993 in the SeaTunnel 3.0
>> roadmap.
>> > >>
>> > >> -
>> > >>
>> > >> PR #9993:
>> > >> https://github.com/apache/seatunnel/pull/9993
>> > >>
>> > >> This PR extracts the shade modules into an independent
>> seatunnel-shade
>> > >> project, supports uploading shade jars to Maven Central, and
>> optimizes
>> > the
>> > >> overall packaging efficiency of SeaTunnel.
>> > >>
>> > >> The proposal points out that current shade module builds, especially
>> > >> Hadoop3 Uber and Hadoop AWS, add significant build time. By
>> publishing
>> > >> shade jars separately and reusing them as dependencies, SeaTunnel can
>> > >> reduce repeated build cost and improve release stability.
>> > >>
>> > >> 4.
>> > >>
>> > >> Table-level fault isolation for multi-table synchronization
>> > >>
>> > >> yi zeng proposed including PR #10600 in the SeaTunnel 3.0 roadmap.
>> > >>
>> > >> -
>> > >>
>> > >> PR #10600: [Feature][Core] Support table-level fault isolation for
>> > >> multi-table synchronization
>> > >> https://github.com/apache/seatunnel/pull/10600
>> > >>
>> > >> This PR introduces table-level fault isolation for multi-table
>> > >> synchronization through a new failure policy:
>> > >>
>> > >> env {
>> > >> multi_table {
>> > >> failure_policy = "CONTINUE_OTHER_TABLES"
>> > >> }
>> > >> }
>> > >>
>> > >> With this enabled, failures can be isolated at table granularity.
>> > Healthy
>> > >> tables can continue processing even if one table fails. Failed tables
>> > are
>> > >> collected and reported with metadata such as table path, failure
>> phase,
>> > >> plugin, and error.
>> > >>
>> > >> The proposal is backward-compatible by default because the existing
>> > >> FAIL_FAST behavior remains unchanged unless users explicitly enable
>> the
>> > new
>> > >> policy.
>> > >>
>> > >> 5.
>> > >>
>> > >> Checkpoint-gated post_sync_action and retention for file source
>> > >> continuous discovery
>> > >>
>> > >> yi zeng also proposed including PR #10563.
>> > >>
>> > >> -
>> > >>
>> > >> PR #10563: [Feature][Connector-V2] Add checkpoint-gated
>> > >> post_sync_action and retention for file source continuous
>> discovery
>> > >> https://github.com/apache/seatunnel/pull/10563
>> > >>
>> > >> This PR adds checkpoint-aware source-side lifecycle management for
>> file
>> > >> source continuous discovery, including:
>> > >>
>> > >> -
>> > >>
>> > >> post_sync_action = delete
>> > >> -
>> > >>
>> > >> post_sync_action = backup
>> > >> -
>> > >>
>> > >> retention_max_age
>> > >>
>> > >> The key idea is that source files should only be deleted or moved
>> after
>> > a
>> > >> checkpoint has completed, so that continuous file ingestion remains
>> safe
>> > >> and recoverable in production environments.
>> > >>
>> > >> 6.
>> > >>
>> > >> STIP-21 / StainTrace: End-to-end data lineage tracing and
>> performance
>> > >> analysis for Zeta
>> > >>
>> > >> David proposed including PR #10491 in the SeaTunnel 3.0 roadmap.
>> > >>
>> > >> -
>> > >>
>> > >> PR #10491:
>> > >> https://github.com/apache/seatunnel/pull/10491
>> > >>
>> > >> StainTrace is the implementation of STIP-21. It provides end-to-end
>> data
>> > >> lineage tracing and performance analysis for the Zeta engine.
>> > >>
>> > >> It adds lightweight timestamp sampling points at six key pipeline
>> > stages:
>> > >>
>> > >> -
>> > >>
>> > >> SOURCE_EMIT
>> > >> -
>> > >>
>> > >> QUEUE_IN
>> > >> -
>> > >>
>> > >> QUEUE_OUT
>> > >> -
>> > >>
>> > >> TRANSFORM_IN
>> > >> -
>> > >>
>> > >> TRANSFORM_OUT
>> > >> -
>> > >>
>> > >> SINK_WRITE_DONE
>> > >>
>> > >> The goal is to track the full journey of sampled rows from source
>> > >> emission to sink write completion, making it easier to measure
>> > end-to-end
>> > >> latency and identify bottlenecks.
>> > >>
>> > >> The proposal emphasizes production safety: the feature is disabled by
>> > >> default, supports configurable sampling, has a per-worker event
>> budget,
>> > >> writes trace data locally as JSONL in OpenTelemetry OTLP JSON format,
>> > and
>> > >> includes an offline analyzer for P95/P99 latency and HTML
>> visualization
>> > >> reports.
>> > >>
>> > >> 7.
>> > >>
>> > >> StateStore abstraction for Zeta engine
>> > >>
>> > >> Doyeon Kim proposed including PR #10812 in the SeaTunnel 3.0 roadmap.
>> > >>
>> > >> -
>> > >>
>> > >> PR #10812:
>> > >> https://github.com/apache/seatunnel/pull/10812
>> > >>
>> > >> This PR introduces an initial StateStore abstraction layer to reduce
>> > >> direct Hazelcast IMap dependencies in the Zeta engine. It makes state
>> > >> responsibilities more explicit and prepares the engine for future
>> > evolution
>> > >> around state storage, node communication, and coordination layers.
>> > >>
>> > >> 8.
>> > >>
>> > >> IMap external storage compaction
>> > >>
>> > >> Doyeon Kim also proposed including PR #10399.
>> > >>
>> > >> -
>> > >>
>> > >> PR #10399:
>> > >> https://github.com/apache/seatunnel/pull/10399
>> > >>
>> > >> This PR introduces LSM-style write and compaction logic for IMap
>> > external
>> > >> storage. It helps control storage growth by flushing sorted immutable
>> > files
>> > >> and compacting persisted files while keeping only the latest record
>> per
>> > key.
>> > >>
>> > >> The proposal is intended to improve stability for long-running Zeta
>> jobs
>> > >> by reducing duplicated records and tombstones in external storage.
>> > >>
>> > >> 9.
>> > >>
>> > >> AI-Native CLI and AI for Data Engineering
>> > >>
>> > >> sean zhang proposed including the AI-Native CLI capability in
>> SeaTunnel
>> > >> 3.0.
>> > >>
>> > >> -
>> > >>
>> > >> PR #10789:
>> > >> https://github.com/apache/seatunnel/pull/10789
>> > >>
>> > >> This PR introduces seatunnel-cli, an AI-powered command-line tool
>> that
>> > >> can generate production-ready pipeline configurations from natural
>> > language.
>> > >>
>> > >> Current capabilities include:
>> > >>
>> > >> -
>> > >>
>> > >> Multi-LLM provider support, including Anthropic, OpenAI, and AWS
>> > >> Bedrock
>> > >> -
>> > >>
>> > >> Runtime connector metadata extraction via Java OptionRule
>> reflection
>> > >> -
>> > >>
>> > >> Source/Sink type-aware validation
>> > >> -
>> > >>
>> > >> Auto-fix loop for generated configurations
>> > >> -
>> > >>
>> > >> Sensitive value redaction before persistence and LLM transmission
>> > >> -
>> > >>
>> > >> Interactive mode with session memory
>> > >> -
>> > >>
>> > >> One-shot mode for CI/CD
>> > >>
>> > >> The proposal also describes a broader “AI for Data Engineering”
>> roadmap,
>> > >> including intelligent schema mapping, natural language transforms,
>> data
>> > >> profiling, failure diagnosis, self-healing pipelines, anomaly
>> detection,
>> > >> performance tuning, cost-aware planning, intelligent scheduling, and
>> > >> eventually autonomous data infrastructure management.
>> > >>
>> > >> 10.
>> > >>
>> > >> Extension of the roadmap discussion window
>> > >>
>> > >> David proposed extending the SeaTunnel 3.0 roadmap discussion window
>> to
>> > >> May 15, 2026, considering the upcoming holiday period and to give
>> > >> contributors more time to share ideas.
>> > >>
>> > >> This message also recognized the quality and breadth of the
>> proposals so
>> > >> far, including StainTrace, AI-Native CLI, StateStore abstraction, and
>> > >> table-level fault isolation.
>> > >>
>> > >> 11.
>> > >>
>> > >> STIP-27: Apache Arrow as an optional internal data model
>> > >>
>> > >> xuepeng wang proposed adding STIP-27 to the SeaTunnel 3.0 roadmap
>> > >> discussion.
>> > >>
>> > >> -
>> > >>
>> > >> Issue #10806:
>> > >> https://github.com/apache/seatunnel/issues/10806
>> > >>
>> > >> This proposal introduces Apache Arrow as an optional internal data
>> > model,
>> > >> not as a replacement for the existing row model.
>> > >>
>> > >> The goal is to improve performance for columnar scenarios such as
>> > >> Parquet, ORC, ClickHouse, Doris, and batch transforms such as
>> > projection,
>> > >> filter, and cast.
>> > >>
>> > >> Expected benefits include better columnar I/O performance, vectorized
>> > >> execution, and lower JVM overhead.
>> > >>
>> > >> 12.
>> > >>
>> > >> STIP-24: Edge collection, EdgeSocket ingress, and SeaTunnel Edge
>> > Agent
>> > >>
>> > >> zhiwei niu proposed adding STIP-24 edge-collection work to the
>> SeaTunnel
>> > >> 3.0 roadmap.
>> > >>
>> > >> The original PRs were:
>> > >>
>> > >> -
>> > >>
>> > >> PR #10871: [Feature][Zeta] STIP-24 Phase 1: Support lightweight
>> > >> EdgeSocket ingress for edge collector MVP
>> > >> https://github.com/apache/seatunnel/pull/10871
>> > >> -
>> > >>
>> > >> PR #10876: [Feature][Zeta] STIP-24 Phase 2: Introduce standalone
>> > >> SeaTunnel Edge Agent MVP and dist integration
>> > >> https://github.com/apache/seatunnel/pull/10876
>> > >>
>> > >> Phase 1 introduces the engine-side ingress foundation for edge
>> > collection
>> > >> through EdgeSocket, including authenticated ingress, source-side
>> > protocol
>> > >> handling, and task-group address discovery support in Zeta.
>> > >>
>> > >> Phase 2 introduces a standalone SeaTunnel Edge Agent runtime,
>> including
>> > >> core, transport, connector, local file/log/event inputs, SQLite WAL
>> > >> durability, EdgeSocket transport interoperability, and distribution
>> > >> integration.
>> > >>
>> > >> The goal is to support an end-to-end edge collection workflow:
>> > >>
>> > >> edge host -> Zeta pipeline
>> > >>
>> > >> zhiwei later updated the PR references for STIP-24:
>> > >>
>> > >> -
>> > >>
>> > >> PR #10871 was updated to PR #10878
>> > >> https://github.com/apache/seatunnel/pull/10878
>> > >> -
>> > >>
>> > >> PR #10876 was updated to PR #10879
>> > >> https://github.com/apache/seatunnel/pull/10879
>> > >>
>> > >>
>> > >> 13.
>> > >>
>> > >> Prometheus metrics for Zeta Engine IMap state stores
>> > >>
>> > >> JeremyXin proposed including PR #10860 in the SeaTunnel 3.0 roadmap.
>> > >>
>> > >> -
>> > >>
>> > >> PR #10860:
>> > >> https://github.com/apache/seatunnel/pull/10860
>> > >>
>> > >> This PR improves Zeta Engine observability by exposing Prometheus
>> > metrics
>> > >> for core engine state stores backed by Hazelcast IMap.
>> > >>
>> > >> It covers internal stores such as running job information, job
>> states,
>> > >> checkpoints, and slot ownership. It exposes metrics such as local
>> entry
>> > >> counts, backup entry counts, and heap cost.
>> > >>
>> > >> The proposal aims to help users detect state growth, data skew, and
>> > >> memory pressure earlier, improving troubleshooting, capacity
>> planning,
>> > and
>> > >> production stability.
>> > >>
>> > >> 14.
>> > >>
>> > >> Real-time observability for Zeta engine tasks
>> > >>
>> > >> zhangshenghang proposed improving real-time observability for Zeta
>> > engine
>> > >> tasks.
>> > >>
>> > >> This direction adds the ability to monitor the runtime busyness
>> level of
>> > >> Zeta engine tasks, including real-time task metrics for Source,
>> > Transform,
>> > >> and Sink execution. It also includes queue and backpressure-related
>> > >> information exposed through REST API and Web UI.
>> > >>
>> > >> The goal is to help users understand whether a job is blocked by
>> source
>> > >> reading, transform processing, sink writing, or intermediate queues.
>> > >>
>> > >> 15.
>> > >>
>> > >> Error data bypass and dead-letter queue capability for Transform
>> and
>> > >> Sink
>> > >>
>> > >> zhangshenghang also proposed row-level error handling for Transform
>> and
>> > >> Sink stages.
>> > >>
>> > >> When configured, SeaTunnel can collect failed records with useful
>> > >> context, including:
>> > >>
>> > >> -
>> > >>
>> > >> error stage
>> > >> -
>> > >>
>> > >> plugin type
>> > >> -
>> > >>
>> > >> plugin name
>> > >> -
>> > >>
>> > >> source table
>> > >> -
>> > >>
>> > >> error message
>> > >> -
>> > >>
>> > >> exception class
>> > >> -
>> > >>
>> > >> original data
>> > >> -
>> > >>
>> > >> occurrence time
>> > >>
>> > >> This allows jobs to bypass selected dirty or error records instead of
>> > >> always failing fast, while keeping the default behavior unchanged
>> when
>> > the
>> > >> feature is not enabled.
>> > >>
>> > >> This is useful for long-running synchronization jobs where users need
>> > >> better fault tolerance and later inspection of bad data.
>> > >>
>> > >> 16.
>> > >>
>> > >> Improve cloud-native deployment experience
>> > >>
>> > >> zhangshenghang also suggested improving the cloud-native deployment
>> > >> experience for SeaTunnel 3.0.
>> > >>
>> > >> The proposal mentions Kubernetes and Helm support as important areas,
>> > >> including better guidance and behavior around:
>> > >>
>> > >> -
>> > >>
>> > >> deployment
>> > >> -
>> > >>
>> > >> restart
>> > >> -
>> > >>
>> > >> persistent checkpoint storage
>> > >> -
>> > >>
>> > >> configuration management
>> > >> -
>> > >>
>> > >> secret management
>> > >> -
>> > >>
>> > >> rolling upgrade
>> > >> -
>> > >>
>> > >> resource isolation
>> > >>
>> > >>
>> > >> 17.
>> > >>
>> > >> Clean up deprecated behavior carefully for 3.0
>> > >>
>> > >> zhangshenghang also suggested that SeaTunnel 3.0 is a good
>> opportunity
>> > to
>> > >> clean up deprecated options and old APIs.
>> > >>
>> > >> At the same time, any incompatible change should be handled
>> carefully,
>> > >> with clear migration documentation, examples, and release notes.
>> > >>
>> > >> Possible discussion groups
>> > >>
>> > >> Based on the above proposals, I think the roadmap discussion can be
>> > >> grouped into the following areas:
>> > >>
>> > >> 1.
>> > >>
>> > >> Zeta engine observability
>> > >>
>> > >>
>> > >> -
>> > >>
>> > >> STIP-21 / StainTrace
>> > >> -
>> > >>
>> > >> Prometheus metrics for IMap state stores
>> > >> -
>> > >>
>> > >> Real-time task metrics
>> > >> -
>> > >>
>> > >> Queue and backpressure visibility
>> > >> -
>> > >>
>> > >> REST API and Web UI exposure
>> > >>
>> > >>
>> > >> 2.
>> > >>
>> > >> Zeta state management and storage evolution
>> > >>
>> > >>
>> > >> -
>> > >>
>> > >> StateStore abstraction
>> > >> -
>> > >>
>> > >> IMap external storage compaction
>> > >> -
>> > >>
>> > >> IMap state metrics
>> > >> -
>> > >>
>> > >> Future storage backend evolution
>> > >>
>> > >>
>> > >> 3.
>> > >>
>> > >> Runtime and connector semantics
>> > >>
>> > >>
>> > >> -
>> > >>
>> > >> Engine-level timer flush for Sink connectors
>> > >> -
>> > >>
>> > >> JDBC adoption of timer flush
>> > >> -
>> > >>
>> > >> Checkpoint-gated file source lifecycle management
>> > >> -
>> > >>
>> > >> Table-level fault isolation
>> > >> -
>> > >>
>> > >> Row-level error bypass and dead-letter queue
>> > >>
>> > >>
>> > >> 4.
>> > >>
>> > >> AI for Data Engineering
>> > >>
>> > >>
>> > >> -
>> > >>
>> > >> AI-Native CLI
>> > >> -
>> > >>
>> > >> Natural language pipeline generation
>> > >> -
>> > >>
>> > >> Connector metadata reflection
>> > >> -
>> > >>
>> > >> Auto-fix loop
>> > >> -
>> > >>
>> > >> Security and sensitive value redaction
>> > >> -
>> > >>
>> > >> Future AI-assisted schema mapping, diagnosis, optimization, and
>> > >> autonomous data management
>> > >>
>> > >>
>> > >> 5.
>> > >>
>> > >> Edge collection
>> > >>
>> > >>
>> > >> -
>> > >>
>> > >> EdgeSocket ingress
>> > >> -
>> > >>
>> > >> SeaTunnel Edge Agent
>> > >> -
>> > >>
>> > >> Edge host to Zeta pipeline workflow
>> > >> -
>> > >>
>> > >> Authentication, local durability, and transport interoperability
>> > >>
>> > >>
>> > >> 6.
>> > >>
>> > >> Performance and internal data model
>> > >>
>> > >>
>> > >> -
>> > >>
>> > >> Apache Arrow as an optional internal data model
>> > >> -
>> > >>
>> > >> Columnar I/O
>> > >> -
>> > >>
>> > >> Vectorized execution
>> > >> -
>> > >>
>> > >> Lower JVM overhead
>> > >>
>> > >>
>> > >> 7.
>> > >>
>> > >> Build, release, and project structure
>> > >>
>> > >>
>> > >> -
>> > >>
>> > >> Independent seatunnel-shade project
>> > >> -
>> > >>
>> > >> Maven Central distribution of shade jars
>> > >> -
>> > >>
>> > >> Reduced build and release time
>> > >>
>> > >>
>> > >> 8.
>> > >>
>> > >> Cloud-native and compatibility
>> > >>
>> > >>
>> > >> -
>> > >>
>> > >> Kubernetes and Helm deployment experience
>> > >> -
>> > >>
>> > >> Persistent checkpoint storage
>> > >> -
>> > >>
>> > >> Secret management
>> > >> -
>> > >>
>> > >> Rolling upgrade
>> > >> -
>> > >>
>> > >> Resource isolation
>> > >> -
>> > >>
>> > >> Deprecated behavior cleanup
>> > >> -
>> > >>
>> > >> Migration documentation and release notes
>> > >>
>> > >> Next steps
>> > >>
>> > >> I suggest we continue to review these proposals and decide:
>> > >>
>> > >> 1.
>> > >>
>> > >> Which items should be included in the SeaTunnel 3.0 roadmap.
>> > >> 2.
>> > >>
>> > >> Which items should be marked as experimental.
>> > >> 3.
>> > >>
>> > >> Which items require an STIP or further design discussion.
>> > >> 4.
>> > >>
>> > >> Which items can be merged independently before the 3.0 release.
>> > >> 5.
>> > >>
>> > >> Which items need migration documentation, compatibility notes, or
>> > >> release notes.
>> > >>
>> > >> Again, this is only a summary of the current discussion thread.
>> Please
>> > >> help correct or supplement anything I missed.
>> > >>
>> > >>
>> > >>
>> > >> Best Regards,
>> > >> David
>> > >> Linkedin: https://www.linkedin.com/in/davidzollo
>> > >>
>> > >>
>> > >>
>> > >> On Thu, May 14, 2026 at 2:59 PM zhangshenghang <shenghang@apache.org
>> >
>> > >> wrote:
>> > >>
>> > >>> Hi community,
>> > >>>
>> > >>> For the SeaTunnel 3.0 roadmap
>> > >>>
>> > >>> I have already submitted two related PRs, and besides those, I would
>> > like
>> > >>> to propose the following directions for 3.0:
>> > >>>
>> > >>> 1. Real-time observability for Zeta engine tasks
>> > >>>
>> > >>> This work adds the ability to monitor the runtime busyness level of
>> > Zeta
>> > >>> engine tasks. It introduces real-time task metrics for Source,
>> > Transform,
>> > >>> and Sink execution, including queue/backpressure-related
>> information,
>> > and
>> > >>> exposes these metrics through REST API and Web UI. This can help
>> users
>> > >>> understand whether a job is blocked by source reading, transform
>> > >>> processing, sink writing, or intermediate queues, which is very
>> > important
>> > >>> for production troubleshooting and performance tuning.
>> > >>>
>> > >>> 2. Error data bypass and dead-letter queue capability for Transform
>> and
>> > >>> Sink
>> > >>>
>> > >>> This work introduces row-level error handling for Transform and Sink
>> > >>> stages. When configured, SeaTunnel can collect failed records with
>> > useful
>> > >>> context such as error stage, plugin type, plugin name, source table,
>> > >>> error
>> > >>> message, exception class, original data, and occurrence time. This
>> > allows
>> > >>> jobs to bypass selected dirty/error records instead of always
>> failing
>> > >>> fast,
>> > >>> while keeping the default behavior unchanged when the feature is not
>> > >>> enabled. This is useful for long-running synchronization jobs where
>> > users
>> > >>> need better fault tolerance and later inspection of bad data.
>> > >>>
>> > >>> 3. Improve cloud-native deployment experience
>> > >>>
>> > >>> Kubernetes and Helm support are important for new users. We should
>> > >>> improve
>> > >>> guidance and behavior around deployment, restart, persistent
>> checkpoint
>> > >>> storage, configuration management, secret management, rolling
>> upgrade,
>> > >>> and
>> > >>> resource isolation in cloud-native environments.
>> > >>>
>> > >>> 4. Clean up deprecated behavior carefully for 3.0
>> > >>>
>> > >>> Since 3.0 is a major version, it is a good opportunity to clean up
>> > >>> deprecated options and old APIs, but we should do this carefully.
>> Any
>> > >>> incompatible change should have clear migration documentation,
>> > examples,
>> > >>> and release notes.
>> > >>>
>> > >>>
>> > >>> Regards,
>> > >>> Jast (Shenghang)
>> > >>>
>> > >>>
>> > >>> JeremyXin <73...@qq.com> 于2026年5月14日周四 14:27写道：
>> > >>>
>> > >>> > Hi everyone, I’d like to propose adding this PR to the SeaTunnel
>> 3.0
>> > >>> > roadmap.
>> > >>> >
>> > >>> >
>> > >>> > - PR#10860
>> > >>> > - link: https://github.com/apache/seatunnel/pull/10860
>> > >>> >
>> > >>> >
>> > >>> > This PR improves Zeta Engine observability by exposing Prometheus
>> > >>> metrics
>> > >>> > for core engine state stores backed by Hazelcast IMap.
>> > >>> >
>> > >>> >
>> > >>> > It covers important internal stores such as running job
>> information,
>> > >>> job
>> > >>> > states, checkpoints, slot ownership and provides visibility into
>> > their
>> > >>> > local entry counts, backup entry counts, and heap cost.
>> > >>> >
>> > >>> >
>> > >>> > Including this in SeaTunnel 3.0 would make the engine easier to
>> > monitor
>> > >>> > and operate in production. It helps users detect state growth,
>> data
>> > >>> skew,
>> > >>> > and memory pressure earlier, which improves troubleshooting
>> > efficiency,
>> > >>> > cluster capacity planning, and overall operational stability.
>> > >>> >
>> > >>> >
>> > >>> > Best regards,
>> > >>> > JeremyXin
>> > >>> >
>> > >>> >
>> > >>> > 原始邮件
>> > >>> >
>> > >>> >
>> > >>> > 发件人：zhiwei niu <zniu70696@gmail.com>
>> > >>> > 发件时间：2026年5月12日 10:27
>> > >>> > 收件人：dev <dev@seatunnel.apache.org>
>> > >>> > 主题：Re: [DISCUSS] Apache SeaTunnel 3.0 Roadmap
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> > Hi everyone,
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> As part of Roadmap 3.0 progress tracking, the STIP-24 PR links have been
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> updated to keep references consistent and easier to follow.
>> > >>> >
>> > >>> >
>> Please use the updated PR links below:
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> [Feature][Zeta] STIP-24 Phase 1: Support lightweight EdgeSocket ingress
>> > >>> > for edge collector MVP
>> > >>> > - PR #10871(
>> > >>> > https://github.com/apache/seatunnel/pull/10871)
>> > >>> > —> PR#10878(
>> > >>> > https://github.com/apache/seatunnel/pull/10878)
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> [Feature][Zeta] STIP-24 Phase 2: Introduce standalone SeaTunnel Edge Agent
>> > >>> > MVP and dist integration
>> > >>> > - PR #10876(
>> > >>> > https://github.com/apache/seatunnel/pull/10876)
>> > >>> > — PR#10879(
>> > >>> > https://github.com/apache/seatunnel/pull/10879)
>> > >>> >
>> > >>> >
>> > >>> > zhiwei niu <zniu70696@gmail.com
>> > >>> > > 于2026年5月12日周二 01:31写道：
>> > >>> >
>> > >>> > > Hi everyone,
>> > >>> > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> > In addition to STIP-23, I’d like to propose the STIP-24 edge-collection
>> > >>> > > work for SeaTunnel 3.0.
>> > >>> > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> > - PR #10871 — [Feature][Zeta] STIP-24 Phase 1: Support lightweight
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> > EdgeSocket ingress for edge collector MVP
>> > >>> > > https://github.com/apache/seatunnel/pull/10871
>> > >>> > > What it adds:
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> > Introduces the engine-side ingress foundation for edge collection via
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> > EdgeSocket, including authenticated ingress, source-side protocol handling,
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> > and task-group address discovery support in Zeta.
>> > >>> > > Why 3.0:
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> > This is the foundational engine capability for remote edge data collection
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> > scenarios. It is backward-compatible and establishes the protocol/runtime
>> > >>> >
>> > >>>
>> >
>> > path required for the edge-side agent.
>> > >>> > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> > - PR #10876 — [Feature][Zeta] STIP-24 Phase 2: Introduce standalone
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> > SeaTunnel Edge Agent MVP and dist integration
>> > >>> > > https://github.com/apache/seatunnel/pull/10876
>> > >>> > > What it adds:
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> > Introduces a standalone SeaTunnel Edge Agent runtime
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> > (core/transport/connector), local file/log/event inputs, SQLite WAL
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> > durability, EdgeSocket transport interop, and distribution integration.
>> > >>> > > Why 3.0:
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> > This delivers the first practical end-to-end edge collection workflow
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> > (edge host -> Zeta pipeline), which significantly expands SeaTunnel’s
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> > deployment topology and real-world ingestion coverage.
>> > >>> > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> > PR #10876 depends on #10871; both are under active review.
>> > >>> > >
>> > >>> > > Best regards,
>> > >>> > > zhiwei.niu
>> > >>> > >
>> > >>> > > xuepeng wang <wxuepeng5@gmail.com
>> > >>> > > 于2026年5月8日周五 14:03写道：
>> > >>> > >
>> > >>> > >> Hi team,
>> > >>> > >>
>> > >>> > >>
>> > >>> > >>
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> I'd like to add STIP-27 [1] to the SeaTunnel 3.0 Roadmap discussion.
>> > >>> > >>
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> This proposal introduces Apache Arrow as an optional internal data model
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> (not replacing the existing row model). It aims to improve performance for
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> columnar scenarios (Parquet/ORC, ClickHouse/Doris) and batch transforms
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> (projection/filter/cast), while keeping the current architecture stable.
>> > >>> > >>
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> Key benefits: better columnar I/O performance, vectorized execution,
>> > >>> > >> lower JVM overhead.
>> > >>> > >>
>> > >>> > >> [1]
>> > >>> > https://github.com/apache/seatunnel/issues/10806
>> > >>> > >> <https://github.com/apache/seatunnel/issues/10806>>
>> >;
>> > >>> > >> Thoughts?
>> > >>> > >>
>> > >>> > >> Regards,
>> > >>> > >> xuepeng(github id:chl-wxp)
>> > >>> > >>
>> > >>> > >> David Zollo <davidzollo365@gmail.com
>> > >>> > > 于2026年4月30日周四 21:17写道：
>> > >>> > >>
>> > >>> > >> > Hi everyone,
>> > >>> > >> >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > First of all, thank you all for the fantastic proposals so far — the
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > quality and breadth of ideas in this thread have been genuinely
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > exciting to read. From StainTrace to the AI-Native CLI, from
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > StateStore abstraction to table-level fault isolation, it's clear that
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > this community has a bold and thoughtful vision for what SeaTunnel 3.0
>> > >>> > >> > can become.
>> > >>> > >> >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > Given that the Golden Week holiday is just around the corner, and many
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > of our contributors are based in China or taking time off during this
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > period, I'd like to extend the discussion window to May 15th. This
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > gives everyone enough breathing room to enjoy the holiday, come back
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > refreshed, and still have time to share their thoughts without feeling
>> > >>> > >> > rushed.
>> > >>> > >> >
>> > >>> >
>> > >>>
>> > >> > Happy Labor Day to everyone!
>> > >>> > >> >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > We highly welcome everyone's suggestions for version 3.0! Feel free to
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > leave a message if you have any ideas.
>> > >>> > >> >
>> > >>> > >> >
>> > >>> > >> >
>> > >>> > >> > Best Regards,
>> > >>> > >> > David
>> > >>> > >> > Linkedin:
>> > >>> > https://www.linkedin.com/in/davidzollo
>> > >>> > >> >
>> > >>> > <https://www.linkedin.com/in/davidzollo>> >>;
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > On Wed, Apr 29, 2026 at 8:50 PM sean zhang <
>> > >>> > seanzhang92@gmail.com>
>> > >>> > >> wrote:
>> > >>> > >> > >
>> > >>> >
>> >> > > Hi SeaTunnel community,
>> > >>> > >> > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > I'd like to propose including the AI-Native CLI capability in the
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > SeaTunnel 3.0 release, and share a broader vision for the "AI for Data
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > Engineering" direction. I believe this aligns well with the 3.0
>> > >>> > >> milestone
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > as a significant user-experience leap.
>> > >>> > >> > > This PR
>> > >>> > https://github.com/apache/seatunnel/pull/10789 introduces
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > `seatunnel-cli`, an AI-powered command-line tool that
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > generates production-ready pipeline configurations from natural
>> > >>> > >> language
>> > >>> > <
>> > >>>
>> >
>> https://github.com/apache/seatunnel/pull/10789 introduces>> > > seatunnel-cli, an AI-powered command-line tool that>> > > generates production-ready pipeline configurations from natural>> language
>> > >>> >
>> > >>> > :
>> > >>> > >> > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > $ seatunnel "Sync the users table from MySQL to S3 in parquet
>> > >>> > >> format"
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > ✅ Generated config: mysql_users_to_s3.conf
>> > >>> > >> > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > Key capabilities in the current implementation:
>> > >>> > >> > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > - Multi-LLM provider support (Anthropic / OpenAI / AWS Bedrock)
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > with auto-detection
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > - Runtime connector metadata extraction via Java OptionRule
>> > >>> > >> reflection,
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > ensuring 100% accurate parameter knowledge for all 100+ connectors
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > - Source/Sink type-aware validation (Jdbc source options ≠ Jdbc sink
>> > >>> > >> > > options)
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > - Auto-fix loop: generated configs are validated and self-corrected
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > (up to 3 iterations)
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > - Security hardening: sensitive values (passwords, tokens, keys) are
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > automatically redacted to ${ENV_VAR} placeholders before
>> > >>> > >> persistence
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > and LLM transmission
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > - Interactive mode with session memory and one-shot mode for CI/CD
>> > >>> > >> > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > Future Roadmap — AI for Data Engineering
>> > >>> > >> > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > Looking beyond config generation, I see a multi-level roadmap where
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > AI becomes deeply integrated into SeaTunnel's core value proposition:
>> > >>> > >> > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > ### Level 2: AI Understands Data (Next phase)
>> > >>> > >> > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > - Intelligent Schema Mapping: AI reads source schema, infers target
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > schema, and auto-generates type conversions and DDL
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > - Natural Language Transforms: "mask phone numbers, merge first and
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > last name, filter last 7 days" → SQL Transform auto-generation
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > - Data Profiling: AI samples source data, detects anomalies (NULLs,
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > outliers, format violations), and suggests data quality transforms
>> > >>> > >> > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > ### Level 3: AI Operates Pipelines
>> > >>> > >> > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > - Intelligent Failure Diagnosis: AI analyzes error logs, pinpoints
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > root cause (connection refused → MySQL down vs firewall), suggests
>> > >>> > >> > fix
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > - Self-Healing Pipelines: auto-detect DDL changes in CDC sources,
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > auto-evolve sink schema, zero-downtime recovery
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > - Anomaly Detection: monitor data flow volume/latency, alert when
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > throughput drops 99% with correlated root cause
>> > >>> > >> > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > ### Level 4: AI Optimizes Everything
>> > >>> > >> > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > - Performance Tuning: AI analyzes bottlenecks (batch_size too small,
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > wrong file format, insufficient parallelism), suggests
>> > >>> > >> optimization
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > - Cost-Aware Planning: given N tables to sync, AI proposes pipeline
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > topology that balances cost vs latency vs resource usage
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > - Intelligent Scheduling: learn from historical patterns to optimize
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > sync frequency per table
>> > >>> > >> > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > ### Level 5: Autonomous Data Platform (Long-term vision)
>> > >>> > >> > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > - Conversational data infrastructure management: "onboard the
>> > >>> > >> payment
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > system to the data lake, ensure GDPR compliance" → AI
>> > >>> > >> auto-discovers
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > sources, identifies PII, generates pipelines with masking
>> > >>> > >> transforms,
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > deploys with monitoring and alerting
>> > >>> > >> > >
>> > >>> > >> > >
>> > >>> > >> > > Doyeon Kim <
>> > >>> > ekagnlek2@gmail.com> 于2026年4月29日周三 18:17写道：
>> > >>> > >> > >
>> > >>> > >> > > > Hi everyone,
>> > >>> > >> > > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > I’d like to propose the following Zeta engine improvements for
>> > >>> > >> > SeaTunnel
>> > >>> > >> > > > 3.0.
>> > >>> > >> > > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > 1. StateStore abstraction
>> > >>> > >> > > > PR:
>> > >>> > https://github.com/apache/seatunnel/pull/10812
>> > >>> > >> > > >
>> > >>> > <
>> > >>>
>> >
>> https://github.com/apache/seatunnel/pull/10812>> > > >
>> > >>> >
>> > >>> > ;
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > This introduces an initial StateStore abstraction layer to reduce
>> > >>> > >> > direct
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > Hazelcast IMap dependencies in the engine. It makes different state
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > responsibilities more explicit and prepares the engine for future
>> > >>> > >> > backend
>> > >>> > >> > > > evolution.
>> > >>> > >> > > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > I think this is relevant for 3.0 because it is a foundational step
>> > >>> > >> > toward
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > decoupling the engine from Hazelcast-specific APIs and enabling
>> > >>> > >> future
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > improvements around state storage, node communication, and
>> > >>> > >> coordination
>> > >>> > >> > > > layers.
>> > >>> > >> > > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > 2. IMap external storage compaction
>> > >>> > >> > > > PR:
>> > >>> > https://github.com/apache/seatunnel/pull/10399
>> > >>> > >> > > >
>> > >>> > <
>> > >>>
>> >
>> https://github.com/apache/seatunnel/pull/10399>> > > >
>> > >>> >
>> > >>> > ;
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > This introduces LSM-style write and compaction logic for IMap
>> > >>> > >> external
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > storage. It helps control storage growth by flushing sorted
>> > >>> > >> immutable
>> > >>> > >> > files
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > and compacting persisted files while keeping only the latest record
>> > >>> > >> per
>> > >>> > >> > > > key.
>> > >>> > >> > > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > I think this should be considered for 3.0 because long-running Zeta
>> > >>> > >> > jobs
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > may accumulate duplicated records and tombstones in external storage
>> > >>> > >> > over
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > time. Compaction can improve stability and reduce the risk of
>> > >>> > >> unbounded
>> > >>> > >> > > > storage growth.
>> > >>> > >> > > >
>> > >>> > >> > > > Best Regards,
>> > >>> > >> > > > Doyeon
>> > >>> >
>> > >>>
>> >
>> >> > > > (Github ID: dybyte)
>> > >>> > >> > > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > On Wed, Apr 29, 2026 at 7:13 PM David Zollo <
>> > >>> > >> davidzollo365@gmail.com>
>> > >>> > >> > > > wrote:
>> > >>> > >> > > >
>> > >>> >
>> >> > > > > Hi guys,
>> > >>> > >> > > > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > I'd like to propose including PR #10491 in the SeaTunnel 3.0
>> > >>> > >> roadmap,
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > here are the details with additional packaging time analysis:
>> > >>> > >> > > > >
>> > >>> >
>> > >>>
>> >
>> >> > > > > ### PR link
>> > >>> > >> > > > >
>> > >>> > https://github.com/apache/seatunnel/pull/10491
>> > >>> > >> > > > >
>> > >>> > <
>> > >>>
>> >
>> https://github.com/apache/seatunnel/pull/10491>> > > > >
>> > >>> >
>> > >>> > ;
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > StainTrace (PR #10491) is the concrete implementation of STIP-21,
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > delivering end-to-end data lineage tracing and performance
>> > >>> > >> analysis
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > capabilities for the Apache SeaTunnel Zeta engine. It injects
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > lightweight timestamp sampling points at 6 key pipeline stages —
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > SOURCE_EMIT → QUEUE_IN → QUEUE_OUT → TRANSFORM_IN → TRANSFORM_OUT
>> > >>> > >> →
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > SINK_WRITE_DONE — tracking each sampled row's complete journey
>> > >>> > >> from
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > source emission to sink write completion, enabling precise
>> > >>> > >> end-to-end
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > latency measurement and bottleneck identification.
>> > >>> > >> > > > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > The system is designed with production safety as a first-class
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > concern: it is disabled by default with very low overhead, and
>> > >>> > >> when
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > enabled it operates at a configurable sampling rate with a strict
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > per-worker per-second event budget, achieving approximately 0.1~1%
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > overhead at 1% sampling. Trace data is written locally as JSONL
>> > >>> > >> files
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > in OpenTelemetry OTLP JSON format with no external dependencies.
>> > >>> > >> An
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > accompanying offline analyzer reads these files to compute P95/P99
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > latency statistics, identify bottleneck stages, and generate an
>> > >>> > >> HTML
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > visualization report with Chinese/English language toggle support.
>> > >>> > >> > > > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > The value to SeaTunnel lies in closing a critical observability
>> > >>> > >> gap
>> > >>> > >> > at
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > the engine level. Before this feature, users facing latency
>> > >>> > >> spikes or
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > throughput bottlenecks had no structured way to pinpoint whether
>> > >>> > >> the
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > problem resided in queue backpressure, transform processing, or
>> > >>> > >> sink
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > write time — they could only rely on logs and guesswork.
>> > >>> > >> StainTrace
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > makes per-stage timing transparent and actionable, which is
>> > >>> > >> > especially
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > impactful for large-scale, multi-table, long-chain production
>> > >>> > >> > > > > pipelines.
>> > >>> > >> > > > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > The case for including it in the 3.0 roadmap is equally strong. As
>> > >>> > >> > the
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > implementation of STIP-21, StainTrace is a framework-level
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > observability capability — connector-agnostic and immediately
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > effective across all jobs once merged. One of SeaTunnel 3.0's
>> > >>> > >> central
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > goals is to become a more mature, production-ready data
>> > >>> > >> integration
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > engine, and observability is an indispensable foundation for that
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > goal. Establishing it at a major version boundary like 3.0
>> > >>> > >> delivers
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > far greater long-term value to users.
>> > >>> > >> > > > >
>> > >>> > >> > > > >
>> > >>> > >> > > > >
>> > >>> > >> > > > >
>> > >>> > >> > > > >
>> > >>> > >> > > > >
>> > >>> >
>> > >>>
>> > >> > > > > Best Regards,
>> > >>> > >> > > > > David
>> > >>> >
>> >> > > > > Linkedin:
>> > >>> > https://www.linkedin.com/in/davidzollo
>> > >>> > >> > > > >
>> > >>> > <
>> > >>>
>> >
>> https://www.linkedin.com/in/davidzollo>> > > > >
>> > >>> >
>> > >>> > ;
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > On Wed, Apr 29, 2026 at 2:48 PM yi zeng <
>> > >>> > yzeng1618@gmail.com>
>> > >>> > >> wrote:
>> > >>> > >> > > > > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > Hi everyone,
>> > >>> > >> > > > > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > I’d like to propose including the following two PRs in the
>> > >>> > >> > *SeaTunnel
>> > >>> > >> > > > 3.0
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > roadmap*. Both focus on strengthening *correctness, reliability,
>> > >>> > >> > and
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > operational safety* for core multi-table and continuous
>> > >>> > >> ingestion
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > scenarios, which I believe are key pillars for 3.0.
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > ------------------------------
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > PR #10600 — [Feature][Core] Support table-level fault isolation
>> > >>> > >> for
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > multi-table synchronization
>> > >>> > >> > > > > >
>> > >>> > >> > > > > >
>> > >>> > https://github.com/apache/seatunnel/pull/10600
>> > >>> > >> > > > > >
>> > >>> > <
>> > >>>
>> >
>> https://github.com/apache/seatunnel/pull/10600>> > > > > >
>> > >>> >
>> > >>> > ;
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > *What it adds / solves*
>> > >>> > >> > > > > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > This PR introduces *table-level fault isolation* for multi-table
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > synchronization by adding a new failure policy:
>> > >>> >
>> >> > > > > > code
>> > >>> > >> > > > > > 复制
>> > >>> > >> > > > > >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > env {
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > multi_table {
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > failure_policy = "CONTINUE_OTHER_TABLES"
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > }
>> > >>> > >> > > > > > }
>> > >>> > >> > > > > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > With this enabled:
>> > >>> > >> > > > > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > - Failures are isolated at *table granularity*, instead of
>> > >>> > >> > failing
>> > >>> > >> > > > the
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > entire job.
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > - Healthy tables continue processing even if one table fails.
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > - Failed tables are collected and reported with clear
>> > >>> > >> metadata
>> > >>> > >> > > > (table
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > path, failure phase, plugin, error).
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > - Batch jobs report failed tables in final results; streaming
>> > >>> > >> > jobs
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > continue running with healthy tables.
>> > >>> > >> > > > > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > This addresses a long-standing pain point where a single
>> > >>> > >> > problematic
>> > >>> > >> > > > > table
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > can block progress for all others.
>> > >>> > >> > > > > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > *Why it should be included in 3.0*
>> > >>> > >> > > > > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > - *Improved fault tolerance model*: 3.0 aims to be more
>> > >>> > >> > resilient
>> > >>> > >> > > > and
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > production-ready; table-level isolation is a foundational
>> > >>> > >> > capability
>> > >>> > >> > > > > for
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > large-scale, multi-table pipelines.
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > - *Backward-compatible by default*: The default behavior
>> > >>> > >> remains
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > FAIL_FAST, so existing jobs are unaffected unless users
>> > >>> > >> > explicitly
>> > >>> > >> > > > opt
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > in.
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > - *Framework-level improvement*: This is not
>> > >>> > >> > connector-specific; it
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > enhances the core execution and failure-handling semantics of
>> > >>> > >> > > > > SeaTunnel.
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > - *Mature design & coverage*: Includes core logic changes,
>> > >>> > >> > runtime
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > handling, reporting, and E2E tests, making it suitable for a
>> > >>> > >> > major
>> > >>> > >> > > > > version
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > milestone.
>> > >>> > >> > > > > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > ------------------------------
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > PR #10563 — [Feature][Connector-V2] Add checkpoint-gated
>> > >>> > >> > > > post_sync_action
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > and retention for file source continuous discovery
>> > >>> > >> > > > > >
>> > >>> > >> > > > > >
>> > >>> > https://github.com/apache/seatunnel/pull/10563
>> > >>> > >> > > > > >
>> > >>> > <
>> > >>>
>> >
>> https://github.com/apache/seatunnel/pull/10563>> > > > > >
>> > >>> >
>> > >>> > ;
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > *What it adds / solves*
>> > >>> > >> > > > > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > This PR adds *checkpoint-gated source-side lifecycle management*
>> > >>> > >> > for
>> > >>> > >> > > > file
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > source continuous discovery, including:
>> > >>> > >> > > > > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > - post_sync_action = delete
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > Delete source files only *after a checkpoint has completed*
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > - post_sync_action = backup
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > Move source files to a backup_path after checkpoint
>> > >>> > >> completion
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > - Optional backup retention via retention_max_age
>> > >>> > >> > > > > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > All operations are *checkpoint-aware*, ensuring source files are
>> > >>> > >> > never
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > deleted or moved before the data is durably processed.
>> > >>> > >> > > > > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > *Why it should be included in 3.0*
>> > >>> > >> > > > > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > - *Correctness-first continuous ingestion*: Safe source
>> > >>> > >> cleanup
>> > >>> > >> > tied
>> > >>> > >> > > > > to
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > checkpoint completion is essential for exactly-once and
>> > >>> > >> > recoverable
>> > >>> > >> > > > > file
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > ingestion, which aligns strongly with 3.0 quality goals.
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > - *Production-grade operational semantics*: Without
>> > >>> > >> > checkpoint-gated
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > cleanup, continuous file ingestion is risky in real
>> > >>> > >> production
>> > >>> >
>> >> > > > > environments.
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > - *Extends Connector-V2 maturity*: This PR deepens the
>> > >>> > >> > operational
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > completeness of Connector-V2, especially for file-based
>> > >>> > >> sources.
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > - *Well-tested and documented*: Includes unit tests, E2E
>> > >>> > >> tests
>> > >>> > >> > > > (Local
>> > >>> > >> > > > > +
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > HDFS), and documentation updates (EN + ZH).
>> > >>> > >> > > > > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > ------------------------------
>> > >>> > >> > > > > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > Both PRs are *opt-in*, backward-compatible, and focused on
>> > >>> > >> making
>> > >>> > >> > > > > SeaTunnel
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > more robust, predictable, and safe in real-world deployments. I
>> > >>> > >> > believe
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > they fit naturally into the SeaTunnel 3.0 scope, especially
>> > >>> > >> given
>> > >>> > >> > the
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > emphasis on stability, correctness, and large-scale usage.
>> > >>> > >> > > > > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > Happy to discuss details or answer questions.
>> > >>> > >> > > > > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > Zhang Donghao <
>> > >>> > zhangdonghao@apache.org> 于2026年4月29日周三 14:38写道：
>> > >>> > >> > > > > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > Hi everyone,
>> > >>> >
>> >> > > > > > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > I'd like to propose including PR #9993 in the SeaTunnel 3.0
>> > >>> > >> > roadmap,
>> > >>> > >> > > > > here
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > are the details with additional packaging time analysis:
>> > >>> >
>> >> > > > > > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > ### PR link
>> > >>> >
>> > >>>
>> >
>> >> > > > > > >
>> > >>> > https://github.com/apache/seatunnel/pull/9993
>> > >>> >
>> >> > > > > > >
>> > >>> > <
>> > >>>
>> >
>> https://github.com/apache/seatunnel/pull/9993>> > > > > > >
>> > >>> >
>> > >>> > ;
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > ### Short description
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > This PR extracts the shade modules into an independent
>> > >>> > >> > > > seatunnel-shade
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > project, enables uploading shade-jars to the central
>> > >>> > >> repository,
>> > >>> > >> > and
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > optimizes the overall packaging efficiency of SeaTunnel.
>> > >>> >
>> >> > > > > > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > ### Packaging time analysis (current state before PR
>> > >>> > >> > optimization)
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > The current build time breakdown of individual shade modules
>> > >>> > >> is
>> > >>> > >> > as
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > follows, highlighting the inefficiency of the integrated build
>> > >>> > >> > > > > approach:
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > ```
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > [INFO] SeaTunnel : Shade : 3.0.0 ..........................
>> > >>> > >> > SUCCESS [
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > 12.661 s]
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > [INFO] SeaTunnel : Shade : Hazelcast 5.1-3.0.0 ............
>> > >>> > >> > SUCCESS [
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > 8.192 s]
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > [INFO] SeaTunnel : Shade : Hadoop3 Uber 3.1.4-3.0.0 .......
>> > >>> > >> > SUCCESS
>> > >>> > >> > > > > [02:29
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > min]
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > [INFO] SeaTunnel : Shade : Jackson 2.13.3-3.0.0 ...........
>> > >>> > >> > SUCCESS [
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > 3.289 s]
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > [INFO] SeaTunnel : Shade : Guava 27.0-jre-3.0.0 ...........
>> > >>> > >> > SUCCESS [
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > 2.960 s]
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > [INFO] SeaTunnel : Shade : Hadoop : AWS 3.1.4-3.0.0 .......
>> > >>> > >> > SUCCESS
>> > >>> > >> > > > > [05:05
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > min]
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > [INFO] SeaTunnel : Shade : Arrow 15.0.1-3.0.0 .............
>> > >>> > >> > SUCCESS [
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > 5.880 s]
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > [INFO] SeaTunnel : Shade : Hikari 4.0.3-3.0.0 .............
>> > >>> > >> > SUCCESS [
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > 1.085 s]
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > [INFO] SeaTunnel : Shade : Thrift-Service 1.0.0-3.0.0 .....
>> > >>> > >> > SUCCESS [
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > 1.572 s]
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > [INFO] SeaTunnel : Shade : Janino 3.0.11-3.0.0 ............
>> > >>> > >> > SUCCESS [
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > 1.235 s]
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > [INFO] SeaTunnel : Shade : Scala 2.13.11-3.0.0 ............
>> > >>> > >> > SUCCESS [
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > 6.586 s]
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > [INFO] SeaTunnel : Shade : Jetty 9.4.56.v20240826-3.0.0 ...
>> > >>> > >> > SUCCESS [
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > 2.421 s]
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > [INFO] SeaTunnel : Shade : Commons Lang3 3.18.0-3.0.0 .....
>> > >>> > >> > SUCCESS [
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > 2.044 s]
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > ```
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > Key observations from the time breakdown:
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > 1. **Long-tail modules dominate build time**: Two modules
>> > >>> > >> > account for
>> > >>> > >> > > > > the
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > vast majority of shade build time — `Hadoop3 Uber` (2 minutes
>> > >>> > >> 29
>> > >>> > >> > > > > seconds)
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > and `Hadoop : AWS` (5 minutes 5 seconds), totaling ~7.5
>> > >>> > >> minutes,
>> > >>> > >> > > > while
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > other modules take only a few seconds each.
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > 2. **Integrated build waste**: In the current structure, these
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > time-consuming shade modules are built alongside core modules
>> > >>> > >> > every
>> > >>> > >> > > > > time
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > SeaTunnel is packaged, even when there are no changes to shade
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > dependencies. This results in repeated long build cycles for
>> > >>> >
>> > >>>
>> >
>> >> > > > > developers and
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > release workflows.
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > 3. **Total shade build time**: The cumulative build time of
>> > >>> > >> all
>> > >>> > >> > shade
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > modules is approximately 8 minutes (excluding
>> > >>> > >> > overlapping/parallel
>> > >>> > >> > > > > build
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > optimizations), which is a significant portion of the overall
>> > >>> > >> > > > SeaTunnel
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > packaging process.
>> > >>> >
>> >> > > > > > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > ### Reason for including in 3.0
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > 1. **Build efficiency optimization (quantifiable gains)**:
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > - By isolating shade modules into an independent project
>> > >>> > >> and
>> > >>> > >> > > > > uploading
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > shade-jars to the central repository, we eliminate the need to
>> > >>> > >> > > > rebuild
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > these modules in every SeaTunnel packaging cycle. For daily
>> > >>> > >> > > > > development,
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > this reduces the total build time by ~8 minutes per build (for
>> > >>> > >> > > > > developers
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > not modifying shade code).
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > - For the 3.0 release process, we can pre-build and publish
>> > >>> > >> > > > > shade-jars
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > to the central repository once, then reference them as
>> > >>> > >> > dependencies —
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > cutting the release packaging time by a substantial margin and
>> > >>> >
>> > >> > > > > reducing the
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > risk of build failures caused by long-running shade module
>> > >>> > >> > builds.
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > 2. **Dependency management enhancement**: Uploading shade-jars
>> > >>> > >> > to the
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > central repository standardizes dependency distribution,
>> > >>> > >> avoiding
>> > >>> > >> > > > > conflicts
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > caused by internal shade module duplication across
>> > >>> > >> subprojects,
>> > >>> > >> > and
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > improving the stability of the 3.0 version's dependency
>> > >>> > >> system.
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > 3. **Project structure rationalization**: Isolating shade
>> > >>> > >> modules
>> > >>> > >> > > > > aligns
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > with the modular design concept of 3.0, making the project
>> > >>> > >> > structure
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > clearer, easier to maintain, and facilitating future
>> > >>> > >> expansion of
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > shade-related capabilities (e.g., adding new shade modules for
>> > >>> > >> > > > > additional
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > dependencies).
>> > >>> >
>> >> > > > > > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > This optimization directly addresses the identified packaging
>> > >>> > >> > > > > bottlenecks
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > (long-tail shade modules) and delivers measurable time savings
>> > >>> > >> > for
>> > >>> > >> > > > both
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > development and release workflows — a critical improvement for
>> > >>> > >> > the
>> > >>> > >> > > > > smooth
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > delivery and maintainability of SeaTunnel 3.0.
>> > >>> >
>> >> > > > > > >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > Best regards,
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> >> > > > > > > zhangdonghao
>> > >>> >
>> >> > > > > > >
>> > >>> > >> > > > >
>> > >>> > >> > > >
>> > >>> > >> >
>> > >>> > >>
>> > >>> > >
>> > >>>
>> > >>
>> >
>>
>
Re: [DISCUSS] Apache SeaTunnel 3.0 Roadmap
Posted by David Zollo <da...@gmail.com>.
Hi guys,
Thanks for your hard work and make a great progress. Now we only have four
PRs left to be reviewed and merged.
Below is the list of remaining PRs pending merge in the Roadmap:
1. [#10563](https://github.com/apache/seatunnel/pull/10563)
[Feature][Connector-V2] Add checkpoint-gated post_sync_action and retention
for file source continuous discovery @yzeng1618
2. [#9993](https://github.com/apache/seatunnel/pull/9993)
[Feature][shade]Refactor the seatunnel-shade module. @hawk9821
3. [#10879](https://github.com/apache/seatunnel/pull/10879) [Feature][Zeta]
STIP-24 Phase 2: Introduce standalone SeaTunnel Edge Agent MVP and dist
integration @nzw921rx
4. [#10306](https://github.com/apache/seatunnel/pull/10306) [Feature][Core]
Error Data Bypass Capability for Transform & Sink (Dead Letter Queue)
@zhangshenghang
Guys, please help review the PR for the 3.0 Roadmap when you have time.
Best Regards,
David
Linkedin: https://www.linkedin.com/in/davidzollo
On Thu, Jul 9, 2026 at 10:56 PM zhiwei niu <zn...@gmail.com> wrote:
> Hi community,
>
> I would like to propose including STIP-28 in the SeaTunnel 3.0 roadmap.
>
> STIP-28 enhances SeaTunnel's OptionRule framework from basic
> presence/equality validation to richer declarative configuration
> validation. This allows connector and transform configuration constraints
> to be expressed as structured metadata instead of being hidden in scattered
> runtime if/throw logic.
>
> Related PRs:
>
> - Main PR: apache/seatunnel#10977
> https://github.com/apache/seatunnel/pull/10977
>
> Completes the rich OptionRule validation framework.
>
> - Follow-up PR: apache/seatunnel#11228
> https://github.com/apache/seatunnel/pull/11228
>
> Preserves extension validation error context.
>
> - Follow-up PR: apache/seatunnel#11121
> https://github.com/apache/seatunnel/pull/11121
>
> Fixes validation error aggregation.
>
> - Follow-up PR: apache/seatunnel#11010
> https://github.com/apache/seatunnel/pull/11010
>
> Adds Map-type condition validation.
>
> - Follow-up PR: apache/seatunnel#11022
> https://github.com/apache/seatunnel/pull/11022
>
> Supports combining structural and value rules.
>
> - Follow-up PR: apache/seatunnel#11048
> https://github.com/apache/seatunnel/pull/11048
>
> Adds custom extension validation support.
>
> Why this should be included in 3.0:
>
> - It makes SeaTunnel's configuration contract clearer and more explicit.
> - Users can catch more configuration errors earlier through --check.
> - REST API, Web UI, CLI metadata export, and future AI-assisted config
> generation can consume richer validation metadata.
> - Connector and transform developers can reduce duplicated imperative
> validation logic.
> - It provides a stronger foundation for future configuration generation,
> validation, explanation, and repair capabilities.
>
> STIP-28 framework capabilities have already been merged, and some
> connectors have already completed migration. Therefore, I believe STIP-28
> is ready to be included in the SeaTunnel 3.0 roadmap.
>
> Best Regards,
> zhiwei.niu
>
> David Zollo <da...@gmail.com> 于2026年7月9日周四 21:19写道：
>
> > Hi guys,
> >
> > Thanks for your hard work and make a great progress. Now we only have the
> > following PRs left to be reviewed and merged.
> > Below is the list of remaining PRs pending merge in the Roadmap:
> >
> > 1. [#10563](https://github.com/apache/seatunnel/pull/10563)
> > [Feature][Connector-V2] Add checkpoint-gated post_sync_action and
> retention
> > for file source continuous discovery @yzeng1618 PR conflicts and need to
> > solve
> > 2. [#9993](https://github.com/apache/seatunnel/pull/9993)
> > [Feature][shade]Refactor the seatunnel-shade module. @hawk9821 PR
> > conflicts and need to solve
> > 3. [#10879](https://github.com/apache/seatunnel/pull/10879)
> > [Feature][Zeta]
> > STIP-24 Phase 2: Introduce standalone SeaTunnel Edge Agent MVP and dist
> > integration @nzw921rx
> > 4. [#10800](https://github.com/apache/seatunnel/pull/10800)
> > [Feature][Zeta]
> > STIP-23 Phase 1: Add engine timer flush core flow and FlushSignal
> handling
> > @nzw921rx
> > 5. [#10801](https://github.com/apache/seatunnel/pull/10801)
> > [Feature][Connector-V2] STIP-23 Phase 2: JDBC connector adoption for
> engine
> > timer flush @nzw921rx
> > 6. [#10306](https://github.com/apache/seatunnel/pull/10306)
> > [Feature][Core]
> > Error Data Bypass Capability for Transform & Sink (Dead Letter Queue)
> > @zhangshenghang
> >
> >
> > Guys, please help review the PR for the 3.0 Roadmap when you have time.
> >
> >
> > Best Regards,
> > David
> > Linkedin: https://www.linkedin.com/in/davidzollo
> >
> >
> >
> > On Wed, Jul 1, 2026 at 2:54 PM David Zollo <da...@gmail.com>
> > wrote:
> >
> > > Hi guys,
> > >
> > > I created an issue for the 3.0 roadmap to track progress[1]. Given that
> > > most PRs have been merged so far, we plan to finish merging the
> remaining
> > > PRs by the end of July and then kick off the release process!
> > >
> > > 1. https://github.com/apache/seatunnel/issues/11021
> > >
> > >
> > >
> > >
> > > Best Regards,
> > > David
> > > Linkedin: https://www.linkedin.com/in/davidzollo
> > >
> > >
> > >
> > > On Sat, May 16, 2026 at 12:32 PM David Zollo <da...@gmail.com>
> > > wrote:
> > >
> > >> Hi SeaTunnel community,
> > >>
> > >> Thanks everyone for the active discussion around the Apache SeaTunnel
> > 3.0
> > >> roadmap.
> > >>
> > >> To make the follow-up review easier, I summarized the proposals
> > collected
> > >> in this thread so far. This is only a discussion summary, not a final
> > >> decision. Please feel free to correct me if I missed anything or
> > >> misunderstood any proposal.
> > >>
> > >> 1.
> > >>
> > >> SeaTunnel 3.0 roadmap discussion
> > >>
> > >> The discussion was started to collect proposals for Apache SeaTunnel
> 3.0
> > >> after PR #10510, which updated the version to 3.0.0, was merged into
> the
> > >> dev branch on March 11, 2026.
> > >>
> > >> Several already-merged PRs were mentioned as possible parts of the 3.0
> > >> scope:
> > >>
> > >> -
> > >>
> > >> PR #10586: [Feature][Zeta] Add datasource spi
> > >> https://github.com/apache/seatunnel/pull/10586
> > >> -
> > >>
> > >> PR #10585: [Feature][Zeta] Add /option-rules REST API for runtime
> > >> connector metadata
> > >> https://github.com/apache/seatunnel/pull/10585
> > >> -
> > >>
> > >> PR #10489: [Feature][Transform] Add Zeta SQL UDF context and
> > >> lifecycle hooks with e2e/docs updates
> > >> https://github.com/apache/seatunnel/pull/10489
> > >>
> > >> The purpose of this thread is to collect PRs, features, improvements,
> > and
> > >> ideas that the community believes should be considered for SeaTunnel
> > 3.0.
> > >>
> > >> 2.
> > >>
> > >> STIP-23: Engine-level timer flush for Sink connectors
> > >>
> > >> zhiwei niu proposed including STIP-23 in the SeaTunnel 3.0 roadmap.
> The
> > >> proposal focuses on engine-level timer flush support for Sink
> > connectors.
> > >>
> > >> Related PRs:
> > >>
> > >> -
> > >>
> > >> PR #10800: [Feature][Zeta] Phase 1: Engine timer flush core
> > >> https://github.com/apache/seatunnel/pull/10800
> > >>
> > >> This PR adds sink.flush.interval and introduces a FlushSignal
> > >> control-plane path through the Zeta engine. It allows sinks to
> flush
> > based
> > >> on a wall-clock interval without each sink managing its own timer
> > thread.
> > >> -
> > >>
> > >> PR #10801: [Feature][Connector-V2] Phase 2: JDBC connector adoption
> > >> https://github.com/apache/seatunnel/pull/10801
> > >>
> > >> This PR wires registerFlushAction into the JDBC non-XA sink writer
> > >> through a new enable_timer_flush option. It also includes
> multi-table
> > >> aggregated flush and MySQL CDC E2E tests.
> > >> -
> > >>
> > >> PR #10802: [Docs][Connector-V2] Phase 3: Timer flush feature matrix
> > >> https://github.com/apache/seatunnel/pull/10802
> > >>
> > >> This PR adds a timer flush support marker to sink connector
> > >> documentation and documents the sink.flush.interval and
> > registerFlushAction
> > >> contract.
> > >>
> > >> The main value is to make end-to-end latency more predictable for
> > >> buffered sinks, especially in idle or low-throughput streaming jobs.
> > >>
> > >> 3.
> > >>
> > >> Independent seatunnel-shade project and packaging efficiency
> > >> optimization
> > >>
> > >> Zhang Donghao proposed including PR #9993 in the SeaTunnel 3.0
> roadmap.
> > >>
> > >> -
> > >>
> > >> PR #9993:
> > >> https://github.com/apache/seatunnel/pull/9993
> > >>
> > >> This PR extracts the shade modules into an independent seatunnel-shade
> > >> project, supports uploading shade jars to Maven Central, and optimizes
> > the
> > >> overall packaging efficiency of SeaTunnel.
> > >>
> > >> The proposal points out that current shade module builds, especially
> > >> Hadoop3 Uber and Hadoop AWS, add significant build time. By publishing
> > >> shade jars separately and reusing them as dependencies, SeaTunnel can
> > >> reduce repeated build cost and improve release stability.
> > >>
> > >> 4.
> > >>
> > >> Table-level fault isolation for multi-table synchronization
> > >>
> > >> yi zeng proposed including PR #10600 in the SeaTunnel 3.0 roadmap.
> > >>
> > >> -
> > >>
> > >> PR #10600: [Feature][Core] Support table-level fault isolation for
> > >> multi-table synchronization
> > >> https://github.com/apache/seatunnel/pull/10600
> > >>
> > >> This PR introduces table-level fault isolation for multi-table
> > >> synchronization through a new failure policy:
> > >>
> > >> env {
> > >> multi_table {
> > >> failure_policy = "CONTINUE_OTHER_TABLES"
> > >> }
> > >> }
> > >>
> > >> With this enabled, failures can be isolated at table granularity.
> > Healthy
> > >> tables can continue processing even if one table fails. Failed tables
> > are
> > >> collected and reported with metadata such as table path, failure
> phase,
> > >> plugin, and error.
> > >>
> > >> The proposal is backward-compatible by default because the existing
> > >> FAIL_FAST behavior remains unchanged unless users explicitly enable
> the
> > new
> > >> policy.
> > >>
> > >> 5.
> > >>
> > >> Checkpoint-gated post_sync_action and retention for file source
> > >> continuous discovery
> > >>
> > >> yi zeng also proposed including PR #10563.
> > >>
> > >> -
> > >>
> > >> PR #10563: [Feature][Connector-V2] Add checkpoint-gated
> > >> post_sync_action and retention for file source continuous discovery
> > >> https://github.com/apache/seatunnel/pull/10563
> > >>
> > >> This PR adds checkpoint-aware source-side lifecycle management for
> file
> > >> source continuous discovery, including:
> > >>
> > >> -
> > >>
> > >> post_sync_action = delete
> > >> -
> > >>
> > >> post_sync_action = backup
> > >> -
> > >>
> > >> retention_max_age
> > >>
> > >> The key idea is that source files should only be deleted or moved
> after
> > a
> > >> checkpoint has completed, so that continuous file ingestion remains
> safe
> > >> and recoverable in production environments.
> > >>
> > >> 6.
> > >>
> > >> STIP-21 / StainTrace: End-to-end data lineage tracing and
> performance
> > >> analysis for Zeta
> > >>
> > >> David proposed including PR #10491 in the SeaTunnel 3.0 roadmap.
> > >>
> > >> -
> > >>
> > >> PR #10491:
> > >> https://github.com/apache/seatunnel/pull/10491
> > >>
> > >> StainTrace is the implementation of STIP-21. It provides end-to-end
> data
> > >> lineage tracing and performance analysis for the Zeta engine.
> > >>
> > >> It adds lightweight timestamp sampling points at six key pipeline
> > stages:
> > >>
> > >> -
> > >>
> > >> SOURCE_EMIT
> > >> -
> > >>
> > >> QUEUE_IN
> > >> -
> > >>
> > >> QUEUE_OUT
> > >> -
> > >>
> > >> TRANSFORM_IN
> > >> -
> > >>
> > >> TRANSFORM_OUT
> > >> -
> > >>
> > >> SINK_WRITE_DONE
> > >>
> > >> The goal is to track the full journey of sampled rows from source
> > >> emission to sink write completion, making it easier to measure
> > end-to-end
> > >> latency and identify bottlenecks.
> > >>
> > >> The proposal emphasizes production safety: the feature is disabled by
> > >> default, supports configurable sampling, has a per-worker event
> budget,
> > >> writes trace data locally as JSONL in OpenTelemetry OTLP JSON format,
> > and
> > >> includes an offline analyzer for P95/P99 latency and HTML
> visualization
> > >> reports.
> > >>
> > >> 7.
> > >>
> > >> StateStore abstraction for Zeta engine
> > >>
> > >> Doyeon Kim proposed including PR #10812 in the SeaTunnel 3.0 roadmap.
> > >>
> > >> -
> > >>
> > >> PR #10812:
> > >> https://github.com/apache/seatunnel/pull/10812
> > >>
> > >> This PR introduces an initial StateStore abstraction layer to reduce
> > >> direct Hazelcast IMap dependencies in the Zeta engine. It makes state
> > >> responsibilities more explicit and prepares the engine for future
> > evolution
> > >> around state storage, node communication, and coordination layers.
> > >>
> > >> 8.
> > >>
> > >> IMap external storage compaction
> > >>
> > >> Doyeon Kim also proposed including PR #10399.
> > >>
> > >> -
> > >>
> > >> PR #10399:
> > >> https://github.com/apache/seatunnel/pull/10399
> > >>
> > >> This PR introduces LSM-style write and compaction logic for IMap
> > external
> > >> storage. It helps control storage growth by flushing sorted immutable
> > files
> > >> and compacting persisted files while keeping only the latest record
> per
> > key.
> > >>
> > >> The proposal is intended to improve stability for long-running Zeta
> jobs
> > >> by reducing duplicated records and tombstones in external storage.
> > >>
> > >> 9.
> > >>
> > >> AI-Native CLI and AI for Data Engineering
> > >>
> > >> sean zhang proposed including the AI-Native CLI capability in
> SeaTunnel
> > >> 3.0.
> > >>
> > >> -
> > >>
> > >> PR #10789:
> > >> https://github.com/apache/seatunnel/pull/10789
> > >>
> > >> This PR introduces seatunnel-cli, an AI-powered command-line tool that
> > >> can generate production-ready pipeline configurations from natural
> > language.
> > >>
> > >> Current capabilities include:
> > >>
> > >> -
> > >>
> > >> Multi-LLM provider support, including Anthropic, OpenAI, and AWS
> > >> Bedrock
> > >> -
> > >>
> > >> Runtime connector metadata extraction via Java OptionRule
> reflection
> > >> -
> > >>
> > >> Source/Sink type-aware validation
> > >> -
> > >>
> > >> Auto-fix loop for generated configurations
> > >> -
> > >>
> > >> Sensitive value redaction before persistence and LLM transmission
> > >> -
> > >>
> > >> Interactive mode with session memory
> > >> -
> > >>
> > >> One-shot mode for CI/CD
> > >>
> > >> The proposal also describes a broader “AI for Data Engineering”
> roadmap,
> > >> including intelligent schema mapping, natural language transforms,
> data
> > >> profiling, failure diagnosis, self-healing pipelines, anomaly
> detection,
> > >> performance tuning, cost-aware planning, intelligent scheduling, and
> > >> eventually autonomous data infrastructure management.
> > >>
> > >> 10.
> > >>
> > >> Extension of the roadmap discussion window
> > >>
> > >> David proposed extending the SeaTunnel 3.0 roadmap discussion window
> to
> > >> May 15, 2026, considering the upcoming holiday period and to give
> > >> contributors more time to share ideas.
> > >>
> > >> This message also recognized the quality and breadth of the proposals
> so
> > >> far, including StainTrace, AI-Native CLI, StateStore abstraction, and
> > >> table-level fault isolation.
> > >>
> > >> 11.
> > >>
> > >> STIP-27: Apache Arrow as an optional internal data model
> > >>
> > >> xuepeng wang proposed adding STIP-27 to the SeaTunnel 3.0 roadmap
> > >> discussion.
> > >>
> > >> -
> > >>
> > >> Issue #10806:
> > >> https://github.com/apache/seatunnel/issues/10806
> > >>
> > >> This proposal introduces Apache Arrow as an optional internal data
> > model,
> > >> not as a replacement for the existing row model.
> > >>
> > >> The goal is to improve performance for columnar scenarios such as
> > >> Parquet, ORC, ClickHouse, Doris, and batch transforms such as
> > projection,
> > >> filter, and cast.
> > >>
> > >> Expected benefits include better columnar I/O performance, vectorized
> > >> execution, and lower JVM overhead.
> > >>
> > >> 12.
> > >>
> > >> STIP-24: Edge collection, EdgeSocket ingress, and SeaTunnel Edge
> > Agent
> > >>
> > >> zhiwei niu proposed adding STIP-24 edge-collection work to the
> SeaTunnel
> > >> 3.0 roadmap.
> > >>
> > >> The original PRs were:
> > >>
> > >> -
> > >>
> > >> PR #10871: [Feature][Zeta] STIP-24 Phase 1: Support lightweight
> > >> EdgeSocket ingress for edge collector MVP
> > >> https://github.com/apache/seatunnel/pull/10871
> > >> -
> > >>
> > >> PR #10876: [Feature][Zeta] STIP-24 Phase 2: Introduce standalone
> > >> SeaTunnel Edge Agent MVP and dist integration
> > >> https://github.com/apache/seatunnel/pull/10876
> > >>
> > >> Phase 1 introduces the engine-side ingress foundation for edge
> > collection
> > >> through EdgeSocket, including authenticated ingress, source-side
> > protocol
> > >> handling, and task-group address discovery support in Zeta.
> > >>
> > >> Phase 2 introduces a standalone SeaTunnel Edge Agent runtime,
> including
> > >> core, transport, connector, local file/log/event inputs, SQLite WAL
> > >> durability, EdgeSocket transport interoperability, and distribution
> > >> integration.
> > >>
> > >> The goal is to support an end-to-end edge collection workflow:
> > >>
> > >> edge host -> Zeta pipeline
> > >>
> > >> zhiwei later updated the PR references for STIP-24:
> > >>
> > >> -
> > >>
> > >> PR #10871 was updated to PR #10878
> > >> https://github.com/apache/seatunnel/pull/10878
> > >> -
> > >>
> > >> PR #10876 was updated to PR #10879
> > >> https://github.com/apache/seatunnel/pull/10879
> > >>
> > >>
> > >> 13.
> > >>
> > >> Prometheus metrics for Zeta Engine IMap state stores
> > >>
> > >> JeremyXin proposed including PR #10860 in the SeaTunnel 3.0 roadmap.
> > >>
> > >> -
> > >>
> > >> PR #10860:
> > >> https://github.com/apache/seatunnel/pull/10860
> > >>
> > >> This PR improves Zeta Engine observability by exposing Prometheus
> > metrics
> > >> for core engine state stores backed by Hazelcast IMap.
> > >>
> > >> It covers internal stores such as running job information, job states,
> > >> checkpoints, and slot ownership. It exposes metrics such as local
> entry
> > >> counts, backup entry counts, and heap cost.
> > >>
> > >> The proposal aims to help users detect state growth, data skew, and
> > >> memory pressure earlier, improving troubleshooting, capacity planning,
> > and
> > >> production stability.
> > >>
> > >> 14.
> > >>
> > >> Real-time observability for Zeta engine tasks
> > >>
> > >> zhangshenghang proposed improving real-time observability for Zeta
> > engine
> > >> tasks.
> > >>
> > >> This direction adds the ability to monitor the runtime busyness level
> of
> > >> Zeta engine tasks, including real-time task metrics for Source,
> > Transform,
> > >> and Sink execution. It also includes queue and backpressure-related
> > >> information exposed through REST API and Web UI.
> > >>
> > >> The goal is to help users understand whether a job is blocked by
> source
> > >> reading, transform processing, sink writing, or intermediate queues.
> > >>
> > >> 15.
> > >>
> > >> Error data bypass and dead-letter queue capability for Transform
> and
> > >> Sink
> > >>
> > >> zhangshenghang also proposed row-level error handling for Transform
> and
> > >> Sink stages.
> > >>
> > >> When configured, SeaTunnel can collect failed records with useful
> > >> context, including:
> > >>
> > >> -
> > >>
> > >> error stage
> > >> -
> > >>
> > >> plugin type
> > >> -
> > >>
> > >> plugin name
> > >> -
> > >>
> > >> source table
> > >> -
> > >>
> > >> error message
> > >> -
> > >>
> > >> exception class
> > >> -
> > >>
> > >> original data
> > >> -
> > >>
> > >> occurrence time
> > >>
> > >> This allows jobs to bypass selected dirty or error records instead of
> > >> always failing fast, while keeping the default behavior unchanged when
> > the
> > >> feature is not enabled.
> > >>
> > >> This is useful for long-running synchronization jobs where users need
> > >> better fault tolerance and later inspection of bad data.
> > >>
> > >> 16.
> > >>
> > >> Improve cloud-native deployment experience
> > >>
> > >> zhangshenghang also suggested improving the cloud-native deployment
> > >> experience for SeaTunnel 3.0.
> > >>
> > >> The proposal mentions Kubernetes and Helm support as important areas,
> > >> including better guidance and behavior around:
> > >>
> > >> -
> > >>
> > >> deployment
> > >> -
> > >>
> > >> restart
> > >> -
> > >>
> > >> persistent checkpoint storage
> > >> -
> > >>
> > >> configuration management
> > >> -
> > >>
> > >> secret management
> > >> -
> > >>
> > >> rolling upgrade
> > >> -
> > >>
> > >> resource isolation
> > >>
> > >>
> > >> 17.
> > >>
> > >> Clean up deprecated behavior carefully for 3.0
> > >>
> > >> zhangshenghang also suggested that SeaTunnel 3.0 is a good opportunity
> > to
> > >> clean up deprecated options and old APIs.
> > >>
> > >> At the same time, any incompatible change should be handled carefully,
> > >> with clear migration documentation, examples, and release notes.
> > >>
> > >> Possible discussion groups
> > >>
> > >> Based on the above proposals, I think the roadmap discussion can be
> > >> grouped into the following areas:
> > >>
> > >> 1.
> > >>
> > >> Zeta engine observability
> > >>
> > >>
> > >> -
> > >>
> > >> STIP-21 / StainTrace
> > >> -
> > >>
> > >> Prometheus metrics for IMap state stores
> > >> -
> > >>
> > >> Real-time task metrics
> > >> -
> > >>
> > >> Queue and backpressure visibility
> > >> -
> > >>
> > >> REST API and Web UI exposure
> > >>
> > >>
> > >> 2.
> > >>
> > >> Zeta state management and storage evolution
> > >>
> > >>
> > >> -
> > >>
> > >> StateStore abstraction
> > >> -
> > >>
> > >> IMap external storage compaction
> > >> -
> > >>
> > >> IMap state metrics
> > >> -
> > >>
> > >> Future storage backend evolution
> > >>
> > >>
> > >> 3.
> > >>
> > >> Runtime and connector semantics
> > >>
> > >>
> > >> -
> > >>
> > >> Engine-level timer flush for Sink connectors
> > >> -
> > >>
> > >> JDBC adoption of timer flush
> > >> -
> > >>
> > >> Checkpoint-gated file source lifecycle management
> > >> -
> > >>
> > >> Table-level fault isolation
> > >> -
> > >>
> > >> Row-level error bypass and dead-letter queue
> > >>
> > >>
> > >> 4.
> > >>
> > >> AI for Data Engineering
> > >>
> > >>
> > >> -
> > >>
> > >> AI-Native CLI
> > >> -
> > >>
> > >> Natural language pipeline generation
> > >> -
> > >>
> > >> Connector metadata reflection
> > >> -
> > >>
> > >> Auto-fix loop
> > >> -
> > >>
> > >> Security and sensitive value redaction
> > >> -
> > >>
> > >> Future AI-assisted schema mapping, diagnosis, optimization, and
> > >> autonomous data management
> > >>
> > >>
> > >> 5.
> > >>
> > >> Edge collection
> > >>
> > >>
> > >> -
> > >>
> > >> EdgeSocket ingress
> > >> -
> > >>
> > >> SeaTunnel Edge Agent
> > >> -
> > >>
> > >> Edge host to Zeta pipeline workflow
> > >> -
> > >>
> > >> Authentication, local durability, and transport interoperability
> > >>
> > >>
> > >> 6.
> > >>
> > >> Performance and internal data model
> > >>
> > >>
> > >> -
> > >>
> > >> Apache Arrow as an optional internal data model
> > >> -
> > >>
> > >> Columnar I/O
> > >> -
> > >>
> > >> Vectorized execution
> > >> -
> > >>
> > >> Lower JVM overhead
> > >>
> > >>
> > >> 7.
> > >>
> > >> Build, release, and project structure
> > >>
> > >>
> > >> -
> > >>
> > >> Independent seatunnel-shade project
> > >> -
> > >>
> > >> Maven Central distribution of shade jars
> > >> -
> > >>
> > >> Reduced build and release time
> > >>
> > >>
> > >> 8.
> > >>
> > >> Cloud-native and compatibility
> > >>
> > >>
> > >> -
> > >>
> > >> Kubernetes and Helm deployment experience
> > >> -
> > >>
> > >> Persistent checkpoint storage
> > >> -
> > >>
> > >> Secret management
> > >> -
> > >>
> > >> Rolling upgrade
> > >> -
> > >>
> > >> Resource isolation
> > >> -
> > >>
> > >> Deprecated behavior cleanup
> > >> -
> > >>
> > >> Migration documentation and release notes
> > >>
> > >> Next steps
> > >>
> > >> I suggest we continue to review these proposals and decide:
> > >>
> > >> 1.
> > >>
> > >> Which items should be included in the SeaTunnel 3.0 roadmap.
> > >> 2.
> > >>
> > >> Which items should be marked as experimental.
> > >> 3.
> > >>
> > >> Which items require an STIP or further design discussion.
> > >> 4.
> > >>
> > >> Which items can be merged independently before the 3.0 release.
> > >> 5.
> > >>
> > >> Which items need migration documentation, compatibility notes, or
> > >> release notes.
> > >>
> > >> Again, this is only a summary of the current discussion thread. Please
> > >> help correct or supplement anything I missed.
> > >>
> > >>
> > >>
> > >> Best Regards,
> > >> David
> > >> Linkedin: https://www.linkedin.com/in/davidzollo
> > >>
> > >>
> > >>
> > >> On Thu, May 14, 2026 at 2:59 PM zhangshenghang <sh...@apache.org>
> > >> wrote:
> > >>
> > >>> Hi community,
> > >>>
> > >>> For the SeaTunnel 3.0 roadmap
> > >>>
> > >>> I have already submitted two related PRs, and besides those, I would
> > like
> > >>> to propose the following directions for 3.0:
> > >>>
> > >>> 1. Real-time observability for Zeta engine tasks
> > >>>
> > >>> This work adds the ability to monitor the runtime busyness level of
> > Zeta
> > >>> engine tasks. It introduces real-time task metrics for Source,
> > Transform,
> > >>> and Sink execution, including queue/backpressure-related information,
> > and
> > >>> exposes these metrics through REST API and Web UI. This can help
> users
> > >>> understand whether a job is blocked by source reading, transform
> > >>> processing, sink writing, or intermediate queues, which is very
> > important
> > >>> for production troubleshooting and performance tuning.
> > >>>
> > >>> 2. Error data bypass and dead-letter queue capability for Transform
> and
> > >>> Sink
> > >>>
> > >>> This work introduces row-level error handling for Transform and Sink
> > >>> stages. When configured, SeaTunnel can collect failed records with
> > useful
> > >>> context such as error stage, plugin type, plugin name, source table,
> > >>> error
> > >>> message, exception class, original data, and occurrence time. This
> > allows
> > >>> jobs to bypass selected dirty/error records instead of always failing
> > >>> fast,
> > >>> while keeping the default behavior unchanged when the feature is not
> > >>> enabled. This is useful for long-running synchronization jobs where
> > users
> > >>> need better fault tolerance and later inspection of bad data.
> > >>>
> > >>> 3. Improve cloud-native deployment experience
> > >>>
> > >>> Kubernetes and Helm support are important for new users. We should
> > >>> improve
> > >>> guidance and behavior around deployment, restart, persistent
> checkpoint
> > >>> storage, configuration management, secret management, rolling
> upgrade,
> > >>> and
> > >>> resource isolation in cloud-native environments.
> > >>>
> > >>> 4. Clean up deprecated behavior carefully for 3.0
> > >>>
> > >>> Since 3.0 is a major version, it is a good opportunity to clean up
> > >>> deprecated options and old APIs, but we should do this carefully. Any
> > >>> incompatible change should have clear migration documentation,
> > examples,
> > >>> and release notes.
> > >>>
> > >>>
> > >>> Regards,
> > >>> Jast (Shenghang)
> > >>>
> > >>>
> > >>> JeremyXin <73...@qq.com> 于2026年5月14日周四 14:27写道：
> > >>>
> > >>> > Hi everyone, I’d like to propose adding this PR to the SeaTunnel
> 3.0
> > >>> > roadmap.
> > >>> >
> > >>> >
> > >>> > - PR#10860
> > >>> > - link: https://github.com/apache/seatunnel/pull/10860
> > >>> >
> > >>> >
> > >>> > This PR improves Zeta Engine observability by exposing Prometheus
> > >>> metrics
> > >>> > for core engine state stores backed by Hazelcast IMap.
> > >>> >
> > >>> >
> > >>> > It covers important internal stores such as running job
> information,
> > >>> job
> > >>> > states, checkpoints, slot ownership and provides visibility into
> > their
> > >>> > local entry counts, backup entry counts, and heap cost.
> > >>> >
> > >>> >
> > >>> > Including this in SeaTunnel 3.0 would make the engine easier to
> > monitor
> > >>> > and operate in production. It helps users detect state growth, data
> > >>> skew,
> > >>> > and memory pressure earlier, which improves troubleshooting
> > efficiency,
> > >>> > cluster capacity planning, and overall operational stability.
> > >>> >
> > >>> >
> > >>> > Best regards,
> > >>> > JeremyXin
> > >>> >
> > >>> >
> > >>> > 原始邮件
> > >>> >
> > >>> >
> > >>> > 发件人：zhiwei niu <zniu70696@gmail.com>
> > >>> > 发件时间：2026年5月12日 10:27
> > >>> > 收件人：dev <dev@seatunnel.apache.org>
> > >>> > 主题：Re: [DISCUSS] Apache SeaTunnel 3.0 Roadmap
> > >>> >
> > >>> >
> > >>> >
> > >>> > Hi everyone,
> > >>> >
> > >>> >
> > >>> >
> > >>>
> >
> As part of Roadmap 3.0 progress tracking, the STIP-24 PR links have been
> > >>> >
> > >>> >
> > >>>
> >
> updated to keep references consistent and easier to follow.
> > >>> >
> > >>> >
> Please use the updated PR links below:
> > >>> >
> > >>> >
> > >>> >
> > >>>
> >
> [Feature][Zeta] STIP-24 Phase 1: Support lightweight EdgeSocket ingress
> > >>> > for edge collector MVP
> > >>> > - PR #10871(
> > >>> > https://github.com/apache/seatunnel/pull/10871)
> > >>> > —> PR#10878(
> > >>> > https://github.com/apache/seatunnel/pull/10878)
> > >>> >
> > >>> >
> > >>> >
> > >>>
> >
> [Feature][Zeta] STIP-24 Phase 2: Introduce standalone SeaTunnel Edge Agent
> > >>> > MVP and dist integration
> > >>> > - PR #10876(
> > >>> > https://github.com/apache/seatunnel/pull/10876)
> > >>> > — PR#10879(
> > >>> > https://github.com/apache/seatunnel/pull/10879)
> > >>> >
> > >>> >
> > >>> > zhiwei niu <zniu70696@gmail.com
> > >>> > > 于2026年5月12日周二 01:31写道：
> > >>> >
> > >>> > > Hi everyone,
> > >>> > >
> > >>> >
> > >>> >
> > >>>
> >
> > In addition to STIP-23, I’d like to propose the STIP-24 edge-collection
> > >>> > > work for SeaTunnel 3.0.
> > >>> > >
> > >>> >
> > >>> >
> > >>>
> >
> > - PR #10871 — [Feature][Zeta] STIP-24 Phase 1: Support lightweight
> > >>> >
> > >>> >
> > >>>
> >
> > EdgeSocket ingress for edge collector MVP
> > >>> > > https://github.com/apache/seatunnel/pull/10871
> > >>> > > What it adds:
> > >>> >
> > >>> >
> > >>>
> >
> > Introduces the engine-side ingress foundation for edge collection via
> > >>> >
> > >>> >
> > >>>
> >
> > EdgeSocket, including authenticated ingress, source-side protocol handling,
> > >>> >
> > >>> >
> > >>>
> >
> > and task-group address discovery support in Zeta.
> > >>> > > Why 3.0:
> > >>> >
> > >>> >
> > >>>
> >
> > This is the foundational engine capability for remote edge data collection
> > >>> >
> > >>> >
> > >>>
> >
> > scenarios. It is backward-compatible and establishes the protocol/runtime
> > >>> >
> > >>>
> > > path required for the edge-side agent.
> > >>> > >
> > >>> >
> > >>> >
> > >>>
> >
> > - PR #10876 — [Feature][Zeta] STIP-24 Phase 2: Introduce standalone
> > >>> >
> > >>> >
> > >>>
> >
> > SeaTunnel Edge Agent MVP and dist integration
> > >>> > > https://github.com/apache/seatunnel/pull/10876
> > >>> > > What it adds:
> > >>> >
> > >>> >
> > >>>
> >
> > Introduces a standalone SeaTunnel Edge Agent runtime
> > >>> >
> > >>> >
> > >>>
> >
> > (core/transport/connector), local file/log/event inputs, SQLite WAL
> > >>> >
> > >>> >
> > >>>
> >
> > durability, EdgeSocket transport interop, and distribution integration.
> > >>> > > Why 3.0:
> > >>> >
> > >>> >
> > >>>
> >
> > This delivers the first practical end-to-end edge collection workflow
> > >>> >
> > >>> >
> > >>>
> >
> > (edge host -> Zeta pipeline), which significantly expands SeaTunnel’s
> > >>> >
> > >>> >
> > >>>
> >
> > deployment topology and real-world ingestion coverage.
> > >>> > >
> > >>> >
> > >>> >
> > >>>
> >
> > PR #10876 depends on #10871; both are under active review.
> > >>> > >
> > >>> > > Best regards,
> > >>> > > zhiwei.niu
> > >>> > >
> > >>> > > xuepeng wang <wxuepeng5@gmail.com
> > >>> > > 于2026年5月8日周五 14:03写道：
> > >>> > >
> > >>> > >> Hi team,
> > >>> > >>
> > >>> > >>
> > >>> > >>
> > >>> >
> > >>> >
> > >>>
> >
> >> I'd like to add STIP-27 [1] to the SeaTunnel 3.0 Roadmap discussion.
> > >>> > >>
> > >>> >
> > >>> >
> > >>>
> >
> >> This proposal introduces Apache Arrow as an optional internal data model
> > >>> >
> > >>> >
> > >>>
> >
> >> (not replacing the existing row model). It aims to improve performance for
> > >>> >
> > >>> >
> > >>>
> >
> >> columnar scenarios (Parquet/ORC, ClickHouse/Doris) and batch transforms
> > >>> >
> > >>> >
> > >>>
> >
> >> (projection/filter/cast), while keeping the current architecture stable.
> > >>> > >>
> > >>> >
> > >>> >
> > >>>
> >
> >> Key benefits: better columnar I/O performance, vectorized execution,
> > >>> > >> lower JVM overhead.
> > >>> > >>
> > >>> > >> [1]
> > >>> > https://github.com/apache/seatunnel/issues/10806
> > >>> > >> <https://github.com/apache/seatunnel/issues/10806>>>;
> > >>> > >> Thoughts?
> > >>> > >>
> > >>> > >> Regards,
> > >>> > >> xuepeng(github id:chl-wxp)
> > >>> > >>
> > >>> > >> David Zollo <davidzollo365@gmail.com
> > >>> > > 于2026年4月30日周四 21:17写道：
> > >>> > >>
> > >>> > >> > Hi everyone,
> > >>> > >> >
> > >>> >
> > >>> >
> > >>>
> >
> >> > First of all, thank you all for the fantastic proposals so far — the
> > >>> >
> > >>> >
> > >>>
> >
> >> > quality and breadth of ideas in this thread have been genuinely
> > >>> >
> > >>> >
> > >>>
> >
> >> > exciting to read. From StainTrace to the AI-Native CLI, from
> > >>> >
> > >>> >
> > >>>
> >
> >> > StateStore abstraction to table-level fault isolation, it's clear that
> > >>> >
> > >>> >
> > >>>
> >
> >> > this community has a bold and thoughtful vision for what SeaTunnel 3.0
> > >>> > >> > can become.
> > >>> > >> >
> > >>> >
> > >>> >
> > >>>
> >
> >> > Given that the Golden Week holiday is just around the corner, and many
> > >>> >
> > >>> >
> > >>>
> >
> >> > of our contributors are based in China or taking time off during this
> > >>> >
> > >>> >
> > >>>
> >
> >> > period, I'd like to extend the discussion window to May 15th. This
> > >>> >
> > >>> >
> > >>>
> >
> >> > gives everyone enough breathing room to enjoy the holiday, come back
> > >>> >
> > >>> >
> > >>>
> >
> >> > refreshed, and still have time to share their thoughts without feeling
> > >>> > >> > rushed.
> > >>> > >> >
> > >>> >
> > >>>
> > >> > Happy Labor Day to everyone!
> > >>> > >> >
> > >>> >
> > >>> >
> > >>>
> >
> >> > We highly welcome everyone's suggestions for version 3.0! Feel free to
> > >>> >
> > >>> >
> > >>>
> >
> >> > leave a message if you have any ideas.
> > >>> > >> >
> > >>> > >> >
> > >>> > >> >
> > >>> > >> > Best Regards,
> > >>> > >> > David
> > >>> > >> > Linkedin:
> > >>> > https://www.linkedin.com/in/davidzollo
> > >>> > >> >
> > >>> > <https://www.linkedin.com/in/davidzollo>> >>;
> > >>> >
> > >>> >
> > >>>
> >
> >> > On Wed, Apr 29, 2026 at 8:50 PM sean zhang <
> > >>> > seanzhang92@gmail.com>
> > >>> > >> wrote:
> > >>> > >> > >
> > >>> > >> > > Hi SeaTunnel community,
> > >>> > >> > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > I'd like to propose including the AI-Native CLI capability in the
> > >>> >
> > >>> >
> > >>>
> >
> >> > > SeaTunnel 3.0 release, and share a broader vision for the "AI for Data
> > >>> >
> > >>> >
> > >>>
> >
> >> > > Engineering" direction. I believe this aligns well with the 3.0
> > >>> > >> milestone
> > >>> >
> > >>> >
> > >>>
> >
> >> > > as a significant user-experience leap.
> > >>> > >> > > This PR
> > >>> > https://github.com/apache/seatunnel/pull/10789 introduces
> > >>> >
> > >>> >
> > >>>
> >
> >> > > `seatunnel-cli`, an AI-powered command-line tool that
> > >>> >
> > >>> >
> > >>>
> >
> >> > > generates production-ready pipeline configurations from natural
> > >>> > >> language
> > >>> > <
> > >>>
> >
> https://github.com/apache/seatunnel/pull/10789 introduces>> > > seatunnel-cli, an AI-powered command-line tool that>> > > generates production-ready pipeline configurations from natural>> language
> > >>> >
> > >>> > :
> > >>> > >> > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > $ seatunnel "Sync the users table from MySQL to S3 in parquet
> > >>> > >> format"
> > >>> >
> > >>> >
> > >>>
> >
> >> > > ✅ Generated config: mysql_users_to_s3.conf
> > >>> > >> > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > Key capabilities in the current implementation:
> > >>> > >> > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > - Multi-LLM provider support (Anthropic / OpenAI / AWS Bedrock)
> > >>> >
> > >>> >
> > >>>
> >
> >> > > with auto-detection
> > >>> >
> > >>> >
> > >>>
> >
> >> > > - Runtime connector metadata extraction via Java OptionRule
> > >>> > >> reflection,
> > >>> >
> > >>> >
> > >>>
> >
> >> > > ensuring 100% accurate parameter knowledge for all 100+ connectors
> > >>> >
> > >>> >
> > >>>
> >
> >> > > - Source/Sink type-aware validation (Jdbc source options ≠ Jdbc sink
> > >>> > >> > > options)
> > >>> >
> > >>> >
> > >>>
> >
> >> > > - Auto-fix loop: generated configs are validated and self-corrected
> > >>> >
> > >>> >
> > >>>
> >
> >> > > (up to 3 iterations)
> > >>> >
> > >>> >
> > >>>
> >
> >> > > - Security hardening: sensitive values (passwords, tokens, keys) are
> > >>> >
> > >>> >
> > >>>
> >
> >> > > automatically redacted to ${ENV_VAR} placeholders before
> > >>> > >> persistence
> > >>> >
> > >>> >
> > >>>
> >
> >> > > and LLM transmission
> > >>> >
> > >>> >
> > >>>
> >
> >> > > - Interactive mode with session memory and one-shot mode for CI/CD
> > >>> > >> > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > Future Roadmap — AI for Data Engineering
> > >>> > >> > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > Looking beyond config generation, I see a multi-level roadmap where
> > >>> >
> > >>> >
> > >>>
> >
> >> > > AI becomes deeply integrated into SeaTunnel's core value proposition:
> > >>> > >> > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > ### Level 2: AI Understands Data (Next phase)
> > >>> > >> > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > - Intelligent Schema Mapping: AI reads source schema, infers target
> > >>> >
> > >>> >
> > >>>
> >
> >> > > schema, and auto-generates type conversions and DDL
> > >>> >
> > >>> >
> > >>>
> >
> >> > > - Natural Language Transforms: "mask phone numbers, merge first and
> > >>> >
> > >>> >
> > >>>
> >
> >> > > last name, filter last 7 days" → SQL Transform auto-generation
> > >>> >
> > >>> >
> > >>>
> >
> >> > > - Data Profiling: AI samples source data, detects anomalies (NULLs,
> > >>> >
> > >>> >
> > >>>
> >
> >> > > outliers, format violations), and suggests data quality transforms
> > >>> > >> > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > ### Level 3: AI Operates Pipelines
> > >>> > >> > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > - Intelligent Failure Diagnosis: AI analyzes error logs, pinpoints
> > >>> >
> > >>> >
> > >>>
> >
> >> > > root cause (connection refused → MySQL down vs firewall), suggests
> > >>> > >> > fix
> > >>> >
> > >>> >
> > >>>
> >
> >> > > - Self-Healing Pipelines: auto-detect DDL changes in CDC sources,
> > >>> >
> > >>> >
> > >>>
> >
> >> > > auto-evolve sink schema, zero-downtime recovery
> > >>> >
> > >>> >
> > >>>
> >
> >> > > - Anomaly Detection: monitor data flow volume/latency, alert when
> > >>> >
> > >>> >
> > >>>
> >
> >> > > throughput drops 99% with correlated root cause
> > >>> > >> > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > ### Level 4: AI Optimizes Everything
> > >>> > >> > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > - Performance Tuning: AI analyzes bottlenecks (batch_size too small,
> > >>> >
> > >>> >
> > >>>
> >
> >> > > wrong file format, insufficient parallelism), suggests
> > >>> > >> optimization
> > >>> >
> > >>> >
> > >>>
> >
> >> > > - Cost-Aware Planning: given N tables to sync, AI proposes pipeline
> > >>> >
> > >>> >
> > >>>
> >
> >> > > topology that balances cost vs latency vs resource usage
> > >>> >
> > >>> >
> > >>>
> >
> >> > > - Intelligent Scheduling: learn from historical patterns to optimize
> > >>> >
> > >>> >
> > >>>
> >
> >> > > sync frequency per table
> > >>> > >> > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > ### Level 5: Autonomous Data Platform (Long-term vision)
> > >>> > >> > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > - Conversational data infrastructure management: "onboard the
> > >>> > >> payment
> > >>> >
> > >>> >
> > >>>
> >
> >> > > system to the data lake, ensure GDPR compliance" → AI
> > >>> > >> auto-discovers
> > >>> >
> > >>> >
> > >>>
> >
> >> > > sources, identifies PII, generates pipelines with masking
> > >>> > >> transforms,
> > >>> >
> > >>> >
> > >>>
> >
> >> > > deploys with monitoring and alerting
> > >>> > >> > >
> > >>> > >> > >
> > >>> > >> > > Doyeon Kim <
> > >>> > ekagnlek2@gmail.com> 于2026年4月29日周三 18:17写道：
> > >>> > >> > >
> > >>> > >> > > > Hi everyone,
> > >>> > >> > > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > I’d like to propose the following Zeta engine improvements for
> > >>> > >> > SeaTunnel
> > >>> > >> > > > 3.0.
> > >>> > >> > > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > 1. StateStore abstraction
> > >>> > >> > > > PR:
> > >>> > https://github.com/apache/seatunnel/pull/10812
> > >>> > >> > > >
> > >>> > <
> > >>>
> >
> https://github.com/apache/seatunnel/pull/10812>> > > >
> > >>> >
> > >>> > ;
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > This introduces an initial StateStore abstraction layer to reduce
> > >>> > >> > direct
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > Hazelcast IMap dependencies in the engine. It makes different state
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > responsibilities more explicit and prepares the engine for future
> > >>> > >> > backend
> > >>> > >> > > > evolution.
> > >>> > >> > > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > I think this is relevant for 3.0 because it is a foundational step
> > >>> > >> > toward
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > decoupling the engine from Hazelcast-specific APIs and enabling
> > >>> > >> future
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > improvements around state storage, node communication, and
> > >>> > >> coordination
> > >>> > >> > > > layers.
> > >>> > >> > > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > 2. IMap external storage compaction
> > >>> > >> > > > PR:
> > >>> > https://github.com/apache/seatunnel/pull/10399
> > >>> > >> > > >
> > >>> > <
> > >>>
> >
> https://github.com/apache/seatunnel/pull/10399>> > > >
> > >>> >
> > >>> > ;
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > This introduces LSM-style write and compaction logic for IMap
> > >>> > >> external
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > storage. It helps control storage growth by flushing sorted
> > >>> > >> immutable
> > >>> > >> > files
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > and compacting persisted files while keeping only the latest record
> > >>> > >> per
> > >>> > >> > > > key.
> > >>> > >> > > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > I think this should be considered for 3.0 because long-running Zeta
> > >>> > >> > jobs
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > may accumulate duplicated records and tombstones in external storage
> > >>> > >> > over
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > time. Compaction can improve stability and reduce the risk of
> > >>> > >> unbounded
> > >>> > >> > > > storage growth.
> > >>> > >> > > >
> > >>> > >> > > > Best Regards,
> > >>> > >> > > > Doyeon
> > >>> >
> > >>>
> > >> > > > (Github ID: dybyte)
> > >>> > >> > > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > On Wed, Apr 29, 2026 at 7:13 PM David Zollo <
> > >>> > >> davidzollo365@gmail.com>
> > >>> > >> > > > wrote:
> > >>> > >> > > >
> > >>> > >> > > > > Hi guys,
> > >>> > >> > > > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > I'd like to propose including PR #10491 in the SeaTunnel 3.0
> > >>> > >> roadmap,
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > here are the details with additional packaging time analysis:
> > >>> > >> > > > >
> > >>> >
> > >>>
> >
> >> > > > > ### PR link
> > >>> > >> > > > >
> > >>> > https://github.com/apache/seatunnel/pull/10491
> > >>> > >> > > > >
> > >>> > <
> > >>>
> >
> https://github.com/apache/seatunnel/pull/10491>> > > > >
> > >>> >
> > >>> > ;
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > StainTrace (PR #10491) is the concrete implementation of STIP-21,
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > delivering end-to-end data lineage tracing and performance
> > >>> > >> analysis
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > capabilities for the Apache SeaTunnel Zeta engine. It injects
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > lightweight timestamp sampling points at 6 key pipeline stages —
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > SOURCE_EMIT → QUEUE_IN → QUEUE_OUT → TRANSFORM_IN → TRANSFORM_OUT
> > >>> > >> →
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > SINK_WRITE_DONE — tracking each sampled row's complete journey
> > >>> > >> from
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > source emission to sink write completion, enabling precise
> > >>> > >> end-to-end
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > latency measurement and bottleneck identification.
> > >>> > >> > > > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > The system is designed with production safety as a first-class
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > concern: it is disabled by default with very low overhead, and
> > >>> > >> when
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > enabled it operates at a configurable sampling rate with a strict
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > per-worker per-second event budget, achieving approximately 0.1~1%
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > overhead at 1% sampling. Trace data is written locally as JSONL
> > >>> > >> files
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > in OpenTelemetry OTLP JSON format with no external dependencies.
> > >>> > >> An
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > accompanying offline analyzer reads these files to compute P95/P99
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > latency statistics, identify bottleneck stages, and generate an
> > >>> > >> HTML
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > visualization report with Chinese/English language toggle support.
> > >>> > >> > > > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > The value to SeaTunnel lies in closing a critical observability
> > >>> > >> gap
> > >>> > >> > at
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > the engine level. Before this feature, users facing latency
> > >>> > >> spikes or
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > throughput bottlenecks had no structured way to pinpoint whether
> > >>> > >> the
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > problem resided in queue backpressure, transform processing, or
> > >>> > >> sink
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > write time — they could only rely on logs and guesswork.
> > >>> > >> StainTrace
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > makes per-stage timing transparent and actionable, which is
> > >>> > >> > especially
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > impactful for large-scale, multi-table, long-chain production
> > >>> > >> > > > > pipelines.
> > >>> > >> > > > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > The case for including it in the 3.0 roadmap is equally strong. As
> > >>> > >> > the
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > implementation of STIP-21, StainTrace is a framework-level
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > observability capability — connector-agnostic and immediately
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > effective across all jobs once merged. One of SeaTunnel 3.0's
> > >>> > >> central
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > goals is to become a more mature, production-ready data
> > >>> > >> integration
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > engine, and observability is an indispensable foundation for that
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > goal. Establishing it at a major version boundary like 3.0
> > >>> > >> delivers
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > far greater long-term value to users.
> > >>> > >> > > > >
> > >>> > >> > > > >
> > >>> > >> > > > >
> > >>> > >> > > > >
> > >>> > >> > > > >
> > >>> > >> > > > >
> > >>> >
> > >>>
> > >> > > > > Best Regards,
> > >>> > >> > > > > David
> > >>> >
> >> > > > > Linkedin:
> > >>> > https://www.linkedin.com/in/davidzollo
> > >>> > >> > > > >
> > >>> > <
> > >>>
> >
> https://www.linkedin.com/in/davidzollo>> > > > >
> > >>> >
> > >>> > ;
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > On Wed, Apr 29, 2026 at 2:48 PM yi zeng <
> > >>> > yzeng1618@gmail.com>
> > >>> > >> wrote:
> > >>> > >> > > > > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > Hi everyone,
> > >>> > >> > > > > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > I’d like to propose including the following two PRs in the
> > >>> > >> > *SeaTunnel
> > >>> > >> > > > 3.0
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > roadmap*. Both focus on strengthening *correctness, reliability,
> > >>> > >> > and
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > operational safety* for core multi-table and continuous
> > >>> > >> ingestion
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > scenarios, which I believe are key pillars for 3.0.
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > ------------------------------
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > PR #10600 — [Feature][Core] Support table-level fault isolation
> > >>> > >> for
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > multi-table synchronization
> > >>> > >> > > > > >
> > >>> > >> > > > > >
> > >>> > https://github.com/apache/seatunnel/pull/10600
> > >>> > >> > > > > >
> > >>> > <
> > >>>
> >
> https://github.com/apache/seatunnel/pull/10600>> > > > > >
> > >>> >
> > >>> > ;
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > *What it adds / solves*
> > >>> > >> > > > > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > This PR introduces *table-level fault isolation* for multi-table
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > synchronization by adding a new failure policy:
> > >>> >
> >> > > > > > code
> > >>> > >> > > > > > 复制
> > >>> > >> > > > > >
> > >>> >
> > >>>
> >
> >> > > > > > env {
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > multi_table {
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > failure_policy = "CONTINUE_OTHER_TABLES"
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > }
> > >>> > >> > > > > > }
> > >>> > >> > > > > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > With this enabled:
> > >>> > >> > > > > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > - Failures are isolated at *table granularity*, instead of
> > >>> > >> > failing
> > >>> > >> > > > the
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > entire job.
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > - Healthy tables continue processing even if one table fails.
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > - Failed tables are collected and reported with clear
> > >>> > >> metadata
> > >>> > >> > > > (table
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > path, failure phase, plugin, error).
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > - Batch jobs report failed tables in final results; streaming
> > >>> > >> > jobs
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > continue running with healthy tables.
> > >>> > >> > > > > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > This addresses a long-standing pain point where a single
> > >>> > >> > problematic
> > >>> > >> > > > > table
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > can block progress for all others.
> > >>> > >> > > > > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > *Why it should be included in 3.0*
> > >>> > >> > > > > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > - *Improved fault tolerance model*: 3.0 aims to be more
> > >>> > >> > resilient
> > >>> > >> > > > and
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > production-ready; table-level isolation is a foundational
> > >>> > >> > capability
> > >>> > >> > > > > for
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > large-scale, multi-table pipelines.
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > - *Backward-compatible by default*: The default behavior
> > >>> > >> remains
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > FAIL_FAST, so existing jobs are unaffected unless users
> > >>> > >> > explicitly
> > >>> > >> > > > opt
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > in.
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > - *Framework-level improvement*: This is not
> > >>> > >> > connector-specific; it
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > enhances the core execution and failure-handling semantics of
> > >>> > >> > > > > SeaTunnel.
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > - *Mature design & coverage*: Includes core logic changes,
> > >>> > >> > runtime
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > handling, reporting, and E2E tests, making it suitable for a
> > >>> > >> > major
> > >>> > >> > > > > version
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > milestone.
> > >>> > >> > > > > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > ------------------------------
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > PR #10563 — [Feature][Connector-V2] Add checkpoint-gated
> > >>> > >> > > > post_sync_action
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > and retention for file source continuous discovery
> > >>> > >> > > > > >
> > >>> > >> > > > > >
> > >>> > https://github.com/apache/seatunnel/pull/10563
> > >>> > >> > > > > >
> > >>> > <
> > >>>
> >
> https://github.com/apache/seatunnel/pull/10563>> > > > > >
> > >>> >
> > >>> > ;
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > *What it adds / solves*
> > >>> > >> > > > > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > This PR adds *checkpoint-gated source-side lifecycle management*
> > >>> > >> > for
> > >>> > >> > > > file
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > source continuous discovery, including:
> > >>> > >> > > > > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > - post_sync_action = delete
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > Delete source files only *after a checkpoint has completed*
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > - post_sync_action = backup
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > Move source files to a backup_path after checkpoint
> > >>> > >> completion
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > - Optional backup retention via retention_max_age
> > >>> > >> > > > > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > All operations are *checkpoint-aware*, ensuring source files are
> > >>> > >> > never
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > deleted or moved before the data is durably processed.
> > >>> > >> > > > > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > *Why it should be included in 3.0*
> > >>> > >> > > > > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > - *Correctness-first continuous ingestion*: Safe source
> > >>> > >> cleanup
> > >>> > >> > tied
> > >>> > >> > > > > to
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > checkpoint completion is essential for exactly-once and
> > >>> > >> > recoverable
> > >>> > >> > > > > file
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > ingestion, which aligns strongly with 3.0 quality goals.
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > - *Production-grade operational semantics*: Without
> > >>> > >> > checkpoint-gated
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > cleanup, continuous file ingestion is risky in real
> > >>> > >> production
> > >>> > >> > > > > environments.
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > - *Extends Connector-V2 maturity*: This PR deepens the
> > >>> > >> > operational
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > completeness of Connector-V2, especially for file-based
> > >>> > >> sources.
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > - *Well-tested and documented*: Includes unit tests, E2E
> > >>> > >> tests
> > >>> > >> > > > (Local
> > >>> > >> > > > > +
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > HDFS), and documentation updates (EN + ZH).
> > >>> > >> > > > > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > ------------------------------
> > >>> > >> > > > > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > Both PRs are *opt-in*, backward-compatible, and focused on
> > >>> > >> making
> > >>> > >> > > > > SeaTunnel
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > more robust, predictable, and safe in real-world deployments. I
> > >>> > >> > believe
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > they fit naturally into the SeaTunnel 3.0 scope, especially
> > >>> > >> given
> > >>> > >> > the
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > emphasis on stability, correctness, and large-scale usage.
> > >>> > >> > > > > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > Happy to discuss details or answer questions.
> > >>> > >> > > > > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > Zhang Donghao <
> > >>> > zhangdonghao@apache.org> 于2026年4月29日周三 14:38写道：
> > >>> > >> > > > > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > Hi everyone,
> > >>> >
> >> > > > > > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > I'd like to propose including PR #9993 in the SeaTunnel 3.0
> > >>> > >> > roadmap,
> > >>> > >> > > > > here
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > are the details with additional packaging time analysis:
> > >>> >
> >> > > > > > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > ### PR link
> > >>> >
> > >>>
> >
> >> > > > > > >
> > >>> > https://github.com/apache/seatunnel/pull/9993
> > >>> > >> > > > > > >
> > >>> > <
> > >>>
> >
> https://github.com/apache/seatunnel/pull/9993>> > > > > > >
> > >>> >
> > >>> > ;
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > ### Short description
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > This PR extracts the shade modules into an independent
> > >>> > >> > > > seatunnel-shade
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > project, enables uploading shade-jars to the central
> > >>> > >> repository,
> > >>> > >> > and
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > optimizes the overall packaging efficiency of SeaTunnel.
> > >>> >
> >> > > > > > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > ### Packaging time analysis (current state before PR
> > >>> > >> > optimization)
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > The current build time breakdown of individual shade modules
> > >>> > >> is
> > >>> > >> > as
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > follows, highlighting the inefficiency of the integrated build
> > >>> > >> > > > > approach:
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > ```
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > [INFO] SeaTunnel : Shade : 3.0.0 ..........................
> > >>> > >> > SUCCESS [
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > 12.661 s]
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > [INFO] SeaTunnel : Shade : Hazelcast 5.1-3.0.0 ............
> > >>> > >> > SUCCESS [
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > 8.192 s]
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > [INFO] SeaTunnel : Shade : Hadoop3 Uber 3.1.4-3.0.0 .......
> > >>> > >> > SUCCESS
> > >>> > >> > > > > [02:29
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > min]
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > [INFO] SeaTunnel : Shade : Jackson 2.13.3-3.0.0 ...........
> > >>> > >> > SUCCESS [
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > 3.289 s]
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > [INFO] SeaTunnel : Shade : Guava 27.0-jre-3.0.0 ...........
> > >>> > >> > SUCCESS [
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > 2.960 s]
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > [INFO] SeaTunnel : Shade : Hadoop : AWS 3.1.4-3.0.0 .......
> > >>> > >> > SUCCESS
> > >>> > >> > > > > [05:05
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > min]
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > [INFO] SeaTunnel : Shade : Arrow 15.0.1-3.0.0 .............
> > >>> > >> > SUCCESS [
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > 5.880 s]
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > [INFO] SeaTunnel : Shade : Hikari 4.0.3-3.0.0 .............
> > >>> > >> > SUCCESS [
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > 1.085 s]
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > [INFO] SeaTunnel : Shade : Thrift-Service 1.0.0-3.0.0 .....
> > >>> > >> > SUCCESS [
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > 1.572 s]
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > [INFO] SeaTunnel : Shade : Janino 3.0.11-3.0.0 ............
> > >>> > >> > SUCCESS [
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > 1.235 s]
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > [INFO] SeaTunnel : Shade : Scala 2.13.11-3.0.0 ............
> > >>> > >> > SUCCESS [
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > 6.586 s]
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > [INFO] SeaTunnel : Shade : Jetty 9.4.56.v20240826-3.0.0 ...
> > >>> > >> > SUCCESS [
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > 2.421 s]
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > [INFO] SeaTunnel : Shade : Commons Lang3 3.18.0-3.0.0 .....
> > >>> > >> > SUCCESS [
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > 2.044 s]
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > ```
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > Key observations from the time breakdown:
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > 1. **Long-tail modules dominate build time**: Two modules
> > >>> > >> > account for
> > >>> > >> > > > > the
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > vast majority of shade build time — `Hadoop3 Uber` (2 minutes
> > >>> > >> 29
> > >>> > >> > > > > seconds)
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > and `Hadoop : AWS` (5 minutes 5 seconds), totaling ~7.5
> > >>> > >> minutes,
> > >>> > >> > > > while
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > other modules take only a few seconds each.
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > 2. **Integrated build waste**: In the current structure, these
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > time-consuming shade modules are built alongside core modules
> > >>> > >> > every
> > >>> > >> > > > > time
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > SeaTunnel is packaged, even when there are no changes to shade
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > dependencies. This results in repeated long build cycles for
> > >>> >
> > >>>
> > >> > > > > developers and
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > release workflows.
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > 3. **Total shade build time**: The cumulative build time of
> > >>> > >> all
> > >>> > >> > shade
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > modules is approximately 8 minutes (excluding
> > >>> > >> > overlapping/parallel
> > >>> > >> > > > > build
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > optimizations), which is a significant portion of the overall
> > >>> > >> > > > SeaTunnel
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > packaging process.
> > >>> >
> >> > > > > > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > ### Reason for including in 3.0
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > 1. **Build efficiency optimization (quantifiable gains)**:
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > - By isolating shade modules into an independent project
> > >>> > >> and
> > >>> > >> > > > > uploading
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > shade-jars to the central repository, we eliminate the need to
> > >>> > >> > > > rebuild
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > these modules in every SeaTunnel packaging cycle. For daily
> > >>> > >> > > > > development,
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > this reduces the total build time by ~8 minutes per build (for
> > >>> > >> > > > > developers
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > not modifying shade code).
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > - For the 3.0 release process, we can pre-build and publish
> > >>> > >> > > > > shade-jars
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > to the central repository once, then reference them as
> > >>> > >> > dependencies —
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > cutting the release packaging time by a substantial margin and
> > >>> >
> > >> > > > > reducing the
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > risk of build failures caused by long-running shade module
> > >>> > >> > builds.
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > 2. **Dependency management enhancement**: Uploading shade-jars
> > >>> > >> > to the
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > central repository standardizes dependency distribution,
> > >>> > >> avoiding
> > >>> > >> > > > > conflicts
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > caused by internal shade module duplication across
> > >>> > >> subprojects,
> > >>> > >> > and
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > improving the stability of the 3.0 version's dependency
> > >>> > >> system.
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > 3. **Project structure rationalization**: Isolating shade
> > >>> > >> modules
> > >>> > >> > > > > aligns
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > with the modular design concept of 3.0, making the project
> > >>> > >> > structure
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > clearer, easier to maintain, and facilitating future
> > >>> > >> expansion of
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > shade-related capabilities (e.g., adding new shade modules for
> > >>> > >> > > > > additional
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > dependencies).
> > >>> >
> >> > > > > > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > This optimization directly addresses the identified packaging
> > >>> > >> > > > > bottlenecks
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > (long-tail shade modules) and delivers measurable time savings
> > >>> > >> > for
> > >>> > >> > > > both
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > development and release workflows — a critical improvement for
> > >>> > >> > the
> > >>> > >> > > > > smooth
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > delivery and maintainability of SeaTunnel 3.0.
> > >>> >
> >> > > > > > >
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > Best regards,
> > >>> >
> > >>> >
> > >>>
> >
> >> > > > > > > zhangdonghao
> > >>> >
> >> > > > > > >
> > >>> > >> > > > >
> > >>> > >> > > >
> > >>> > >> >
> > >>> > >>
> > >>> > >
> > >>>
> > >>
> >
>
Re: [DISCUSS] Apache SeaTunnel 3.0 Roadmap
Posted by zhiwei niu <zn...@gmail.com>.
Hi community,
I would like to propose including STIP-28 in the SeaTunnel 3.0 roadmap.
STIP-28 enhances SeaTunnel's OptionRule framework from basic
presence/equality validation to richer declarative configuration
validation. This allows connector and transform configuration constraints
to be expressed as structured metadata instead of being hidden in scattered
runtime if/throw logic.
Related PRs:
- Main PR: apache/seatunnel#10977
https://github.com/apache/seatunnel/pull/10977
Completes the rich OptionRule validation framework.
- Follow-up PR: apache/seatunnel#11228
https://github.com/apache/seatunnel/pull/11228
Preserves extension validation error context.
- Follow-up PR: apache/seatunnel#11121
https://github.com/apache/seatunnel/pull/11121
Fixes validation error aggregation.
- Follow-up PR: apache/seatunnel#11010
https://github.com/apache/seatunnel/pull/11010
Adds Map-type condition validation.
- Follow-up PR: apache/seatunnel#11022
https://github.com/apache/seatunnel/pull/11022
Supports combining structural and value rules.
- Follow-up PR: apache/seatunnel#11048
https://github.com/apache/seatunnel/pull/11048
Adds custom extension validation support.
Why this should be included in 3.0:
- It makes SeaTunnel's configuration contract clearer and more explicit.
- Users can catch more configuration errors earlier through --check.
- REST API, Web UI, CLI metadata export, and future AI-assisted config
generation can consume richer validation metadata.
- Connector and transform developers can reduce duplicated imperative
validation logic.
- It provides a stronger foundation for future configuration generation,
validation, explanation, and repair capabilities.
STIP-28 framework capabilities have already been merged, and some
connectors have already completed migration. Therefore, I believe STIP-28
is ready to be included in the SeaTunnel 3.0 roadmap.
Best Regards,
zhiwei.niu
David Zollo <da...@gmail.com> 于2026年7月9日周四 21:19写道：
> Hi guys,
>
> Thanks for your hard work and make a great progress. Now we only have the
> following PRs left to be reviewed and merged.
> Below is the list of remaining PRs pending merge in the Roadmap:
>
> 1. [#10563](https://github.com/apache/seatunnel/pull/10563)
> [Feature][Connector-V2] Add checkpoint-gated post_sync_action and retention
> for file source continuous discovery @yzeng1618 PR conflicts and need to
> solve
> 2. [#9993](https://github.com/apache/seatunnel/pull/9993)
> [Feature][shade]Refactor the seatunnel-shade module. @hawk9821 PR
> conflicts and need to solve
> 3. [#10879](https://github.com/apache/seatunnel/pull/10879)
> [Feature][Zeta]
> STIP-24 Phase 2: Introduce standalone SeaTunnel Edge Agent MVP and dist
> integration @nzw921rx
> 4. [#10800](https://github.com/apache/seatunnel/pull/10800)
> [Feature][Zeta]
> STIP-23 Phase 1: Add engine timer flush core flow and FlushSignal handling
> @nzw921rx
> 5. [#10801](https://github.com/apache/seatunnel/pull/10801)
> [Feature][Connector-V2] STIP-23 Phase 2: JDBC connector adoption for engine
> timer flush @nzw921rx
> 6. [#10306](https://github.com/apache/seatunnel/pull/10306)
> [Feature][Core]
> Error Data Bypass Capability for Transform & Sink (Dead Letter Queue)
> @zhangshenghang
>
>
> Guys, please help review the PR for the 3.0 Roadmap when you have time.
>
>
> Best Regards,
> David
> Linkedin: https://www.linkedin.com/in/davidzollo
>
>
>
> On Wed, Jul 1, 2026 at 2:54 PM David Zollo <da...@gmail.com>
> wrote:
>
> > Hi guys,
> >
> > I created an issue for the 3.0 roadmap to track progress[1]. Given that
> > most PRs have been merged so far, we plan to finish merging the remaining
> > PRs by the end of July and then kick off the release process!
> >
> > 1. https://github.com/apache/seatunnel/issues/11021
> >
> >
> >
> >
> > Best Regards,
> > David
> > Linkedin: https://www.linkedin.com/in/davidzollo
> >
> >
> >
> > On Sat, May 16, 2026 at 12:32 PM David Zollo <da...@gmail.com>
> > wrote:
> >
> >> Hi SeaTunnel community,
> >>
> >> Thanks everyone for the active discussion around the Apache SeaTunnel
> 3.0
> >> roadmap.
> >>
> >> To make the follow-up review easier, I summarized the proposals
> collected
> >> in this thread so far. This is only a discussion summary, not a final
> >> decision. Please feel free to correct me if I missed anything or
> >> misunderstood any proposal.
> >>
> >> 1.
> >>
> >> SeaTunnel 3.0 roadmap discussion
> >>
> >> The discussion was started to collect proposals for Apache SeaTunnel 3.0
> >> after PR #10510, which updated the version to 3.0.0, was merged into the
> >> dev branch on March 11, 2026.
> >>
> >> Several already-merged PRs were mentioned as possible parts of the 3.0
> >> scope:
> >>
> >> -
> >>
> >> PR #10586: [Feature][Zeta] Add datasource spi
> >> https://github.com/apache/seatunnel/pull/10586
> >> -
> >>
> >> PR #10585: [Feature][Zeta] Add /option-rules REST API for runtime
> >> connector metadata
> >> https://github.com/apache/seatunnel/pull/10585
> >> -
> >>
> >> PR #10489: [Feature][Transform] Add Zeta SQL UDF context and
> >> lifecycle hooks with e2e/docs updates
> >> https://github.com/apache/seatunnel/pull/10489
> >>
> >> The purpose of this thread is to collect PRs, features, improvements,
> and
> >> ideas that the community believes should be considered for SeaTunnel
> 3.0.
> >>
> >> 2.
> >>
> >> STIP-23: Engine-level timer flush for Sink connectors
> >>
> >> zhiwei niu proposed including STIP-23 in the SeaTunnel 3.0 roadmap. The
> >> proposal focuses on engine-level timer flush support for Sink
> connectors.
> >>
> >> Related PRs:
> >>
> >> -
> >>
> >> PR #10800: [Feature][Zeta] Phase 1: Engine timer flush core
> >> https://github.com/apache/seatunnel/pull/10800
> >>
> >> This PR adds sink.flush.interval and introduces a FlushSignal
> >> control-plane path through the Zeta engine. It allows sinks to flush
> based
> >> on a wall-clock interval without each sink managing its own timer
> thread.
> >> -
> >>
> >> PR #10801: [Feature][Connector-V2] Phase 2: JDBC connector adoption
> >> https://github.com/apache/seatunnel/pull/10801
> >>
> >> This PR wires registerFlushAction into the JDBC non-XA sink writer
> >> through a new enable_timer_flush option. It also includes multi-table
> >> aggregated flush and MySQL CDC E2E tests.
> >> -
> >>
> >> PR #10802: [Docs][Connector-V2] Phase 3: Timer flush feature matrix
> >> https://github.com/apache/seatunnel/pull/10802
> >>
> >> This PR adds a timer flush support marker to sink connector
> >> documentation and documents the sink.flush.interval and
> registerFlushAction
> >> contract.
> >>
> >> The main value is to make end-to-end latency more predictable for
> >> buffered sinks, especially in idle or low-throughput streaming jobs.
> >>
> >> 3.
> >>
> >> Independent seatunnel-shade project and packaging efficiency
> >> optimization
> >>
> >> Zhang Donghao proposed including PR #9993 in the SeaTunnel 3.0 roadmap.
> >>
> >> -
> >>
> >> PR #9993:
> >> https://github.com/apache/seatunnel/pull/9993
> >>
> >> This PR extracts the shade modules into an independent seatunnel-shade
> >> project, supports uploading shade jars to Maven Central, and optimizes
> the
> >> overall packaging efficiency of SeaTunnel.
> >>
> >> The proposal points out that current shade module builds, especially
> >> Hadoop3 Uber and Hadoop AWS, add significant build time. By publishing
> >> shade jars separately and reusing them as dependencies, SeaTunnel can
> >> reduce repeated build cost and improve release stability.
> >>
> >> 4.
> >>
> >> Table-level fault isolation for multi-table synchronization
> >>
> >> yi zeng proposed including PR #10600 in the SeaTunnel 3.0 roadmap.
> >>
> >> -
> >>
> >> PR #10600: [Feature][Core] Support table-level fault isolation for
> >> multi-table synchronization
> >> https://github.com/apache/seatunnel/pull/10600
> >>
> >> This PR introduces table-level fault isolation for multi-table
> >> synchronization through a new failure policy:
> >>
> >> env {
> >> multi_table {
> >> failure_policy = "CONTINUE_OTHER_TABLES"
> >> }
> >> }
> >>
> >> With this enabled, failures can be isolated at table granularity.
> Healthy
> >> tables can continue processing even if one table fails. Failed tables
> are
> >> collected and reported with metadata such as table path, failure phase,
> >> plugin, and error.
> >>
> >> The proposal is backward-compatible by default because the existing
> >> FAIL_FAST behavior remains unchanged unless users explicitly enable the
> new
> >> policy.
> >>
> >> 5.
> >>
> >> Checkpoint-gated post_sync_action and retention for file source
> >> continuous discovery
> >>
> >> yi zeng also proposed including PR #10563.
> >>
> >> -
> >>
> >> PR #10563: [Feature][Connector-V2] Add checkpoint-gated
> >> post_sync_action and retention for file source continuous discovery
> >> https://github.com/apache/seatunnel/pull/10563
> >>
> >> This PR adds checkpoint-aware source-side lifecycle management for file
> >> source continuous discovery, including:
> >>
> >> -
> >>
> >> post_sync_action = delete
> >> -
> >>
> >> post_sync_action = backup
> >> -
> >>
> >> retention_max_age
> >>
> >> The key idea is that source files should only be deleted or moved after
> a
> >> checkpoint has completed, so that continuous file ingestion remains safe
> >> and recoverable in production environments.
> >>
> >> 6.
> >>
> >> STIP-21 / StainTrace: End-to-end data lineage tracing and performance
> >> analysis for Zeta
> >>
> >> David proposed including PR #10491 in the SeaTunnel 3.0 roadmap.
> >>
> >> -
> >>
> >> PR #10491:
> >> https://github.com/apache/seatunnel/pull/10491
> >>
> >> StainTrace is the implementation of STIP-21. It provides end-to-end data
> >> lineage tracing and performance analysis for the Zeta engine.
> >>
> >> It adds lightweight timestamp sampling points at six key pipeline
> stages:
> >>
> >> -
> >>
> >> SOURCE_EMIT
> >> -
> >>
> >> QUEUE_IN
> >> -
> >>
> >> QUEUE_OUT
> >> -
> >>
> >> TRANSFORM_IN
> >> -
> >>
> >> TRANSFORM_OUT
> >> -
> >>
> >> SINK_WRITE_DONE
> >>
> >> The goal is to track the full journey of sampled rows from source
> >> emission to sink write completion, making it easier to measure
> end-to-end
> >> latency and identify bottlenecks.
> >>
> >> The proposal emphasizes production safety: the feature is disabled by
> >> default, supports configurable sampling, has a per-worker event budget,
> >> writes trace data locally as JSONL in OpenTelemetry OTLP JSON format,
> and
> >> includes an offline analyzer for P95/P99 latency and HTML visualization
> >> reports.
> >>
> >> 7.
> >>
> >> StateStore abstraction for Zeta engine
> >>
> >> Doyeon Kim proposed including PR #10812 in the SeaTunnel 3.0 roadmap.
> >>
> >> -
> >>
> >> PR #10812:
> >> https://github.com/apache/seatunnel/pull/10812
> >>
> >> This PR introduces an initial StateStore abstraction layer to reduce
> >> direct Hazelcast IMap dependencies in the Zeta engine. It makes state
> >> responsibilities more explicit and prepares the engine for future
> evolution
> >> around state storage, node communication, and coordination layers.
> >>
> >> 8.
> >>
> >> IMap external storage compaction
> >>
> >> Doyeon Kim also proposed including PR #10399.
> >>
> >> -
> >>
> >> PR #10399:
> >> https://github.com/apache/seatunnel/pull/10399
> >>
> >> This PR introduces LSM-style write and compaction logic for IMap
> external
> >> storage. It helps control storage growth by flushing sorted immutable
> files
> >> and compacting persisted files while keeping only the latest record per
> key.
> >>
> >> The proposal is intended to improve stability for long-running Zeta jobs
> >> by reducing duplicated records and tombstones in external storage.
> >>
> >> 9.
> >>
> >> AI-Native CLI and AI for Data Engineering
> >>
> >> sean zhang proposed including the AI-Native CLI capability in SeaTunnel
> >> 3.0.
> >>
> >> -
> >>
> >> PR #10789:
> >> https://github.com/apache/seatunnel/pull/10789
> >>
> >> This PR introduces seatunnel-cli, an AI-powered command-line tool that
> >> can generate production-ready pipeline configurations from natural
> language.
> >>
> >> Current capabilities include:
> >>
> >> -
> >>
> >> Multi-LLM provider support, including Anthropic, OpenAI, and AWS
> >> Bedrock
> >> -
> >>
> >> Runtime connector metadata extraction via Java OptionRule reflection
> >> -
> >>
> >> Source/Sink type-aware validation
> >> -
> >>
> >> Auto-fix loop for generated configurations
> >> -
> >>
> >> Sensitive value redaction before persistence and LLM transmission
> >> -
> >>
> >> Interactive mode with session memory
> >> -
> >>
> >> One-shot mode for CI/CD
> >>
> >> The proposal also describes a broader “AI for Data Engineering” roadmap,
> >> including intelligent schema mapping, natural language transforms, data
> >> profiling, failure diagnosis, self-healing pipelines, anomaly detection,
> >> performance tuning, cost-aware planning, intelligent scheduling, and
> >> eventually autonomous data infrastructure management.
> >>
> >> 10.
> >>
> >> Extension of the roadmap discussion window
> >>
> >> David proposed extending the SeaTunnel 3.0 roadmap discussion window to
> >> May 15, 2026, considering the upcoming holiday period and to give
> >> contributors more time to share ideas.
> >>
> >> This message also recognized the quality and breadth of the proposals so
> >> far, including StainTrace, AI-Native CLI, StateStore abstraction, and
> >> table-level fault isolation.
> >>
> >> 11.
> >>
> >> STIP-27: Apache Arrow as an optional internal data model
> >>
> >> xuepeng wang proposed adding STIP-27 to the SeaTunnel 3.0 roadmap
> >> discussion.
> >>
> >> -
> >>
> >> Issue #10806:
> >> https://github.com/apache/seatunnel/issues/10806
> >>
> >> This proposal introduces Apache Arrow as an optional internal data
> model,
> >> not as a replacement for the existing row model.
> >>
> >> The goal is to improve performance for columnar scenarios such as
> >> Parquet, ORC, ClickHouse, Doris, and batch transforms such as
> projection,
> >> filter, and cast.
> >>
> >> Expected benefits include better columnar I/O performance, vectorized
> >> execution, and lower JVM overhead.
> >>
> >> 12.
> >>
> >> STIP-24: Edge collection, EdgeSocket ingress, and SeaTunnel Edge
> Agent
> >>
> >> zhiwei niu proposed adding STIP-24 edge-collection work to the SeaTunnel
> >> 3.0 roadmap.
> >>
> >> The original PRs were:
> >>
> >> -
> >>
> >> PR #10871: [Feature][Zeta] STIP-24 Phase 1: Support lightweight
> >> EdgeSocket ingress for edge collector MVP
> >> https://github.com/apache/seatunnel/pull/10871
> >> -
> >>
> >> PR #10876: [Feature][Zeta] STIP-24 Phase 2: Introduce standalone
> >> SeaTunnel Edge Agent MVP and dist integration
> >> https://github.com/apache/seatunnel/pull/10876
> >>
> >> Phase 1 introduces the engine-side ingress foundation for edge
> collection
> >> through EdgeSocket, including authenticated ingress, source-side
> protocol
> >> handling, and task-group address discovery support in Zeta.
> >>
> >> Phase 2 introduces a standalone SeaTunnel Edge Agent runtime, including
> >> core, transport, connector, local file/log/event inputs, SQLite WAL
> >> durability, EdgeSocket transport interoperability, and distribution
> >> integration.
> >>
> >> The goal is to support an end-to-end edge collection workflow:
> >>
> >> edge host -> Zeta pipeline
> >>
> >> zhiwei later updated the PR references for STIP-24:
> >>
> >> -
> >>
> >> PR #10871 was updated to PR #10878
> >> https://github.com/apache/seatunnel/pull/10878
> >> -
> >>
> >> PR #10876 was updated to PR #10879
> >> https://github.com/apache/seatunnel/pull/10879
> >>
> >>
> >> 13.
> >>
> >> Prometheus metrics for Zeta Engine IMap state stores
> >>
> >> JeremyXin proposed including PR #10860 in the SeaTunnel 3.0 roadmap.
> >>
> >> -
> >>
> >> PR #10860:
> >> https://github.com/apache/seatunnel/pull/10860
> >>
> >> This PR improves Zeta Engine observability by exposing Prometheus
> metrics
> >> for core engine state stores backed by Hazelcast IMap.
> >>
> >> It covers internal stores such as running job information, job states,
> >> checkpoints, and slot ownership. It exposes metrics such as local entry
> >> counts, backup entry counts, and heap cost.
> >>
> >> The proposal aims to help users detect state growth, data skew, and
> >> memory pressure earlier, improving troubleshooting, capacity planning,
> and
> >> production stability.
> >>
> >> 14.
> >>
> >> Real-time observability for Zeta engine tasks
> >>
> >> zhangshenghang proposed improving real-time observability for Zeta
> engine
> >> tasks.
> >>
> >> This direction adds the ability to monitor the runtime busyness level of
> >> Zeta engine tasks, including real-time task metrics for Source,
> Transform,
> >> and Sink execution. It also includes queue and backpressure-related
> >> information exposed through REST API and Web UI.
> >>
> >> The goal is to help users understand whether a job is blocked by source
> >> reading, transform processing, sink writing, or intermediate queues.
> >>
> >> 15.
> >>
> >> Error data bypass and dead-letter queue capability for Transform and
> >> Sink
> >>
> >> zhangshenghang also proposed row-level error handling for Transform and
> >> Sink stages.
> >>
> >> When configured, SeaTunnel can collect failed records with useful
> >> context, including:
> >>
> >> -
> >>
> >> error stage
> >> -
> >>
> >> plugin type
> >> -
> >>
> >> plugin name
> >> -
> >>
> >> source table
> >> -
> >>
> >> error message
> >> -
> >>
> >> exception class
> >> -
> >>
> >> original data
> >> -
> >>
> >> occurrence time
> >>
> >> This allows jobs to bypass selected dirty or error records instead of
> >> always failing fast, while keeping the default behavior unchanged when
> the
> >> feature is not enabled.
> >>
> >> This is useful for long-running synchronization jobs where users need
> >> better fault tolerance and later inspection of bad data.
> >>
> >> 16.
> >>
> >> Improve cloud-native deployment experience
> >>
> >> zhangshenghang also suggested improving the cloud-native deployment
> >> experience for SeaTunnel 3.0.
> >>
> >> The proposal mentions Kubernetes and Helm support as important areas,
> >> including better guidance and behavior around:
> >>
> >> -
> >>
> >> deployment
> >> -
> >>
> >> restart
> >> -
> >>
> >> persistent checkpoint storage
> >> -
> >>
> >> configuration management
> >> -
> >>
> >> secret management
> >> -
> >>
> >> rolling upgrade
> >> -
> >>
> >> resource isolation
> >>
> >>
> >> 17.
> >>
> >> Clean up deprecated behavior carefully for 3.0
> >>
> >> zhangshenghang also suggested that SeaTunnel 3.0 is a good opportunity
> to
> >> clean up deprecated options and old APIs.
> >>
> >> At the same time, any incompatible change should be handled carefully,
> >> with clear migration documentation, examples, and release notes.
> >>
> >> Possible discussion groups
> >>
> >> Based on the above proposals, I think the roadmap discussion can be
> >> grouped into the following areas:
> >>
> >> 1.
> >>
> >> Zeta engine observability
> >>
> >>
> >> -
> >>
> >> STIP-21 / StainTrace
> >> -
> >>
> >> Prometheus metrics for IMap state stores
> >> -
> >>
> >> Real-time task metrics
> >> -
> >>
> >> Queue and backpressure visibility
> >> -
> >>
> >> REST API and Web UI exposure
> >>
> >>
> >> 2.
> >>
> >> Zeta state management and storage evolution
> >>
> >>
> >> -
> >>
> >> StateStore abstraction
> >> -
> >>
> >> IMap external storage compaction
> >> -
> >>
> >> IMap state metrics
> >> -
> >>
> >> Future storage backend evolution
> >>
> >>
> >> 3.
> >>
> >> Runtime and connector semantics
> >>
> >>
> >> -
> >>
> >> Engine-level timer flush for Sink connectors
> >> -
> >>
> >> JDBC adoption of timer flush
> >> -
> >>
> >> Checkpoint-gated file source lifecycle management
> >> -
> >>
> >> Table-level fault isolation
> >> -
> >>
> >> Row-level error bypass and dead-letter queue
> >>
> >>
> >> 4.
> >>
> >> AI for Data Engineering
> >>
> >>
> >> -
> >>
> >> AI-Native CLI
> >> -
> >>
> >> Natural language pipeline generation
> >> -
> >>
> >> Connector metadata reflection
> >> -
> >>
> >> Auto-fix loop
> >> -
> >>
> >> Security and sensitive value redaction
> >> -
> >>
> >> Future AI-assisted schema mapping, diagnosis, optimization, and
> >> autonomous data management
> >>
> >>
> >> 5.
> >>
> >> Edge collection
> >>
> >>
> >> -
> >>
> >> EdgeSocket ingress
> >> -
> >>
> >> SeaTunnel Edge Agent
> >> -
> >>
> >> Edge host to Zeta pipeline workflow
> >> -
> >>
> >> Authentication, local durability, and transport interoperability
> >>
> >>
> >> 6.
> >>
> >> Performance and internal data model
> >>
> >>
> >> -
> >>
> >> Apache Arrow as an optional internal data model
> >> -
> >>
> >> Columnar I/O
> >> -
> >>
> >> Vectorized execution
> >> -
> >>
> >> Lower JVM overhead
> >>
> >>
> >> 7.
> >>
> >> Build, release, and project structure
> >>
> >>
> >> -
> >>
> >> Independent seatunnel-shade project
> >> -
> >>
> >> Maven Central distribution of shade jars
> >> -
> >>
> >> Reduced build and release time
> >>
> >>
> >> 8.
> >>
> >> Cloud-native and compatibility
> >>
> >>
> >> -
> >>
> >> Kubernetes and Helm deployment experience
> >> -
> >>
> >> Persistent checkpoint storage
> >> -
> >>
> >> Secret management
> >> -
> >>
> >> Rolling upgrade
> >> -
> >>
> >> Resource isolation
> >> -
> >>
> >> Deprecated behavior cleanup
> >> -
> >>
> >> Migration documentation and release notes
> >>
> >> Next steps
> >>
> >> I suggest we continue to review these proposals and decide:
> >>
> >> 1.
> >>
> >> Which items should be included in the SeaTunnel 3.0 roadmap.
> >> 2.
> >>
> >> Which items should be marked as experimental.
> >> 3.
> >>
> >> Which items require an STIP or further design discussion.
> >> 4.
> >>
> >> Which items can be merged independently before the 3.0 release.
> >> 5.
> >>
> >> Which items need migration documentation, compatibility notes, or
> >> release notes.
> >>
> >> Again, this is only a summary of the current discussion thread. Please
> >> help correct or supplement anything I missed.
> >>
> >>
> >>
> >> Best Regards,
> >> David
> >> Linkedin: https://www.linkedin.com/in/davidzollo
> >>
> >>
> >>
> >> On Thu, May 14, 2026 at 2:59 PM zhangshenghang <sh...@apache.org>
> >> wrote:
> >>
> >>> Hi community,
> >>>
> >>> For the SeaTunnel 3.0 roadmap
> >>>
> >>> I have already submitted two related PRs, and besides those, I would
> like
> >>> to propose the following directions for 3.0:
> >>>
> >>> 1. Real-time observability for Zeta engine tasks
> >>>
> >>> This work adds the ability to monitor the runtime busyness level of
> Zeta
> >>> engine tasks. It introduces real-time task metrics for Source,
> Transform,
> >>> and Sink execution, including queue/backpressure-related information,
> and
> >>> exposes these metrics through REST API and Web UI. This can help users
> >>> understand whether a job is blocked by source reading, transform
> >>> processing, sink writing, or intermediate queues, which is very
> important
> >>> for production troubleshooting and performance tuning.
> >>>
> >>> 2. Error data bypass and dead-letter queue capability for Transform and
> >>> Sink
> >>>
> >>> This work introduces row-level error handling for Transform and Sink
> >>> stages. When configured, SeaTunnel can collect failed records with
> useful
> >>> context such as error stage, plugin type, plugin name, source table,
> >>> error
> >>> message, exception class, original data, and occurrence time. This
> allows
> >>> jobs to bypass selected dirty/error records instead of always failing
> >>> fast,
> >>> while keeping the default behavior unchanged when the feature is not
> >>> enabled. This is useful for long-running synchronization jobs where
> users
> >>> need better fault tolerance and later inspection of bad data.
> >>>
> >>> 3. Improve cloud-native deployment experience
> >>>
> >>> Kubernetes and Helm support are important for new users. We should
> >>> improve
> >>> guidance and behavior around deployment, restart, persistent checkpoint
> >>> storage, configuration management, secret management, rolling upgrade,
> >>> and
> >>> resource isolation in cloud-native environments.
> >>>
> >>> 4. Clean up deprecated behavior carefully for 3.0
> >>>
> >>> Since 3.0 is a major version, it is a good opportunity to clean up
> >>> deprecated options and old APIs, but we should do this carefully. Any
> >>> incompatible change should have clear migration documentation,
> examples,
> >>> and release notes.
> >>>
> >>>
> >>> Regards,
> >>> Jast (Shenghang)
> >>>
> >>>
> >>> JeremyXin <73...@qq.com> 于2026年5月14日周四 14:27写道：
> >>>
> >>> > Hi everyone, I’d like to propose adding this PR to the SeaTunnel 3.0
> >>> > roadmap.
> >>> >
> >>> >
> >>> > - PR#10860
> >>> > - link: https://github.com/apache/seatunnel/pull/10860
> >>> >
> >>> >
> >>> > This PR improves Zeta Engine observability by exposing Prometheus
> >>> metrics
> >>> > for core engine state stores backed by Hazelcast IMap.
> >>> >
> >>> >
> >>> > It covers important internal stores such as running job information,
> >>> job
> >>> > states, checkpoints, slot ownership and provides visibility into
> their
> >>> > local entry counts, backup entry counts, and heap cost.
> >>> >
> >>> >
> >>> > Including this in SeaTunnel 3.0 would make the engine easier to
> monitor
> >>> > and operate in production. It helps users detect state growth, data
> >>> skew,
> >>> > and memory pressure earlier, which improves troubleshooting
> efficiency,
> >>> > cluster capacity planning, and overall operational stability.
> >>> >
> >>> >
> >>> > Best regards,
> >>> > JeremyXin
> >>> >
> >>> >
> >>> > 原始邮件
> >>> >
> >>> >
> >>> > 发件人：zhiwei niu <zniu70696@gmail.com>
> >>> > 发件时间：2026年5月12日 10:27
> >>> > 收件人：dev <dev@seatunnel.apache.org>
> >>> > 主题：Re: [DISCUSS] Apache SeaTunnel 3.0 Roadmap
> >>> >
> >>> >
> >>> >
> >>> > Hi everyone,
> >>> >
> >>> >
> >>> >
> >>>
> As part of Roadmap 3.0 progress tracking, the STIP-24 PR links have been
> >>> >
> >>> >
> >>>
> updated to keep references consistent and easier to follow.
> >>> >
> >>> > Please use the updated PR links below:
> >>> >
> >>> >
> >>> >
> >>>
> [Feature][Zeta] STIP-24 Phase 1: Support lightweight EdgeSocket ingress
> >>> > for edge collector MVP
> >>> > - PR #10871(
> >>> > https://github.com/apache/seatunnel/pull/10871)
> >>> > —> PR#10878(
> >>> > https://github.com/apache/seatunnel/pull/10878)
> >>> >
> >>> >
> >>> >
> >>>
> [Feature][Zeta] STIP-24 Phase 2: Introduce standalone SeaTunnel Edge Agent
> >>> > MVP and dist integration
> >>> > - PR #10876(
> >>> > https://github.com/apache/seatunnel/pull/10876)
> >>> > — PR#10879(
> >>> > https://github.com/apache/seatunnel/pull/10879)
> >>> >
> >>> >
> >>> > zhiwei niu <zniu70696@gmail.com
> >>> > > 于2026年5月12日周二 01:31写道：
> >>> >
> >>> > > Hi everyone,
> >>> > >
> >>> >
> >>> >
> >>>
> > In addition to STIP-23, I’d like to propose the STIP-24 edge-collection
> >>> > > work for SeaTunnel 3.0.
> >>> > >
> >>> >
> >>> >
> >>>
> > - PR #10871 — [Feature][Zeta] STIP-24 Phase 1: Support lightweight
> >>> >
> >>> >
> >>>
> > EdgeSocket ingress for edge collector MVP
> >>> > > https://github.com/apache/seatunnel/pull/10871
> >>> > > What it adds:
> >>> >
> >>> >
> >>>
> > Introduces the engine-side ingress foundation for edge collection via
> >>> >
> >>> >
> >>>
> > EdgeSocket, including authenticated ingress, source-side protocol handling,
> >>> >
> >>> >
> >>>
> > and task-group address discovery support in Zeta.
> >>> > > Why 3.0:
> >>> >
> >>> >
> >>>
> > This is the foundational engine capability for remote edge data collection
> >>> >
> >>> >
> >>>
> > scenarios. It is backward-compatible and establishes the protocol/runtime
> >>> >
> >>>
> > path required for the edge-side agent.
> >>> > >
> >>> >
> >>> >
> >>>
> > - PR #10876 — [Feature][Zeta] STIP-24 Phase 2: Introduce standalone
> >>> >
> >>> >
> >>>
> > SeaTunnel Edge Agent MVP and dist integration
> >>> > > https://github.com/apache/seatunnel/pull/10876
> >>> > > What it adds:
> >>> >
> >>> >
> >>>
> > Introduces a standalone SeaTunnel Edge Agent runtime
> >>> >
> >>> >
> >>>
> > (core/transport/connector), local file/log/event inputs, SQLite WAL
> >>> >
> >>> >
> >>>
> > durability, EdgeSocket transport interop, and distribution integration.
> >>> > > Why 3.0:
> >>> >
> >>> >
> >>>
> > This delivers the first practical end-to-end edge collection workflow
> >>> >
> >>> >
> >>>
> > (edge host -> Zeta pipeline), which significantly expands SeaTunnel’s
> >>> >
> >>> >
> >>>
> > deployment topology and real-world ingestion coverage.
> >>> > >
> >>> >
> >>> >
> >>>
> > PR #10876 depends on #10871; both are under active review.
> >>> > >
> >>> > > Best regards,
> >>> > > zhiwei.niu
> >>> > >
> >>> > > xuepeng wang <wxuepeng5@gmail.com
> >>> > > 于2026年5月8日周五 14:03写道：
> >>> > >
> >>> > >> Hi team,
> >>> > >>
> >>> > >>
> >>> > >>
> >>> >
> >>> >
> >>>
> >> I'd like to add STIP-27 [1] to the SeaTunnel 3.0 Roadmap discussion.
> >>> > >>
> >>> >
> >>> >
> >>>
> >> This proposal introduces Apache Arrow as an optional internal data model
> >>> >
> >>> >
> >>>
> >> (not replacing the existing row model). It aims to improve performance for
> >>> >
> >>> >
> >>>
> >> columnar scenarios (Parquet/ORC, ClickHouse/Doris) and batch transforms
> >>> >
> >>> >
> >>>
> >> (projection/filter/cast), while keeping the current architecture stable.
> >>> > >>
> >>> >
> >>> >
> >>>
> >> Key benefits: better columnar I/O performance, vectorized execution,
> >>> > >> lower JVM overhead.
> >>> > >>
> >>> > >> [1]
> >>> > https://github.com/apache/seatunnel/issues/10806
> >>> > >> <https://github.com/apache/seatunnel/issues/10806>>>;
> >>> > >> Thoughts?
> >>> > >>
> >>> > >> Regards,
> >>> > >> xuepeng(github id:chl-wxp)
> >>> > >>
> >>> > >> David Zollo <davidzollo365@gmail.com
> >>> > > 于2026年4月30日周四 21:17写道：
> >>> > >>
> >>> > >> > Hi everyone,
> >>> > >> >
> >>> >
> >>> >
> >>>
> >> > First of all, thank you all for the fantastic proposals so far — the
> >>> >
> >>> >
> >>>
> >> > quality and breadth of ideas in this thread have been genuinely
> >>> >
> >>> >
> >>>
> >> > exciting to read. From StainTrace to the AI-Native CLI, from
> >>> >
> >>> >
> >>>
> >> > StateStore abstraction to table-level fault isolation, it's clear that
> >>> >
> >>> >
> >>>
> >> > this community has a bold and thoughtful vision for what SeaTunnel 3.0
> >>> > >> > can become.
> >>> > >> >
> >>> >
> >>> >
> >>>
> >> > Given that the Golden Week holiday is just around the corner, and many
> >>> >
> >>> >
> >>>
> >> > of our contributors are based in China or taking time off during this
> >>> >
> >>> >
> >>>
> >> > period, I'd like to extend the discussion window to May 15th. This
> >>> >
> >>> >
> >>>
> >> > gives everyone enough breathing room to enjoy the holiday, come back
> >>> >
> >>> >
> >>>
> >> > refreshed, and still have time to share their thoughts without feeling
> >>> > >> > rushed.
> >>> > >> >
> >>> >
> >>>
> >> > Happy Labor Day to everyone!
> >>> > >> >
> >>> >
> >>> >
> >>>
> >> > We highly welcome everyone's suggestions for version 3.0! Feel free to
> >>> >
> >>> >
> >>>
> >> > leave a message if you have any ideas.
> >>> > >> >
> >>> > >> >
> >>> > >> >
> >>> > >> > Best Regards,
> >>> > >> > David
> >>> > >> > Linkedin:
> >>> > https://www.linkedin.com/in/davidzollo
> >>> > >> >
> >>> > <https://www.linkedin.com/in/davidzollo>> >>;
> >>> >
> >>> >
> >>>
> >> > On Wed, Apr 29, 2026 at 8:50 PM sean zhang <
> >>> > seanzhang92@gmail.com>
> >>> > >> wrote:
> >>> > >> > >
> >>> > >> > > Hi SeaTunnel community,
> >>> > >> > >
> >>> >
> >>> >
> >>>
> >> > > I'd like to propose including the AI-Native CLI capability in the
> >>> >
> >>> >
> >>>
> >> > > SeaTunnel 3.0 release, and share a broader vision for the "AI for Data
> >>> >
> >>> >
> >>>
> >> > > Engineering" direction. I believe this aligns well with the 3.0
> >>> > >> milestone
> >>> >
> >>> >
> >>>
> >> > > as a significant user-experience leap.
> >>> > >> > > This PR
> >>> > https://github.com/apache/seatunnel/pull/10789 introduces
> >>> >
> >>> >
> >>>
> >> > > `seatunnel-cli`, an AI-powered command-line tool that
> >>> >
> >>> >
> >>>
> >> > > generates production-ready pipeline configurations from natural
> >>> > >> language
> >>> > <
> >>>
> https://github.com/apache/seatunnel/pull/10789 introduces>> > > seatunnel-cli, an AI-powered command-line tool that>> > > generates production-ready pipeline configurations from natural>> language
> >>> >
> >>> > :
> >>> > >> > >
> >>> >
> >>> >
> >>>
> >> > > $ seatunnel "Sync the users table from MySQL to S3 in parquet
> >>> > >> format"
> >>> >
> >>> >
> >>>
> >> > > ✅ Generated config: mysql_users_to_s3.conf
> >>> > >> > >
> >>> >
> >>> >
> >>>
> >> > > Key capabilities in the current implementation:
> >>> > >> > >
> >>> >
> >>> >
> >>>
> >> > > - Multi-LLM provider support (Anthropic / OpenAI / AWS Bedrock)
> >>> >
> >>> >
> >>>
> >> > > with auto-detection
> >>> >
> >>> >
> >>>
> >> > > - Runtime connector metadata extraction via Java OptionRule
> >>> > >> reflection,
> >>> >
> >>> >
> >>>
> >> > > ensuring 100% accurate parameter knowledge for all 100+ connectors
> >>> >
> >>> >
> >>>
> >> > > - Source/Sink type-aware validation (Jdbc source options ≠ Jdbc sink
> >>> > >> > > options)
> >>> >
> >>> >
> >>>
> >> > > - Auto-fix loop: generated configs are validated and self-corrected
> >>> >
> >>> >
> >>>
> >> > > (up to 3 iterations)
> >>> >
> >>> >
> >>>
> >> > > - Security hardening: sensitive values (passwords, tokens, keys) are
> >>> >
> >>> >
> >>>
> >> > > automatically redacted to ${ENV_VAR} placeholders before
> >>> > >> persistence
> >>> >
> >>> >
> >>>
> >> > > and LLM transmission
> >>> >
> >>> >
> >>>
> >> > > - Interactive mode with session memory and one-shot mode for CI/CD
> >>> > >> > >
> >>> >
> >>> >
> >>>
> >> > > Future Roadmap — AI for Data Engineering
> >>> > >> > >
> >>> >
> >>> >
> >>>
> >> > > Looking beyond config generation, I see a multi-level roadmap where
> >>> >
> >>> >
> >>>
> >> > > AI becomes deeply integrated into SeaTunnel's core value proposition:
> >>> > >> > >
> >>> >
> >>> >
> >>>
> >> > > ### Level 2: AI Understands Data (Next phase)
> >>> > >> > >
> >>> >
> >>> >
> >>>
> >> > > - Intelligent Schema Mapping: AI reads source schema, infers target
> >>> >
> >>> >
> >>>
> >> > > schema, and auto-generates type conversions and DDL
> >>> >
> >>> >
> >>>
> >> > > - Natural Language Transforms: "mask phone numbers, merge first and
> >>> >
> >>> >
> >>>
> >> > > last name, filter last 7 days" → SQL Transform auto-generation
> >>> >
> >>> >
> >>>
> >> > > - Data Profiling: AI samples source data, detects anomalies (NULLs,
> >>> >
> >>> >
> >>>
> >> > > outliers, format violations), and suggests data quality transforms
> >>> > >> > >
> >>> >
> >>> >
> >>>
> >> > > ### Level 3: AI Operates Pipelines
> >>> > >> > >
> >>> >
> >>> >
> >>>
> >> > > - Intelligent Failure Diagnosis: AI analyzes error logs, pinpoints
> >>> >
> >>> >
> >>>
> >> > > root cause (connection refused → MySQL down vs firewall), suggests
> >>> > >> > fix
> >>> >
> >>> >
> >>>
> >> > > - Self-Healing Pipelines: auto-detect DDL changes in CDC sources,
> >>> >
> >>> >
> >>>
> >> > > auto-evolve sink schema, zero-downtime recovery
> >>> >
> >>> >
> >>>
> >> > > - Anomaly Detection: monitor data flow volume/latency, alert when
> >>> >
> >>> >
> >>>
> >> > > throughput drops 99% with correlated root cause
> >>> > >> > >
> >>> >
> >>> >
> >>>
> >> > > ### Level 4: AI Optimizes Everything
> >>> > >> > >
> >>> >
> >>> >
> >>>
> >> > > - Performance Tuning: AI analyzes bottlenecks (batch_size too small,
> >>> >
> >>> >
> >>>
> >> > > wrong file format, insufficient parallelism), suggests
> >>> > >> optimization
> >>> >
> >>> >
> >>>
> >> > > - Cost-Aware Planning: given N tables to sync, AI proposes pipeline
> >>> >
> >>> >
> >>>
> >> > > topology that balances cost vs latency vs resource usage
> >>> >
> >>> >
> >>>
> >> > > - Intelligent Scheduling: learn from historical patterns to optimize
> >>> >
> >>> >
> >>>
> >> > > sync frequency per table
> >>> > >> > >
> >>> >
> >>> >
> >>>
> >> > > ### Level 5: Autonomous Data Platform (Long-term vision)
> >>> > >> > >
> >>> >
> >>> >
> >>>
> >> > > - Conversational data infrastructure management: "onboard the
> >>> > >> payment
> >>> >
> >>> >
> >>>
> >> > > system to the data lake, ensure GDPR compliance" → AI
> >>> > >> auto-discovers
> >>> >
> >>> >
> >>>
> >> > > sources, identifies PII, generates pipelines with masking
> >>> > >> transforms,
> >>> >
> >>> >
> >>>
> >> > > deploys with monitoring and alerting
> >>> > >> > >
> >>> > >> > >
> >>> > >> > > Doyeon Kim <
> >>> > ekagnlek2@gmail.com> 于2026年4月29日周三 18:17写道：
> >>> > >> > >
> >>> > >> > > > Hi everyone,
> >>> > >> > > >
> >>> >
> >>> >
> >>>
> >> > > > I’d like to propose the following Zeta engine improvements for
> >>> > >> > SeaTunnel
> >>> > >> > > > 3.0.
> >>> > >> > > >
> >>> >
> >>> >
> >>>
> >> > > > 1. StateStore abstraction
> >>> > >> > > > PR:
> >>> > https://github.com/apache/seatunnel/pull/10812
> >>> > >> > > >
> >>> > <
> >>>
> https://github.com/apache/seatunnel/pull/10812>> > > >
> >>> >
> >>> > ;
> >>> >
> >>> >
> >>>
> >> > > > This introduces an initial StateStore abstraction layer to reduce
> >>> > >> > direct
> >>> >
> >>> >
> >>>
> >> > > > Hazelcast IMap dependencies in the engine. It makes different state
> >>> >
> >>> >
> >>>
> >> > > > responsibilities more explicit and prepares the engine for future
> >>> > >> > backend
> >>> > >> > > > evolution.
> >>> > >> > > >
> >>> >
> >>> >
> >>>
> >> > > > I think this is relevant for 3.0 because it is a foundational step
> >>> > >> > toward
> >>> >
> >>> >
> >>>
> >> > > > decoupling the engine from Hazelcast-specific APIs and enabling
> >>> > >> future
> >>> >
> >>> >
> >>>
> >> > > > improvements around state storage, node communication, and
> >>> > >> coordination
> >>> > >> > > > layers.
> >>> > >> > > >
> >>> >
> >>> >
> >>>
> >> > > > 2. IMap external storage compaction
> >>> > >> > > > PR:
> >>> > https://github.com/apache/seatunnel/pull/10399
> >>> > >> > > >
> >>> > <
> >>>
> https://github.com/apache/seatunnel/pull/10399>> > > >
> >>> >
> >>> > ;
> >>> >
> >>> >
> >>>
> >> > > > This introduces LSM-style write and compaction logic for IMap
> >>> > >> external
> >>> >
> >>> >
> >>>
> >> > > > storage. It helps control storage growth by flushing sorted
> >>> > >> immutable
> >>> > >> > files
> >>> >
> >>> >
> >>>
> >> > > > and compacting persisted files while keeping only the latest record
> >>> > >> per
> >>> > >> > > > key.
> >>> > >> > > >
> >>> >
> >>> >
> >>>
> >> > > > I think this should be considered for 3.0 because long-running Zeta
> >>> > >> > jobs
> >>> >
> >>> >
> >>>
> >> > > > may accumulate duplicated records and tombstones in external storage
> >>> > >> > over
> >>> >
> >>> >
> >>>
> >> > > > time. Compaction can improve stability and reduce the risk of
> >>> > >> unbounded
> >>> > >> > > > storage growth.
> >>> > >> > > >
> >>> > >> > > > Best Regards,
> >>> > >> > > > Doyeon
> >>> >
> >>>
> >> > > > (Github ID: dybyte)
> >>> > >> > > >
> >>> >
> >>> >
> >>>
> >> > > > On Wed, Apr 29, 2026 at 7:13 PM David Zollo <
> >>> > >> davidzollo365@gmail.com>
> >>> > >> > > > wrote:
> >>> > >> > > >
> >>> > >> > > > > Hi guys,
> >>> > >> > > > >
> >>> >
> >>> >
> >>>
> >> > > > > I'd like to propose including PR #10491 in the SeaTunnel 3.0
> >>> > >> roadmap,
> >>> >
> >>> >
> >>>
> >> > > > > here are the details with additional packaging time analysis:
> >>> > >> > > > >
> >>> >
> >>>
> >> > > > > ### PR link
> >>> > >> > > > >
> >>> > https://github.com/apache/seatunnel/pull/10491
> >>> > >> > > > >
> >>> > <
> >>>
> https://github.com/apache/seatunnel/pull/10491>> > > > >
> >>> >
> >>> > ;
> >>> >
> >>> >
> >>>
> >> > > > > StainTrace (PR #10491) is the concrete implementation of STIP-21,
> >>> >
> >>> >
> >>>
> >> > > > > delivering end-to-end data lineage tracing and performance
> >>> > >> analysis
> >>> >
> >>> >
> >>>
> >> > > > > capabilities for the Apache SeaTunnel Zeta engine. It injects
> >>> >
> >>> >
> >>>
> >> > > > > lightweight timestamp sampling points at 6 key pipeline stages —
> >>> >
> >>> >
> >>>
> >> > > > > SOURCE_EMIT → QUEUE_IN → QUEUE_OUT → TRANSFORM_IN → TRANSFORM_OUT
> >>> > >> →
> >>> >
> >>> >
> >>>
> >> > > > > SINK_WRITE_DONE — tracking each sampled row's complete journey
> >>> > >> from
> >>> >
> >>> >
> >>>
> >> > > > > source emission to sink write completion, enabling precise
> >>> > >> end-to-end
> >>> >
> >>> >
> >>>
> >> > > > > latency measurement and bottleneck identification.
> >>> > >> > > > >
> >>> >
> >>> >
> >>>
> >> > > > > The system is designed with production safety as a first-class
> >>> >
> >>> >
> >>>
> >> > > > > concern: it is disabled by default with very low overhead, and
> >>> > >> when
> >>> >
> >>> >
> >>>
> >> > > > > enabled it operates at a configurable sampling rate with a strict
> >>> >
> >>> >
> >>>
> >> > > > > per-worker per-second event budget, achieving approximately 0.1~1%
> >>> >
> >>> >
> >>>
> >> > > > > overhead at 1% sampling. Trace data is written locally as JSONL
> >>> > >> files
> >>> >
> >>> >
> >>>
> >> > > > > in OpenTelemetry OTLP JSON format with no external dependencies.
> >>> > >> An
> >>> >
> >>> >
> >>>
> >> > > > > accompanying offline analyzer reads these files to compute P95/P99
> >>> >
> >>> >
> >>>
> >> > > > > latency statistics, identify bottleneck stages, and generate an
> >>> > >> HTML
> >>> >
> >>> >
> >>>
> >> > > > > visualization report with Chinese/English language toggle support.
> >>> > >> > > > >
> >>> >
> >>> >
> >>>
> >> > > > > The value to SeaTunnel lies in closing a critical observability
> >>> > >> gap
> >>> > >> > at
> >>> >
> >>> >
> >>>
> >> > > > > the engine level. Before this feature, users facing latency
> >>> > >> spikes or
> >>> >
> >>> >
> >>>
> >> > > > > throughput bottlenecks had no structured way to pinpoint whether
> >>> > >> the
> >>> >
> >>> >
> >>>
> >> > > > > problem resided in queue backpressure, transform processing, or
> >>> > >> sink
> >>> >
> >>> >
> >>>
> >> > > > > write time — they could only rely on logs and guesswork.
> >>> > >> StainTrace
> >>> >
> >>> >
> >>>
> >> > > > > makes per-stage timing transparent and actionable, which is
> >>> > >> > especially
> >>> >
> >>> >
> >>>
> >> > > > > impactful for large-scale, multi-table, long-chain production
> >>> > >> > > > > pipelines.
> >>> > >> > > > >
> >>> >
> >>> >
> >>>
> >> > > > > The case for including it in the 3.0 roadmap is equally strong. As
> >>> > >> > the
> >>> >
> >>> >
> >>>
> >> > > > > implementation of STIP-21, StainTrace is a framework-level
> >>> >
> >>> >
> >>>
> >> > > > > observability capability — connector-agnostic and immediately
> >>> >
> >>> >
> >>>
> >> > > > > effective across all jobs once merged. One of SeaTunnel 3.0's
> >>> > >> central
> >>> >
> >>> >
> >>>
> >> > > > > goals is to become a more mature, production-ready data
> >>> > >> integration
> >>> >
> >>> >
> >>>
> >> > > > > engine, and observability is an indispensable foundation for that
> >>> >
> >>> >
> >>>
> >> > > > > goal. Establishing it at a major version boundary like 3.0
> >>> > >> delivers
> >>> >
> >>> >
> >>>
> >> > > > > far greater long-term value to users.
> >>> > >> > > > >
> >>> > >> > > > >
> >>> > >> > > > >
> >>> > >> > > > >
> >>> > >> > > > >
> >>> > >> > > > >
> >>> >
> >>>
> >> > > > > Best Regards,
> >>> > >> > > > > David
> >>> > >> > > > > Linkedin:
> >>> > https://www.linkedin.com/in/davidzollo
> >>> > >> > > > >
> >>> > <
> >>>
> https://www.linkedin.com/in/davidzollo>> > > > >
> >>> >
> >>> > ;
> >>> >
> >>> >
> >>>
> >> > > > > On Wed, Apr 29, 2026 at 2:48 PM yi zeng <
> >>> > yzeng1618@gmail.com>
> >>> > >> wrote:
> >>> > >> > > > > >
> >>> >
> >>> >
> >>>
> >> > > > > > Hi everyone,
> >>> > >> > > > > >
> >>> >
> >>> >
> >>>
> >> > > > > > I’d like to propose including the following two PRs in the
> >>> > >> > *SeaTunnel
> >>> > >> > > > 3.0
> >>> >
> >>> >
> >>>
> >> > > > > > roadmap*. Both focus on strengthening *correctness, reliability,
> >>> > >> > and
> >>> >
> >>> >
> >>>
> >> > > > > > operational safety* for core multi-table and continuous
> >>> > >> ingestion
> >>> >
> >>> >
> >>>
> >> > > > > > scenarios, which I believe are key pillars for 3.0.
> >>> >
> >>> >
> >>>
> >> > > > > > ------------------------------
> >>> >
> >>> >
> >>>
> >> > > > > > PR #10600 — [Feature][Core] Support table-level fault isolation
> >>> > >> for
> >>> >
> >>> >
> >>>
> >> > > > > > multi-table synchronization
> >>> > >> > > > > >
> >>> > >> > > > > >
> >>> > https://github.com/apache/seatunnel/pull/10600
> >>> > >> > > > > >
> >>> > <
> >>>
> https://github.com/apache/seatunnel/pull/10600>> > > > > >
> >>> >
> >>> > ;
> >>> >
> >>> >
> >>>
> >> > > > > > *What it adds / solves*
> >>> > >> > > > > >
> >>> >
> >>> >
> >>>
> >> > > > > > This PR introduces *table-level fault isolation* for multi-table
> >>> >
> >>> >
> >>>
> >> > > > > > synchronization by adding a new failure policy:
> >>> > >> > > > > > code
> >>> > >> > > > > > 复制
> >>> > >> > > > > >
> >>> >
> >>>
> >> > > > > > env {
> >>> >
> >>> >
> >>>
> >> > > > > > multi_table {
> >>> >
> >>> >
> >>>
> >> > > > > > failure_policy = "CONTINUE_OTHER_TABLES"
> >>> >
> >>> >
> >>>
> >> > > > > > }
> >>> > >> > > > > > }
> >>> > >> > > > > >
> >>> >
> >>> >
> >>>
> >> > > > > > With this enabled:
> >>> > >> > > > > >
> >>> >
> >>> >
> >>>
> >> > > > > > - Failures are isolated at *table granularity*, instead of
> >>> > >> > failing
> >>> > >> > > > the
> >>> >
> >>> >
> >>>
> >> > > > > > entire job.
> >>> >
> >>> >
> >>>
> >> > > > > > - Healthy tables continue processing even if one table fails.
> >>> >
> >>> >
> >>>
> >> > > > > > - Failed tables are collected and reported with clear
> >>> > >> metadata
> >>> > >> > > > (table
> >>> >
> >>> >
> >>>
> >> > > > > > path, failure phase, plugin, error).
> >>> >
> >>> >
> >>>
> >> > > > > > - Batch jobs report failed tables in final results; streaming
> >>> > >> > jobs
> >>> >
> >>> >
> >>>
> >> > > > > > continue running with healthy tables.
> >>> > >> > > > > >
> >>> >
> >>> >
> >>>
> >> > > > > > This addresses a long-standing pain point where a single
> >>> > >> > problematic
> >>> > >> > > > > table
> >>> >
> >>> >
> >>>
> >> > > > > > can block progress for all others.
> >>> > >> > > > > >
> >>> >
> >>> >
> >>>
> >> > > > > > *Why it should be included in 3.0*
> >>> > >> > > > > >
> >>> >
> >>> >
> >>>
> >> > > > > > - *Improved fault tolerance model*: 3.0 aims to be more
> >>> > >> > resilient
> >>> > >> > > > and
> >>> >
> >>> >
> >>>
> >> > > > > > production-ready; table-level isolation is a foundational
> >>> > >> > capability
> >>> > >> > > > > for
> >>> >
> >>> >
> >>>
> >> > > > > > large-scale, multi-table pipelines.
> >>> >
> >>> >
> >>>
> >> > > > > > - *Backward-compatible by default*: The default behavior
> >>> > >> remains
> >>> >
> >>> >
> >>>
> >> > > > > > FAIL_FAST, so existing jobs are unaffected unless users
> >>> > >> > explicitly
> >>> > >> > > > opt
> >>> >
> >>> >
> >>>
> >> > > > > > in.
> >>> >
> >>> >
> >>>
> >> > > > > > - *Framework-level improvement*: This is not
> >>> > >> > connector-specific; it
> >>> >
> >>> >
> >>>
> >> > > > > > enhances the core execution and failure-handling semantics of
> >>> > >> > > > > SeaTunnel.
> >>> >
> >>> >
> >>>
> >> > > > > > - *Mature design & coverage*: Includes core logic changes,
> >>> > >> > runtime
> >>> >
> >>> >
> >>>
> >> > > > > > handling, reporting, and E2E tests, making it suitable for a
> >>> > >> > major
> >>> > >> > > > > version
> >>> >
> >>> >
> >>>
> >> > > > > > milestone.
> >>> > >> > > > > >
> >>> >
> >>> >
> >>>
> >> > > > > > ------------------------------
> >>> >
> >>> >
> >>>
> >> > > > > > PR #10563 — [Feature][Connector-V2] Add checkpoint-gated
> >>> > >> > > > post_sync_action
> >>> >
> >>> >
> >>>
> >> > > > > > and retention for file source continuous discovery
> >>> > >> > > > > >
> >>> > >> > > > > >
> >>> > https://github.com/apache/seatunnel/pull/10563
> >>> > >> > > > > >
> >>> > <
> >>>
> https://github.com/apache/seatunnel/pull/10563>> > > > > >
> >>> >
> >>> > ;
> >>> >
> >>> >
> >>>
> >> > > > > > *What it adds / solves*
> >>> > >> > > > > >
> >>> >
> >>> >
> >>>
> >> > > > > > This PR adds *checkpoint-gated source-side lifecycle management*
> >>> > >> > for
> >>> > >> > > > file
> >>> >
> >>> >
> >>>
> >> > > > > > source continuous discovery, including:
> >>> > >> > > > > >
> >>> >
> >>> >
> >>>
> >> > > > > > - post_sync_action = delete
> >>> >
> >>> >
> >>>
> >> > > > > > Delete source files only *after a checkpoint has completed*
> >>> >
> >>> >
> >>>
> >> > > > > > - post_sync_action = backup
> >>> >
> >>> >
> >>>
> >> > > > > > Move source files to a backup_path after checkpoint
> >>> > >> completion
> >>> >
> >>> >
> >>>
> >> > > > > > - Optional backup retention via retention_max_age
> >>> > >> > > > > >
> >>> >
> >>> >
> >>>
> >> > > > > > All operations are *checkpoint-aware*, ensuring source files are
> >>> > >> > never
> >>> >
> >>> >
> >>>
> >> > > > > > deleted or moved before the data is durably processed.
> >>> > >> > > > > >
> >>> >
> >>> >
> >>>
> >> > > > > > *Why it should be included in 3.0*
> >>> > >> > > > > >
> >>> >
> >>> >
> >>>
> >> > > > > > - *Correctness-first continuous ingestion*: Safe source
> >>> > >> cleanup
> >>> > >> > tied
> >>> > >> > > > > to
> >>> >
> >>> >
> >>>
> >> > > > > > checkpoint completion is essential for exactly-once and
> >>> > >> > recoverable
> >>> > >> > > > > file
> >>> >
> >>> >
> >>>
> >> > > > > > ingestion, which aligns strongly with 3.0 quality goals.
> >>> >
> >>> >
> >>>
> >> > > > > > - *Production-grade operational semantics*: Without
> >>> > >> > checkpoint-gated
> >>> >
> >>> >
> >>>
> >> > > > > > cleanup, continuous file ingestion is risky in real
> >>> > >> production
> >>> > >> > > > > environments.
> >>> >
> >>> >
> >>>
> >> > > > > > - *Extends Connector-V2 maturity*: This PR deepens the
> >>> > >> > operational
> >>> >
> >>> >
> >>>
> >> > > > > > completeness of Connector-V2, especially for file-based
> >>> > >> sources.
> >>> >
> >>> >
> >>>
> >> > > > > > - *Well-tested and documented*: Includes unit tests, E2E
> >>> > >> tests
> >>> > >> > > > (Local
> >>> > >> > > > > +
> >>> >
> >>> >
> >>>
> >> > > > > > HDFS), and documentation updates (EN + ZH).
> >>> > >> > > > > >
> >>> >
> >>> >
> >>>
> >> > > > > > ------------------------------
> >>> > >> > > > > >
> >>> >
> >>> >
> >>>
> >> > > > > > Both PRs are *opt-in*, backward-compatible, and focused on
> >>> > >> making
> >>> > >> > > > > SeaTunnel
> >>> >
> >>> >
> >>>
> >> > > > > > more robust, predictable, and safe in real-world deployments. I
> >>> > >> > believe
> >>> >
> >>> >
> >>>
> >> > > > > > they fit naturally into the SeaTunnel 3.0 scope, especially
> >>> > >> given
> >>> > >> > the
> >>> >
> >>> >
> >>>
> >> > > > > > emphasis on stability, correctness, and large-scale usage.
> >>> > >> > > > > >
> >>> >
> >>> >
> >>>
> >> > > > > > Happy to discuss details or answer questions.
> >>> > >> > > > > >
> >>> >
> >>> >
> >>>
> >> > > > > > Zhang Donghao <
> >>> > zhangdonghao@apache.org> 于2026年4月29日周三 14:38写道：
> >>> > >> > > > > >
> >>> >
> >>> >
> >>>
> >> > > > > > > Hi everyone,
> >>> > >> > > > > > >
> >>> >
> >>> >
> >>>
> >> > > > > > > I'd like to propose including PR #9993 in the SeaTunnel 3.0
> >>> > >> > roadmap,
> >>> > >> > > > > here
> >>> >
> >>> >
> >>>
> >> > > > > > > are the details with additional packaging time analysis:
> >>> > >> > > > > > >
> >>> >
> >>> >
> >>>
> >> > > > > > > ### PR link
> >>> >
> >>>
> >> > > > > > >
> >>> > https://github.com/apache/seatunnel/pull/9993
> >>> > >> > > > > > >
> >>> > <
> >>>
> https://github.com/apache/seatunnel/pull/9993>> > > > > > >
> >>> >
> >>> > ;
> >>> >
> >>> >
> >>>
> >> > > > > > > ### Short description
> >>> >
> >>> >
> >>>
> >> > > > > > > This PR extracts the shade modules into an independent
> >>> > >> > > > seatunnel-shade
> >>> >
> >>> >
> >>>
> >> > > > > > > project, enables uploading shade-jars to the central
> >>> > >> repository,
> >>> > >> > and
> >>> >
> >>> >
> >>>
> >> > > > > > > optimizes the overall packaging efficiency of SeaTunnel.
> >>> > >> > > > > > >
> >>> >
> >>> >
> >>>
> >> > > > > > > ### Packaging time analysis (current state before PR
> >>> > >> > optimization)
> >>> >
> >>> >
> >>>
> >> > > > > > > The current build time breakdown of individual shade modules
> >>> > >> is
> >>> > >> > as
> >>> >
> >>> >
> >>>
> >> > > > > > > follows, highlighting the inefficiency of the integrated build
> >>> > >> > > > > approach:
> >>> >
> >>> >
> >>>
> >> > > > > > > ```
> >>> >
> >>> >
> >>>
> >> > > > > > > [INFO] SeaTunnel : Shade : 3.0.0 ..........................
> >>> > >> > SUCCESS [
> >>> >
> >>> >
> >>>
> >> > > > > > > 12.661 s]
> >>> >
> >>> >
> >>>
> >> > > > > > > [INFO] SeaTunnel : Shade : Hazelcast 5.1-3.0.0 ............
> >>> > >> > SUCCESS [
> >>> >
> >>> >
> >>>
> >> > > > > > > 8.192 s]
> >>> >
> >>> >
> >>>
> >> > > > > > > [INFO] SeaTunnel : Shade : Hadoop3 Uber 3.1.4-3.0.0 .......
> >>> > >> > SUCCESS
> >>> > >> > > > > [02:29
> >>> >
> >>> >
> >>>
> >> > > > > > > min]
> >>> >
> >>> >
> >>>
> >> > > > > > > [INFO] SeaTunnel : Shade : Jackson 2.13.3-3.0.0 ...........
> >>> > >> > SUCCESS [
> >>> >
> >>> >
> >>>
> >> > > > > > > 3.289 s]
> >>> >
> >>> >
> >>>
> >> > > > > > > [INFO] SeaTunnel : Shade : Guava 27.0-jre-3.0.0 ...........
> >>> > >> > SUCCESS [
> >>> >
> >>> >
> >>>
> >> > > > > > > 2.960 s]
> >>> >
> >>> >
> >>>
> >> > > > > > > [INFO] SeaTunnel : Shade : Hadoop : AWS 3.1.4-3.0.0 .......
> >>> > >> > SUCCESS
> >>> > >> > > > > [05:05
> >>> >
> >>> >
> >>>
> >> > > > > > > min]
> >>> >
> >>> >
> >>>
> >> > > > > > > [INFO] SeaTunnel : Shade : Arrow 15.0.1-3.0.0 .............
> >>> > >> > SUCCESS [
> >>> >
> >>> >
> >>>
> >> > > > > > > 5.880 s]
> >>> >
> >>> >
> >>>
> >> > > > > > > [INFO] SeaTunnel : Shade : Hikari 4.0.3-3.0.0 .............
> >>> > >> > SUCCESS [
> >>> >
> >>> >
> >>>
> >> > > > > > > 1.085 s]
> >>> >
> >>> >
> >>>
> >> > > > > > > [INFO] SeaTunnel : Shade : Thrift-Service 1.0.0-3.0.0 .....
> >>> > >> > SUCCESS [
> >>> >
> >>> >
> >>>
> >> > > > > > > 1.572 s]
> >>> >
> >>> >
> >>>
> >> > > > > > > [INFO] SeaTunnel : Shade : Janino 3.0.11-3.0.0 ............
> >>> > >> > SUCCESS [
> >>> >
> >>> >
> >>>
> >> > > > > > > 1.235 s]
> >>> >
> >>> >
> >>>
> >> > > > > > > [INFO] SeaTunnel : Shade : Scala 2.13.11-3.0.0 ............
> >>> > >> > SUCCESS [
> >>> >
> >>> >
> >>>
> >> > > > > > > 6.586 s]
> >>> >
> >>> >
> >>>
> >> > > > > > > [INFO] SeaTunnel : Shade : Jetty 9.4.56.v20240826-3.0.0 ...
> >>> > >> > SUCCESS [
> >>> >
> >>> >
> >>>
> >> > > > > > > 2.421 s]
> >>> >
> >>> >
> >>>
> >> > > > > > > [INFO] SeaTunnel : Shade : Commons Lang3 3.18.0-3.0.0 .....
> >>> > >> > SUCCESS [
> >>> >
> >>> >
> >>>
> >> > > > > > > 2.044 s]
> >>> >
> >>> >
> >>>
> >> > > > > > > ```
> >>> >
> >>> >
> >>>
> >> > > > > > > Key observations from the time breakdown:
> >>> >
> >>> >
> >>>
> >> > > > > > > 1. **Long-tail modules dominate build time**: Two modules
> >>> > >> > account for
> >>> > >> > > > > the
> >>> >
> >>> >
> >>>
> >> > > > > > > vast majority of shade build time — `Hadoop3 Uber` (2 minutes
> >>> > >> 29
> >>> > >> > > > > seconds)
> >>> >
> >>> >
> >>>
> >> > > > > > > and `Hadoop : AWS` (5 minutes 5 seconds), totaling ~7.5
> >>> > >> minutes,
> >>> > >> > > > while
> >>> >
> >>> >
> >>>
> >> > > > > > > other modules take only a few seconds each.
> >>> >
> >>> >
> >>>
> >> > > > > > > 2. **Integrated build waste**: In the current structure, these
> >>> >
> >>> >
> >>>
> >> > > > > > > time-consuming shade modules are built alongside core modules
> >>> > >> > every
> >>> > >> > > > > time
> >>> >
> >>> >
> >>>
> >> > > > > > > SeaTunnel is packaged, even when there are no changes to shade
> >>> >
> >>> >
> >>>
> >> > > > > > > dependencies. This results in repeated long build cycles for
> >>> >
> >>>
> >> > > > > developers and
> >>> >
> >>> >
> >>>
> >> > > > > > > release workflows.
> >>> >
> >>> >
> >>>
> >> > > > > > > 3. **Total shade build time**: The cumulative build time of
> >>> > >> all
> >>> > >> > shade
> >>> >
> >>> >
> >>>
> >> > > > > > > modules is approximately 8 minutes (excluding
> >>> > >> > overlapping/parallel
> >>> > >> > > > > build
> >>> >
> >>> >
> >>>
> >> > > > > > > optimizations), which is a significant portion of the overall
> >>> > >> > > > SeaTunnel
> >>> >
> >>> >
> >>>
> >> > > > > > > packaging process.
> >>> > >> > > > > > >
> >>> >
> >>> >
> >>>
> >> > > > > > > ### Reason for including in 3.0
> >>> >
> >>> >
> >>>
> >> > > > > > > 1. **Build efficiency optimization (quantifiable gains)**:
> >>> >
> >>> >
> >>>
> >> > > > > > > - By isolating shade modules into an independent project
> >>> > >> and
> >>> > >> > > > > uploading
> >>> >
> >>> >
> >>>
> >> > > > > > > shade-jars to the central repository, we eliminate the need to
> >>> > >> > > > rebuild
> >>> >
> >>> >
> >>>
> >> > > > > > > these modules in every SeaTunnel packaging cycle. For daily
> >>> > >> > > > > development,
> >>> >
> >>> >
> >>>
> >> > > > > > > this reduces the total build time by ~8 minutes per build (for
> >>> > >> > > > > developers
> >>> >
> >>> >
> >>>
> >> > > > > > > not modifying shade code).
> >>> >
> >>> >
> >>>
> >> > > > > > > - For the 3.0 release process, we can pre-build and publish
> >>> > >> > > > > shade-jars
> >>> >
> >>> >
> >>>
> >> > > > > > > to the central repository once, then reference them as
> >>> > >> > dependencies —
> >>> >
> >>> >
> >>>
> >> > > > > > > cutting the release packaging time by a substantial margin and
> >>> >
> >> > > > > reducing the
> >>> >
> >>> >
> >>>
> >> > > > > > > risk of build failures caused by long-running shade module
> >>> > >> > builds.
> >>> >
> >>> >
> >>>
> >> > > > > > > 2. **Dependency management enhancement**: Uploading shade-jars
> >>> > >> > to the
> >>> >
> >>> >
> >>>
> >> > > > > > > central repository standardizes dependency distribution,
> >>> > >> avoiding
> >>> > >> > > > > conflicts
> >>> >
> >>> >
> >>>
> >> > > > > > > caused by internal shade module duplication across
> >>> > >> subprojects,
> >>> > >> > and
> >>> >
> >>> >
> >>>
> >> > > > > > > improving the stability of the 3.0 version's dependency
> >>> > >> system.
> >>> >
> >>> >
> >>>
> >> > > > > > > 3. **Project structure rationalization**: Isolating shade
> >>> > >> modules
> >>> > >> > > > > aligns
> >>> >
> >>> >
> >>>
> >> > > > > > > with the modular design concept of 3.0, making the project
> >>> > >> > structure
> >>> >
> >>> >
> >>>
> >> > > > > > > clearer, easier to maintain, and facilitating future
> >>> > >> expansion of
> >>> >
> >>> >
> >>>
> >> > > > > > > shade-related capabilities (e.g., adding new shade modules for
> >>> > >> > > > > additional
> >>> >
> >>> >
> >>>
> >> > > > > > > dependencies).
> >>> > >> > > > > > >
> >>> >
> >>> >
> >>>
> >> > > > > > > This optimization directly addresses the identified packaging
> >>> > >> > > > > bottlenecks
> >>> >
> >>> >
> >>>
> >> > > > > > > (long-tail shade modules) and delivers measurable time savings
> >>> > >> > for
> >>> > >> > > > both
> >>> >
> >>> >
> >>>
> >> > > > > > > development and release workflows — a critical improvement for
> >>> > >> > the
> >>> > >> > > > > smooth
> >>> >
> >>> >
> >>>
> >> > > > > > > delivery and maintainability of SeaTunnel 3.0.
> >>> > >> > > > > > >
> >>> >
> >>> >
> >>>
> >> > > > > > > Best regards,
> >>> >
> >>> >
> >>>
> >> > > > > > > zhangdonghao
> >>> > >> > > > > > >
> >>> > >> > > > >
> >>> > >> > > >
> >>> > >> >
> >>> > >>
> >>> > >
> >>>
> >>
>
Re: [DISCUSS] Apache SeaTunnel 3.0 Roadmap
Posted by David Zollo <da...@gmail.com>.
Hi guys,
Thanks for your hard work and make a great progress. Now we only have the
following PRs left to be reviewed and merged.
Below is the list of remaining PRs pending merge in the Roadmap:
1. [#10563](https://github.com/apache/seatunnel/pull/10563)
[Feature][Connector-V2] Add checkpoint-gated post_sync_action and retention
for file source continuous discovery @yzeng1618 PR conflicts and need to
solve
2. [#9993](https://github.com/apache/seatunnel/pull/9993)
[Feature][shade]Refactor the seatunnel-shade module. @hawk9821 PR
conflicts and need to solve
3. [#10879](https://github.com/apache/seatunnel/pull/10879) [Feature][Zeta]
STIP-24 Phase 2: Introduce standalone SeaTunnel Edge Agent MVP and dist
integration @nzw921rx
4. [#10800](https://github.com/apache/seatunnel/pull/10800) [Feature][Zeta]
STIP-23 Phase 1: Add engine timer flush core flow and FlushSignal handling
@nzw921rx
5. [#10801](https://github.com/apache/seatunnel/pull/10801)
[Feature][Connector-V2] STIP-23 Phase 2: JDBC connector adoption for engine
timer flush @nzw921rx
6. [#10306](https://github.com/apache/seatunnel/pull/10306) [Feature][Core]
Error Data Bypass Capability for Transform & Sink (Dead Letter Queue)
@zhangshenghang
Guys, please help review the PR for the 3.0 Roadmap when you have time.
Best Regards,
David
Linkedin: https://www.linkedin.com/in/davidzollo
On Wed, Jul 1, 2026 at 2:54 PM David Zollo <da...@gmail.com> wrote:
> Hi guys,
>
> I created an issue for the 3.0 roadmap to track progress[1]. Given that
> most PRs have been merged so far, we plan to finish merging the remaining
> PRs by the end of July and then kick off the release process!
>
> 1. https://github.com/apache/seatunnel/issues/11021
>
>
>
>
> Best Regards,
> David
> Linkedin: https://www.linkedin.com/in/davidzollo
>
>
>
> On Sat, May 16, 2026 at 12:32 PM David Zollo <da...@gmail.com>
> wrote:
>
>> Hi SeaTunnel community,
>>
>> Thanks everyone for the active discussion around the Apache SeaTunnel 3.0
>> roadmap.
>>
>> To make the follow-up review easier, I summarized the proposals collected
>> in this thread so far. This is only a discussion summary, not a final
>> decision. Please feel free to correct me if I missed anything or
>> misunderstood any proposal.
>>
>> 1.
>>
>> SeaTunnel 3.0 roadmap discussion
>>
>> The discussion was started to collect proposals for Apache SeaTunnel 3.0
>> after PR #10510, which updated the version to 3.0.0, was merged into the
>> dev branch on March 11, 2026.
>>
>> Several already-merged PRs were mentioned as possible parts of the 3.0
>> scope:
>>
>> -
>>
>> PR #10586: [Feature][Zeta] Add datasource spi
>> https://github.com/apache/seatunnel/pull/10586
>> -
>>
>> PR #10585: [Feature][Zeta] Add /option-rules REST API for runtime
>> connector metadata
>> https://github.com/apache/seatunnel/pull/10585
>> -
>>
>> PR #10489: [Feature][Transform] Add Zeta SQL UDF context and
>> lifecycle hooks with e2e/docs updates
>> https://github.com/apache/seatunnel/pull/10489
>>
>> The purpose of this thread is to collect PRs, features, improvements, and
>> ideas that the community believes should be considered for SeaTunnel 3.0.
>>
>> 2.
>>
>> STIP-23: Engine-level timer flush for Sink connectors
>>
>> zhiwei niu proposed including STIP-23 in the SeaTunnel 3.0 roadmap. The
>> proposal focuses on engine-level timer flush support for Sink connectors.
>>
>> Related PRs:
>>
>> -
>>
>> PR #10800: [Feature][Zeta] Phase 1: Engine timer flush core
>> https://github.com/apache/seatunnel/pull/10800
>>
>> This PR adds sink.flush.interval and introduces a FlushSignal
>> control-plane path through the Zeta engine. It allows sinks to flush based
>> on a wall-clock interval without each sink managing its own timer thread.
>> -
>>
>> PR #10801: [Feature][Connector-V2] Phase 2: JDBC connector adoption
>> https://github.com/apache/seatunnel/pull/10801
>>
>> This PR wires registerFlushAction into the JDBC non-XA sink writer
>> through a new enable_timer_flush option. It also includes multi-table
>> aggregated flush and MySQL CDC E2E tests.
>> -
>>
>> PR #10802: [Docs][Connector-V2] Phase 3: Timer flush feature matrix
>> https://github.com/apache/seatunnel/pull/10802
>>
>> This PR adds a timer flush support marker to sink connector
>> documentation and documents the sink.flush.interval and registerFlushAction
>> contract.
>>
>> The main value is to make end-to-end latency more predictable for
>> buffered sinks, especially in idle or low-throughput streaming jobs.
>>
>> 3.
>>
>> Independent seatunnel-shade project and packaging efficiency
>> optimization
>>
>> Zhang Donghao proposed including PR #9993 in the SeaTunnel 3.0 roadmap.
>>
>> -
>>
>> PR #9993:
>> https://github.com/apache/seatunnel/pull/9993
>>
>> This PR extracts the shade modules into an independent seatunnel-shade
>> project, supports uploading shade jars to Maven Central, and optimizes the
>> overall packaging efficiency of SeaTunnel.
>>
>> The proposal points out that current shade module builds, especially
>> Hadoop3 Uber and Hadoop AWS, add significant build time. By publishing
>> shade jars separately and reusing them as dependencies, SeaTunnel can
>> reduce repeated build cost and improve release stability.
>>
>> 4.
>>
>> Table-level fault isolation for multi-table synchronization
>>
>> yi zeng proposed including PR #10600 in the SeaTunnel 3.0 roadmap.
>>
>> -
>>
>> PR #10600: [Feature][Core] Support table-level fault isolation for
>> multi-table synchronization
>> https://github.com/apache/seatunnel/pull/10600
>>
>> This PR introduces table-level fault isolation for multi-table
>> synchronization through a new failure policy:
>>
>> env {
>> multi_table {
>> failure_policy = "CONTINUE_OTHER_TABLES"
>> }
>> }
>>
>> With this enabled, failures can be isolated at table granularity. Healthy
>> tables can continue processing even if one table fails. Failed tables are
>> collected and reported with metadata such as table path, failure phase,
>> plugin, and error.
>>
>> The proposal is backward-compatible by default because the existing
>> FAIL_FAST behavior remains unchanged unless users explicitly enable the new
>> policy.
>>
>> 5.
>>
>> Checkpoint-gated post_sync_action and retention for file source
>> continuous discovery
>>
>> yi zeng also proposed including PR #10563.
>>
>> -
>>
>> PR #10563: [Feature][Connector-V2] Add checkpoint-gated
>> post_sync_action and retention for file source continuous discovery
>> https://github.com/apache/seatunnel/pull/10563
>>
>> This PR adds checkpoint-aware source-side lifecycle management for file
>> source continuous discovery, including:
>>
>> -
>>
>> post_sync_action = delete
>> -
>>
>> post_sync_action = backup
>> -
>>
>> retention_max_age
>>
>> The key idea is that source files should only be deleted or moved after a
>> checkpoint has completed, so that continuous file ingestion remains safe
>> and recoverable in production environments.
>>
>> 6.
>>
>> STIP-21 / StainTrace: End-to-end data lineage tracing and performance
>> analysis for Zeta
>>
>> David proposed including PR #10491 in the SeaTunnel 3.0 roadmap.
>>
>> -
>>
>> PR #10491:
>> https://github.com/apache/seatunnel/pull/10491
>>
>> StainTrace is the implementation of STIP-21. It provides end-to-end data
>> lineage tracing and performance analysis for the Zeta engine.
>>
>> It adds lightweight timestamp sampling points at six key pipeline stages:
>>
>> -
>>
>> SOURCE_EMIT
>> -
>>
>> QUEUE_IN
>> -
>>
>> QUEUE_OUT
>> -
>>
>> TRANSFORM_IN
>> -
>>
>> TRANSFORM_OUT
>> -
>>
>> SINK_WRITE_DONE
>>
>> The goal is to track the full journey of sampled rows from source
>> emission to sink write completion, making it easier to measure end-to-end
>> latency and identify bottlenecks.
>>
>> The proposal emphasizes production safety: the feature is disabled by
>> default, supports configurable sampling, has a per-worker event budget,
>> writes trace data locally as JSONL in OpenTelemetry OTLP JSON format, and
>> includes an offline analyzer for P95/P99 latency and HTML visualization
>> reports.
>>
>> 7.
>>
>> StateStore abstraction for Zeta engine
>>
>> Doyeon Kim proposed including PR #10812 in the SeaTunnel 3.0 roadmap.
>>
>> -
>>
>> PR #10812:
>> https://github.com/apache/seatunnel/pull/10812
>>
>> This PR introduces an initial StateStore abstraction layer to reduce
>> direct Hazelcast IMap dependencies in the Zeta engine. It makes state
>> responsibilities more explicit and prepares the engine for future evolution
>> around state storage, node communication, and coordination layers.
>>
>> 8.
>>
>> IMap external storage compaction
>>
>> Doyeon Kim also proposed including PR #10399.
>>
>> -
>>
>> PR #10399:
>> https://github.com/apache/seatunnel/pull/10399
>>
>> This PR introduces LSM-style write and compaction logic for IMap external
>> storage. It helps control storage growth by flushing sorted immutable files
>> and compacting persisted files while keeping only the latest record per key.
>>
>> The proposal is intended to improve stability for long-running Zeta jobs
>> by reducing duplicated records and tombstones in external storage.
>>
>> 9.
>>
>> AI-Native CLI and AI for Data Engineering
>>
>> sean zhang proposed including the AI-Native CLI capability in SeaTunnel
>> 3.0.
>>
>> -
>>
>> PR #10789:
>> https://github.com/apache/seatunnel/pull/10789
>>
>> This PR introduces seatunnel-cli, an AI-powered command-line tool that
>> can generate production-ready pipeline configurations from natural language.
>>
>> Current capabilities include:
>>
>> -
>>
>> Multi-LLM provider support, including Anthropic, OpenAI, and AWS
>> Bedrock
>> -
>>
>> Runtime connector metadata extraction via Java OptionRule reflection
>> -
>>
>> Source/Sink type-aware validation
>> -
>>
>> Auto-fix loop for generated configurations
>> -
>>
>> Sensitive value redaction before persistence and LLM transmission
>> -
>>
>> Interactive mode with session memory
>> -
>>
>> One-shot mode for CI/CD
>>
>> The proposal also describes a broader “AI for Data Engineering” roadmap,
>> including intelligent schema mapping, natural language transforms, data
>> profiling, failure diagnosis, self-healing pipelines, anomaly detection,
>> performance tuning, cost-aware planning, intelligent scheduling, and
>> eventually autonomous data infrastructure management.
>>
>> 10.
>>
>> Extension of the roadmap discussion window
>>
>> David proposed extending the SeaTunnel 3.0 roadmap discussion window to
>> May 15, 2026, considering the upcoming holiday period and to give
>> contributors more time to share ideas.
>>
>> This message also recognized the quality and breadth of the proposals so
>> far, including StainTrace, AI-Native CLI, StateStore abstraction, and
>> table-level fault isolation.
>>
>> 11.
>>
>> STIP-27: Apache Arrow as an optional internal data model
>>
>> xuepeng wang proposed adding STIP-27 to the SeaTunnel 3.0 roadmap
>> discussion.
>>
>> -
>>
>> Issue #10806:
>> https://github.com/apache/seatunnel/issues/10806
>>
>> This proposal introduces Apache Arrow as an optional internal data model,
>> not as a replacement for the existing row model.
>>
>> The goal is to improve performance for columnar scenarios such as
>> Parquet, ORC, ClickHouse, Doris, and batch transforms such as projection,
>> filter, and cast.
>>
>> Expected benefits include better columnar I/O performance, vectorized
>> execution, and lower JVM overhead.
>>
>> 12.
>>
>> STIP-24: Edge collection, EdgeSocket ingress, and SeaTunnel Edge Agent
>>
>> zhiwei niu proposed adding STIP-24 edge-collection work to the SeaTunnel
>> 3.0 roadmap.
>>
>> The original PRs were:
>>
>> -
>>
>> PR #10871: [Feature][Zeta] STIP-24 Phase 1: Support lightweight
>> EdgeSocket ingress for edge collector MVP
>> https://github.com/apache/seatunnel/pull/10871
>> -
>>
>> PR #10876: [Feature][Zeta] STIP-24 Phase 2: Introduce standalone
>> SeaTunnel Edge Agent MVP and dist integration
>> https://github.com/apache/seatunnel/pull/10876
>>
>> Phase 1 introduces the engine-side ingress foundation for edge collection
>> through EdgeSocket, including authenticated ingress, source-side protocol
>> handling, and task-group address discovery support in Zeta.
>>
>> Phase 2 introduces a standalone SeaTunnel Edge Agent runtime, including
>> core, transport, connector, local file/log/event inputs, SQLite WAL
>> durability, EdgeSocket transport interoperability, and distribution
>> integration.
>>
>> The goal is to support an end-to-end edge collection workflow:
>>
>> edge host -> Zeta pipeline
>>
>> zhiwei later updated the PR references for STIP-24:
>>
>> -
>>
>> PR #10871 was updated to PR #10878
>> https://github.com/apache/seatunnel/pull/10878
>> -
>>
>> PR #10876 was updated to PR #10879
>> https://github.com/apache/seatunnel/pull/10879
>>
>>
>> 13.
>>
>> Prometheus metrics for Zeta Engine IMap state stores
>>
>> JeremyXin proposed including PR #10860 in the SeaTunnel 3.0 roadmap.
>>
>> -
>>
>> PR #10860:
>> https://github.com/apache/seatunnel/pull/10860
>>
>> This PR improves Zeta Engine observability by exposing Prometheus metrics
>> for core engine state stores backed by Hazelcast IMap.
>>
>> It covers internal stores such as running job information, job states,
>> checkpoints, and slot ownership. It exposes metrics such as local entry
>> counts, backup entry counts, and heap cost.
>>
>> The proposal aims to help users detect state growth, data skew, and
>> memory pressure earlier, improving troubleshooting, capacity planning, and
>> production stability.
>>
>> 14.
>>
>> Real-time observability for Zeta engine tasks
>>
>> zhangshenghang proposed improving real-time observability for Zeta engine
>> tasks.
>>
>> This direction adds the ability to monitor the runtime busyness level of
>> Zeta engine tasks, including real-time task metrics for Source, Transform,
>> and Sink execution. It also includes queue and backpressure-related
>> information exposed through REST API and Web UI.
>>
>> The goal is to help users understand whether a job is blocked by source
>> reading, transform processing, sink writing, or intermediate queues.
>>
>> 15.
>>
>> Error data bypass and dead-letter queue capability for Transform and
>> Sink
>>
>> zhangshenghang also proposed row-level error handling for Transform and
>> Sink stages.
>>
>> When configured, SeaTunnel can collect failed records with useful
>> context, including:
>>
>> -
>>
>> error stage
>> -
>>
>> plugin type
>> -
>>
>> plugin name
>> -
>>
>> source table
>> -
>>
>> error message
>> -
>>
>> exception class
>> -
>>
>> original data
>> -
>>
>> occurrence time
>>
>> This allows jobs to bypass selected dirty or error records instead of
>> always failing fast, while keeping the default behavior unchanged when the
>> feature is not enabled.
>>
>> This is useful for long-running synchronization jobs where users need
>> better fault tolerance and later inspection of bad data.
>>
>> 16.
>>
>> Improve cloud-native deployment experience
>>
>> zhangshenghang also suggested improving the cloud-native deployment
>> experience for SeaTunnel 3.0.
>>
>> The proposal mentions Kubernetes and Helm support as important areas,
>> including better guidance and behavior around:
>>
>> -
>>
>> deployment
>> -
>>
>> restart
>> -
>>
>> persistent checkpoint storage
>> -
>>
>> configuration management
>> -
>>
>> secret management
>> -
>>
>> rolling upgrade
>> -
>>
>> resource isolation
>>
>>
>> 17.
>>
>> Clean up deprecated behavior carefully for 3.0
>>
>> zhangshenghang also suggested that SeaTunnel 3.0 is a good opportunity to
>> clean up deprecated options and old APIs.
>>
>> At the same time, any incompatible change should be handled carefully,
>> with clear migration documentation, examples, and release notes.
>>
>> Possible discussion groups
>>
>> Based on the above proposals, I think the roadmap discussion can be
>> grouped into the following areas:
>>
>> 1.
>>
>> Zeta engine observability
>>
>>
>> -
>>
>> STIP-21 / StainTrace
>> -
>>
>> Prometheus metrics for IMap state stores
>> -
>>
>> Real-time task metrics
>> -
>>
>> Queue and backpressure visibility
>> -
>>
>> REST API and Web UI exposure
>>
>>
>> 2.
>>
>> Zeta state management and storage evolution
>>
>>
>> -
>>
>> StateStore abstraction
>> -
>>
>> IMap external storage compaction
>> -
>>
>> IMap state metrics
>> -
>>
>> Future storage backend evolution
>>
>>
>> 3.
>>
>> Runtime and connector semantics
>>
>>
>> -
>>
>> Engine-level timer flush for Sink connectors
>> -
>>
>> JDBC adoption of timer flush
>> -
>>
>> Checkpoint-gated file source lifecycle management
>> -
>>
>> Table-level fault isolation
>> -
>>
>> Row-level error bypass and dead-letter queue
>>
>>
>> 4.
>>
>> AI for Data Engineering
>>
>>
>> -
>>
>> AI-Native CLI
>> -
>>
>> Natural language pipeline generation
>> -
>>
>> Connector metadata reflection
>> -
>>
>> Auto-fix loop
>> -
>>
>> Security and sensitive value redaction
>> -
>>
>> Future AI-assisted schema mapping, diagnosis, optimization, and
>> autonomous data management
>>
>>
>> 5.
>>
>> Edge collection
>>
>>
>> -
>>
>> EdgeSocket ingress
>> -
>>
>> SeaTunnel Edge Agent
>> -
>>
>> Edge host to Zeta pipeline workflow
>> -
>>
>> Authentication, local durability, and transport interoperability
>>
>>
>> 6.
>>
>> Performance and internal data model
>>
>>
>> -
>>
>> Apache Arrow as an optional internal data model
>> -
>>
>> Columnar I/O
>> -
>>
>> Vectorized execution
>> -
>>
>> Lower JVM overhead
>>
>>
>> 7.
>>
>> Build, release, and project structure
>>
>>
>> -
>>
>> Independent seatunnel-shade project
>> -
>>
>> Maven Central distribution of shade jars
>> -
>>
>> Reduced build and release time
>>
>>
>> 8.
>>
>> Cloud-native and compatibility
>>
>>
>> -
>>
>> Kubernetes and Helm deployment experience
>> -
>>
>> Persistent checkpoint storage
>> -
>>
>> Secret management
>> -
>>
>> Rolling upgrade
>> -
>>
>> Resource isolation
>> -
>>
>> Deprecated behavior cleanup
>> -
>>
>> Migration documentation and release notes
>>
>> Next steps
>>
>> I suggest we continue to review these proposals and decide:
>>
>> 1.
>>
>> Which items should be included in the SeaTunnel 3.0 roadmap.
>> 2.
>>
>> Which items should be marked as experimental.
>> 3.
>>
>> Which items require an STIP or further design discussion.
>> 4.
>>
>> Which items can be merged independently before the 3.0 release.
>> 5.
>>
>> Which items need migration documentation, compatibility notes, or
>> release notes.
>>
>> Again, this is only a summary of the current discussion thread. Please
>> help correct or supplement anything I missed.
>>
>>
>>
>> Best Regards,
>> David
>> Linkedin: https://www.linkedin.com/in/davidzollo
>>
>>
>>
>> On Thu, May 14, 2026 at 2:59 PM zhangshenghang <sh...@apache.org>
>> wrote:
>>
>>> Hi community,
>>>
>>> For the SeaTunnel 3.0 roadmap
>>>
>>> I have already submitted two related PRs, and besides those, I would like
>>> to propose the following directions for 3.0:
>>>
>>> 1. Real-time observability for Zeta engine tasks
>>>
>>> This work adds the ability to monitor the runtime busyness level of Zeta
>>> engine tasks. It introduces real-time task metrics for Source, Transform,
>>> and Sink execution, including queue/backpressure-related information, and
>>> exposes these metrics through REST API and Web UI. This can help users
>>> understand whether a job is blocked by source reading, transform
>>> processing, sink writing, or intermediate queues, which is very important
>>> for production troubleshooting and performance tuning.
>>>
>>> 2. Error data bypass and dead-letter queue capability for Transform and
>>> Sink
>>>
>>> This work introduces row-level error handling for Transform and Sink
>>> stages. When configured, SeaTunnel can collect failed records with useful
>>> context such as error stage, plugin type, plugin name, source table,
>>> error
>>> message, exception class, original data, and occurrence time. This allows
>>> jobs to bypass selected dirty/error records instead of always failing
>>> fast,
>>> while keeping the default behavior unchanged when the feature is not
>>> enabled. This is useful for long-running synchronization jobs where users
>>> need better fault tolerance and later inspection of bad data.
>>>
>>> 3. Improve cloud-native deployment experience
>>>
>>> Kubernetes and Helm support are important for new users. We should
>>> improve
>>> guidance and behavior around deployment, restart, persistent checkpoint
>>> storage, configuration management, secret management, rolling upgrade,
>>> and
>>> resource isolation in cloud-native environments.
>>>
>>> 4. Clean up deprecated behavior carefully for 3.0
>>>
>>> Since 3.0 is a major version, it is a good opportunity to clean up
>>> deprecated options and old APIs, but we should do this carefully. Any
>>> incompatible change should have clear migration documentation, examples,
>>> and release notes.
>>>
>>>
>>> Regards,
>>> Jast (Shenghang)
>>>
>>>
>>> JeremyXin <73...@qq.com> 于2026年5月14日周四 14:27写道：
>>>
>>> > Hi everyone, I’d like to propose adding this PR to the SeaTunnel 3.0
>>> > roadmap.
>>> >
>>> >
>>> > - PR#10860
>>> > - link: https://github.com/apache/seatunnel/pull/10860
>>> >
>>> >
>>> > This PR improves Zeta Engine observability by exposing Prometheus
>>> metrics
>>> > for core engine state stores backed by Hazelcast IMap.
>>> >
>>> >
>>> > It covers important internal stores such as running job information,
>>> job
>>> > states, checkpoints, slot ownership and provides visibility into their
>>> > local entry counts, backup entry counts, and heap cost.
>>> >
>>> >
>>> > Including this in SeaTunnel 3.0 would make the engine easier to monitor
>>> > and operate in production. It helps users detect state growth, data
>>> skew,
>>> > and memory pressure earlier, which improves troubleshooting efficiency,
>>> > cluster capacity planning, and overall operational stability.
>>> >
>>> >
>>> > Best regards,
>>> > JeremyXin
>>> >
>>> >
>>> > 原始邮件
>>> >
>>> >
>>> > 发件人：zhiwei niu <zniu70696@gmail.com>
>>> > 发件时间：2026年5月12日 10:27
>>> > 收件人：dev <dev@seatunnel.apache.org>
>>> > 主题：Re: [DISCUSS] Apache SeaTunnel 3.0 Roadmap
>>> >
>>> >
>>> >
>>> > Hi everyone,
>>> >
>>> >
>>> >
>>> As part of Roadmap 3.0 progress tracking, the STIP-24 PR links have been
>>> >
>>> >
>>> updated to keep references consistent and easier to follow.
>>> >
>>> > Please use the updated PR links below:
>>> >
>>> >
>>> >
>>> [Feature][Zeta] STIP-24 Phase 1: Support lightweight EdgeSocket ingress
>>> > for edge collector MVP
>>> > - PR #10871(
>>> > https://github.com/apache/seatunnel/pull/10871)
>>> > —> PR#10878(
>>> > https://github.com/apache/seatunnel/pull/10878)
>>> >
>>> >
>>> >
>>> [Feature][Zeta] STIP-24 Phase 2: Introduce standalone SeaTunnel Edge Agent
>>> > MVP and dist integration
>>> > - PR #10876(
>>> > https://github.com/apache/seatunnel/pull/10876)
>>> > — PR#10879(
>>> > https://github.com/apache/seatunnel/pull/10879)
>>> >
>>> >
>>> > zhiwei niu <zniu70696@gmail.com
>>> > > 于2026年5月12日周二 01:31写道：
>>> >
>>> > > Hi everyone,
>>> > >
>>> >
>>> >
>>> > In addition to STIP-23, I’d like to propose the STIP-24 edge-collection
>>> > > work for SeaTunnel 3.0.
>>> > >
>>> >
>>> >
>>> > - PR #10871 — [Feature][Zeta] STIP-24 Phase 1: Support lightweight
>>> >
>>> >
>>> > EdgeSocket ingress for edge collector MVP
>>> > > https://github.com/apache/seatunnel/pull/10871
>>> > > What it adds:
>>> >
>>> >
>>> > Introduces the engine-side ingress foundation for edge collection via
>>> >
>>> >
>>> > EdgeSocket, including authenticated ingress, source-side protocol handling,
>>> >
>>> >
>>> > and task-group address discovery support in Zeta.
>>> > > Why 3.0:
>>> >
>>> >
>>> > This is the foundational engine capability for remote edge data collection
>>> >
>>> >
>>> > scenarios. It is backward-compatible and establishes the protocol/runtime
>>> >
>>> > path required for the edge-side agent.
>>> > >
>>> >
>>> >
>>> > - PR #10876 — [Feature][Zeta] STIP-24 Phase 2: Introduce standalone
>>> >
>>> >
>>> > SeaTunnel Edge Agent MVP and dist integration
>>> > > https://github.com/apache/seatunnel/pull/10876
>>> > > What it adds:
>>> >
>>> >
>>> > Introduces a standalone SeaTunnel Edge Agent runtime
>>> >
>>> >
>>> > (core/transport/connector), local file/log/event inputs, SQLite WAL
>>> >
>>> >
>>> > durability, EdgeSocket transport interop, and distribution integration.
>>> > > Why 3.0:
>>> >
>>> >
>>> > This delivers the first practical end-to-end edge collection workflow
>>> >
>>> >
>>> > (edge host -> Zeta pipeline), which significantly expands SeaTunnel’s
>>> >
>>> >
>>> > deployment topology and real-world ingestion coverage.
>>> > >
>>> >
>>> >
>>> > PR #10876 depends on #10871; both are under active review.
>>> > >
>>> > > Best regards,
>>> > > zhiwei.niu
>>> > >
>>> > > xuepeng wang <wxuepeng5@gmail.com
>>> > > 于2026年5月8日周五 14:03写道：
>>> > >
>>> > >> Hi team,
>>> > >>
>>> > >>
>>> > >>
>>> >
>>> >
>>> >> I'd like to add STIP-27 [1] to the SeaTunnel 3.0 Roadmap discussion.
>>> > >>
>>> >
>>> >
>>> >> This proposal introduces Apache Arrow as an optional internal data model
>>> >
>>> >
>>> >> (not replacing the existing row model). It aims to improve performance for
>>> >
>>> >
>>> >> columnar scenarios (Parquet/ORC, ClickHouse/Doris) and batch transforms
>>> >
>>> >
>>> >> (projection/filter/cast), while keeping the current architecture stable.
>>> > >>
>>> >
>>> >
>>> >> Key benefits: better columnar I/O performance, vectorized execution,
>>> > >> lower JVM overhead.
>>> > >>
>>> > >> [1]
>>> > https://github.com/apache/seatunnel/issues/10806
>>> > >> <https://github.com/apache/seatunnel/issues/10806>>>;
>>> > >> Thoughts?
>>> > >>
>>> > >> Regards,
>>> > >> xuepeng(github id:chl-wxp)
>>> > >>
>>> > >> David Zollo <davidzollo365@gmail.com
>>> > > 于2026年4月30日周四 21:17写道：
>>> > >>
>>> > >> > Hi everyone,
>>> > >> >
>>> >
>>> >
>>> >> > First of all, thank you all for the fantastic proposals so far — the
>>> >
>>> >
>>> >> > quality and breadth of ideas in this thread have been genuinely
>>> >
>>> >
>>> >> > exciting to read. From StainTrace to the AI-Native CLI, from
>>> >
>>> >
>>> >> > StateStore abstraction to table-level fault isolation, it's clear that
>>> >
>>> >
>>> >> > this community has a bold and thoughtful vision for what SeaTunnel 3.0
>>> > >> > can become.
>>> > >> >
>>> >
>>> >
>>> >> > Given that the Golden Week holiday is just around the corner, and many
>>> >
>>> >
>>> >> > of our contributors are based in China or taking time off during this
>>> >
>>> >
>>> >> > period, I'd like to extend the discussion window to May 15th. This
>>> >
>>> >
>>> >> > gives everyone enough breathing room to enjoy the holiday, come back
>>> >
>>> >
>>> >> > refreshed, and still have time to share their thoughts without feeling
>>> > >> > rushed.
>>> > >> >
>>> >
>>> >> > Happy Labor Day to everyone!
>>> > >> >
>>> >
>>> >
>>> >> > We highly welcome everyone's suggestions for version 3.0! Feel free to
>>> >
>>> >
>>> >> > leave a message if you have any ideas.
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> > Best Regards,
>>> > >> > David
>>> > >> > Linkedin:
>>> > https://www.linkedin.com/in/davidzollo
>>> > >> >
>>> > <https://www.linkedin.com/in/davidzollo>> >>;
>>> >
>>> >
>>> >> > On Wed, Apr 29, 2026 at 8:50 PM sean zhang <
>>> > seanzhang92@gmail.com>
>>> > >> wrote:
>>> > >> > >
>>> > >> > > Hi SeaTunnel community,
>>> > >> > >
>>> >
>>> >
>>> >> > > I'd like to propose including the AI-Native CLI capability in the
>>> >
>>> >
>>> >> > > SeaTunnel 3.0 release, and share a broader vision for the "AI for Data
>>> >
>>> >
>>> >> > > Engineering" direction. I believe this aligns well with the 3.0
>>> > >> milestone
>>> >
>>> >
>>> >> > > as a significant user-experience leap.
>>> > >> > > This PR
>>> > https://github.com/apache/seatunnel/pull/10789 introduces
>>> >
>>> >
>>> >> > > `seatunnel-cli`, an AI-powered command-line tool that
>>> >
>>> >
>>> >> > > generates production-ready pipeline configurations from natural
>>> > >> language
>>> > <
>>> https://github.com/apache/seatunnel/pull/10789 introduces>> > > seatunnel-cli, an AI-powered command-line tool that>> > > generates production-ready pipeline configurations from natural>> language
>>> >
>>> > :
>>> > >> > >
>>> >
>>> >
>>> >> > > $ seatunnel "Sync the users table from MySQL to S3 in parquet
>>> > >> format"
>>> >
>>> >
>>> >> > > ✅ Generated config: mysql_users_to_s3.conf
>>> > >> > >
>>> >
>>> >
>>> >> > > Key capabilities in the current implementation:
>>> > >> > >
>>> >
>>> >
>>> >> > > - Multi-LLM provider support (Anthropic / OpenAI / AWS Bedrock)
>>> >
>>> >
>>> >> > > with auto-detection
>>> >
>>> >
>>> >> > > - Runtime connector metadata extraction via Java OptionRule
>>> > >> reflection,
>>> >
>>> >
>>> >> > > ensuring 100% accurate parameter knowledge for all 100+ connectors
>>> >
>>> >
>>> >> > > - Source/Sink type-aware validation (Jdbc source options ≠ Jdbc sink
>>> > >> > > options)
>>> >
>>> >
>>> >> > > - Auto-fix loop: generated configs are validated and self-corrected
>>> >
>>> >
>>> >> > > (up to 3 iterations)
>>> >
>>> >
>>> >> > > - Security hardening: sensitive values (passwords, tokens, keys) are
>>> >
>>> >
>>> >> > > automatically redacted to ${ENV_VAR} placeholders before
>>> > >> persistence
>>> >
>>> >
>>> >> > > and LLM transmission
>>> >
>>> >
>>> >> > > - Interactive mode with session memory and one-shot mode for CI/CD
>>> > >> > >
>>> >
>>> >
>>> >> > > Future Roadmap — AI for Data Engineering
>>> > >> > >
>>> >
>>> >
>>> >> > > Looking beyond config generation, I see a multi-level roadmap where
>>> >
>>> >
>>> >> > > AI becomes deeply integrated into SeaTunnel's core value proposition:
>>> > >> > >
>>> >
>>> >
>>> >> > > ### Level 2: AI Understands Data (Next phase)
>>> > >> > >
>>> >
>>> >
>>> >> > > - Intelligent Schema Mapping: AI reads source schema, infers target
>>> >
>>> >
>>> >> > > schema, and auto-generates type conversions and DDL
>>> >
>>> >
>>> >> > > - Natural Language Transforms: "mask phone numbers, merge first and
>>> >
>>> >
>>> >> > > last name, filter last 7 days" → SQL Transform auto-generation
>>> >
>>> >
>>> >> > > - Data Profiling: AI samples source data, detects anomalies (NULLs,
>>> >
>>> >
>>> >> > > outliers, format violations), and suggests data quality transforms
>>> > >> > >
>>> >
>>> >
>>> >> > > ### Level 3: AI Operates Pipelines
>>> > >> > >
>>> >
>>> >
>>> >> > > - Intelligent Failure Diagnosis: AI analyzes error logs, pinpoints
>>> >
>>> >
>>> >> > > root cause (connection refused → MySQL down vs firewall), suggests
>>> > >> > fix
>>> >
>>> >
>>> >> > > - Self-Healing Pipelines: auto-detect DDL changes in CDC sources,
>>> >
>>> >
>>> >> > > auto-evolve sink schema, zero-downtime recovery
>>> >
>>> >
>>> >> > > - Anomaly Detection: monitor data flow volume/latency, alert when
>>> >
>>> >
>>> >> > > throughput drops 99% with correlated root cause
>>> > >> > >
>>> >
>>> >
>>> >> > > ### Level 4: AI Optimizes Everything
>>> > >> > >
>>> >
>>> >
>>> >> > > - Performance Tuning: AI analyzes bottlenecks (batch_size too small,
>>> >
>>> >
>>> >> > > wrong file format, insufficient parallelism), suggests
>>> > >> optimization
>>> >
>>> >
>>> >> > > - Cost-Aware Planning: given N tables to sync, AI proposes pipeline
>>> >
>>> >
>>> >> > > topology that balances cost vs latency vs resource usage
>>> >
>>> >
>>> >> > > - Intelligent Scheduling: learn from historical patterns to optimize
>>> >
>>> >
>>> >> > > sync frequency per table
>>> > >> > >
>>> >
>>> >
>>> >> > > ### Level 5: Autonomous Data Platform (Long-term vision)
>>> > >> > >
>>> >
>>> >
>>> >> > > - Conversational data infrastructure management: "onboard the
>>> > >> payment
>>> >
>>> >
>>> >> > > system to the data lake, ensure GDPR compliance" → AI
>>> > >> auto-discovers
>>> >
>>> >
>>> >> > > sources, identifies PII, generates pipelines with masking
>>> > >> transforms,
>>> >
>>> >
>>> >> > > deploys with monitoring and alerting
>>> > >> > >
>>> > >> > >
>>> > >> > > Doyeon Kim <
>>> > ekagnlek2@gmail.com> 于2026年4月29日周三 18:17写道：
>>> > >> > >
>>> > >> > > > Hi everyone,
>>> > >> > > >
>>> >
>>> >
>>> >> > > > I’d like to propose the following Zeta engine improvements for
>>> > >> > SeaTunnel
>>> > >> > > > 3.0.
>>> > >> > > >
>>> >
>>> >
>>> >> > > > 1. StateStore abstraction
>>> > >> > > > PR:
>>> > https://github.com/apache/seatunnel/pull/10812
>>> > >> > > >
>>> > <
>>> https://github.com/apache/seatunnel/pull/10812>> > > >
>>> >
>>> > ;
>>> >
>>> >
>>> >> > > > This introduces an initial StateStore abstraction layer to reduce
>>> > >> > direct
>>> >
>>> >
>>> >> > > > Hazelcast IMap dependencies in the engine. It makes different state
>>> >
>>> >
>>> >> > > > responsibilities more explicit and prepares the engine for future
>>> > >> > backend
>>> > >> > > > evolution.
>>> > >> > > >
>>> >
>>> >
>>> >> > > > I think this is relevant for 3.0 because it is a foundational step
>>> > >> > toward
>>> >
>>> >
>>> >> > > > decoupling the engine from Hazelcast-specific APIs and enabling
>>> > >> future
>>> >
>>> >
>>> >> > > > improvements around state storage, node communication, and
>>> > >> coordination
>>> > >> > > > layers.
>>> > >> > > >
>>> >
>>> >
>>> >> > > > 2. IMap external storage compaction
>>> > >> > > > PR:
>>> > https://github.com/apache/seatunnel/pull/10399
>>> > >> > > >
>>> > <
>>> https://github.com/apache/seatunnel/pull/10399>> > > >
>>> >
>>> > ;
>>> >
>>> >
>>> >> > > > This introduces LSM-style write and compaction logic for IMap
>>> > >> external
>>> >
>>> >
>>> >> > > > storage. It helps control storage growth by flushing sorted
>>> > >> immutable
>>> > >> > files
>>> >
>>> >
>>> >> > > > and compacting persisted files while keeping only the latest record
>>> > >> per
>>> > >> > > > key.
>>> > >> > > >
>>> >
>>> >
>>> >> > > > I think this should be considered for 3.0 because long-running Zeta
>>> > >> > jobs
>>> >
>>> >
>>> >> > > > may accumulate duplicated records and tombstones in external storage
>>> > >> > over
>>> >
>>> >
>>> >> > > > time. Compaction can improve stability and reduce the risk of
>>> > >> unbounded
>>> > >> > > > storage growth.
>>> > >> > > >
>>> > >> > > > Best Regards,
>>> > >> > > > Doyeon
>>> >
>>> >> > > > (Github ID: dybyte)
>>> > >> > > >
>>> >
>>> >
>>> >> > > > On Wed, Apr 29, 2026 at 7:13 PM David Zollo <
>>> > >> davidzollo365@gmail.com>
>>> > >> > > > wrote:
>>> > >> > > >
>>> > >> > > > > Hi guys,
>>> > >> > > > >
>>> >
>>> >
>>> >> > > > > I'd like to propose including PR #10491 in the SeaTunnel 3.0
>>> > >> roadmap,
>>> >
>>> >
>>> >> > > > > here are the details with additional packaging time analysis:
>>> > >> > > > >
>>> >
>>> >> > > > > ### PR link
>>> > >> > > > >
>>> > https://github.com/apache/seatunnel/pull/10491
>>> > >> > > > >
>>> > <
>>> https://github.com/apache/seatunnel/pull/10491>> > > > >
>>> >
>>> > ;
>>> >
>>> >
>>> >> > > > > StainTrace (PR #10491) is the concrete implementation of STIP-21,
>>> >
>>> >
>>> >> > > > > delivering end-to-end data lineage tracing and performance
>>> > >> analysis
>>> >
>>> >
>>> >> > > > > capabilities for the Apache SeaTunnel Zeta engine. It injects
>>> >
>>> >
>>> >> > > > > lightweight timestamp sampling points at 6 key pipeline stages —
>>> >
>>> >
>>> >> > > > > SOURCE_EMIT → QUEUE_IN → QUEUE_OUT → TRANSFORM_IN → TRANSFORM_OUT
>>> > >> →
>>> >
>>> >
>>> >> > > > > SINK_WRITE_DONE — tracking each sampled row's complete journey
>>> > >> from
>>> >
>>> >
>>> >> > > > > source emission to sink write completion, enabling precise
>>> > >> end-to-end
>>> >
>>> >
>>> >> > > > > latency measurement and bottleneck identification.
>>> > >> > > > >
>>> >
>>> >
>>> >> > > > > The system is designed with production safety as a first-class
>>> >
>>> >
>>> >> > > > > concern: it is disabled by default with very low overhead, and
>>> > >> when
>>> >
>>> >
>>> >> > > > > enabled it operates at a configurable sampling rate with a strict
>>> >
>>> >
>>> >> > > > > per-worker per-second event budget, achieving approximately 0.1~1%
>>> >
>>> >
>>> >> > > > > overhead at 1% sampling. Trace data is written locally as JSONL
>>> > >> files
>>> >
>>> >
>>> >> > > > > in OpenTelemetry OTLP JSON format with no external dependencies.
>>> > >> An
>>> >
>>> >
>>> >> > > > > accompanying offline analyzer reads these files to compute P95/P99
>>> >
>>> >
>>> >> > > > > latency statistics, identify bottleneck stages, and generate an
>>> > >> HTML
>>> >
>>> >
>>> >> > > > > visualization report with Chinese/English language toggle support.
>>> > >> > > > >
>>> >
>>> >
>>> >> > > > > The value to SeaTunnel lies in closing a critical observability
>>> > >> gap
>>> > >> > at
>>> >
>>> >
>>> >> > > > > the engine level. Before this feature, users facing latency
>>> > >> spikes or
>>> >
>>> >
>>> >> > > > > throughput bottlenecks had no structured way to pinpoint whether
>>> > >> the
>>> >
>>> >
>>> >> > > > > problem resided in queue backpressure, transform processing, or
>>> > >> sink
>>> >
>>> >
>>> >> > > > > write time — they could only rely on logs and guesswork.
>>> > >> StainTrace
>>> >
>>> >
>>> >> > > > > makes per-stage timing transparent and actionable, which is
>>> > >> > especially
>>> >
>>> >
>>> >> > > > > impactful for large-scale, multi-table, long-chain production
>>> > >> > > > > pipelines.
>>> > >> > > > >
>>> >
>>> >
>>> >> > > > > The case for including it in the 3.0 roadmap is equally strong. As
>>> > >> > the
>>> >
>>> >
>>> >> > > > > implementation of STIP-21, StainTrace is a framework-level
>>> >
>>> >
>>> >> > > > > observability capability — connector-agnostic and immediately
>>> >
>>> >
>>> >> > > > > effective across all jobs once merged. One of SeaTunnel 3.0's
>>> > >> central
>>> >
>>> >
>>> >> > > > > goals is to become a more mature, production-ready data
>>> > >> integration
>>> >
>>> >
>>> >> > > > > engine, and observability is an indispensable foundation for that
>>> >
>>> >
>>> >> > > > > goal. Establishing it at a major version boundary like 3.0
>>> > >> delivers
>>> >
>>> >
>>> >> > > > > far greater long-term value to users.
>>> > >> > > > >
>>> > >> > > > >
>>> > >> > > > >
>>> > >> > > > >
>>> > >> > > > >
>>> > >> > > > >
>>> >
>>> >> > > > > Best Regards,
>>> > >> > > > > David
>>> > >> > > > > Linkedin:
>>> > https://www.linkedin.com/in/davidzollo
>>> > >> > > > >
>>> > <
>>> https://www.linkedin.com/in/davidzollo>> > > > >
>>> >
>>> > ;
>>> >
>>> >
>>> >> > > > > On Wed, Apr 29, 2026 at 2:48 PM yi zeng <
>>> > yzeng1618@gmail.com>
>>> > >> wrote:
>>> > >> > > > > >
>>> >
>>> >
>>> >> > > > > > Hi everyone,
>>> > >> > > > > >
>>> >
>>> >
>>> >> > > > > > I’d like to propose including the following two PRs in the
>>> > >> > *SeaTunnel
>>> > >> > > > 3.0
>>> >
>>> >
>>> >> > > > > > roadmap*. Both focus on strengthening *correctness, reliability,
>>> > >> > and
>>> >
>>> >
>>> >> > > > > > operational safety* for core multi-table and continuous
>>> > >> ingestion
>>> >
>>> >
>>> >> > > > > > scenarios, which I believe are key pillars for 3.0.
>>> >
>>> >
>>> >> > > > > > ------------------------------
>>> >
>>> >
>>> >> > > > > > PR #10600 — [Feature][Core] Support table-level fault isolation
>>> > >> for
>>> >
>>> >
>>> >> > > > > > multi-table synchronization
>>> > >> > > > > >
>>> > >> > > > > >
>>> > https://github.com/apache/seatunnel/pull/10600
>>> > >> > > > > >
>>> > <
>>> https://github.com/apache/seatunnel/pull/10600>> > > > > >
>>> >
>>> > ;
>>> >
>>> >
>>> >> > > > > > *What it adds / solves*
>>> > >> > > > > >
>>> >
>>> >
>>> >> > > > > > This PR introduces *table-level fault isolation* for multi-table
>>> >
>>> >
>>> >> > > > > > synchronization by adding a new failure policy:
>>> > >> > > > > > code
>>> > >> > > > > > 复制
>>> > >> > > > > >
>>> >
>>> >> > > > > > env {
>>> >
>>> >
>>> >> > > > > > multi_table {
>>> >
>>> >
>>> >> > > > > > failure_policy = "CONTINUE_OTHER_TABLES"
>>> >
>>> >
>>> >> > > > > > }
>>> > >> > > > > > }
>>> > >> > > > > >
>>> >
>>> >
>>> >> > > > > > With this enabled:
>>> > >> > > > > >
>>> >
>>> >
>>> >> > > > > > - Failures are isolated at *table granularity*, instead of
>>> > >> > failing
>>> > >> > > > the
>>> >
>>> >
>>> >> > > > > > entire job.
>>> >
>>> >
>>> >> > > > > > - Healthy tables continue processing even if one table fails.
>>> >
>>> >
>>> >> > > > > > - Failed tables are collected and reported with clear
>>> > >> metadata
>>> > >> > > > (table
>>> >
>>> >
>>> >> > > > > > path, failure phase, plugin, error).
>>> >
>>> >
>>> >> > > > > > - Batch jobs report failed tables in final results; streaming
>>> > >> > jobs
>>> >
>>> >
>>> >> > > > > > continue running with healthy tables.
>>> > >> > > > > >
>>> >
>>> >
>>> >> > > > > > This addresses a long-standing pain point where a single
>>> > >> > problematic
>>> > >> > > > > table
>>> >
>>> >
>>> >> > > > > > can block progress for all others.
>>> > >> > > > > >
>>> >
>>> >
>>> >> > > > > > *Why it should be included in 3.0*
>>> > >> > > > > >
>>> >
>>> >
>>> >> > > > > > - *Improved fault tolerance model*: 3.0 aims to be more
>>> > >> > resilient
>>> > >> > > > and
>>> >
>>> >
>>> >> > > > > > production-ready; table-level isolation is a foundational
>>> > >> > capability
>>> > >> > > > > for
>>> >
>>> >
>>> >> > > > > > large-scale, multi-table pipelines.
>>> >
>>> >
>>> >> > > > > > - *Backward-compatible by default*: The default behavior
>>> > >> remains
>>> >
>>> >
>>> >> > > > > > FAIL_FAST, so existing jobs are unaffected unless users
>>> > >> > explicitly
>>> > >> > > > opt
>>> >
>>> >
>>> >> > > > > > in.
>>> >
>>> >
>>> >> > > > > > - *Framework-level improvement*: This is not
>>> > >> > connector-specific; it
>>> >
>>> >
>>> >> > > > > > enhances the core execution and failure-handling semantics of
>>> > >> > > > > SeaTunnel.
>>> >
>>> >
>>> >> > > > > > - *Mature design & coverage*: Includes core logic changes,
>>> > >> > runtime
>>> >
>>> >
>>> >> > > > > > handling, reporting, and E2E tests, making it suitable for a
>>> > >> > major
>>> > >> > > > > version
>>> >
>>> >
>>> >> > > > > > milestone.
>>> > >> > > > > >
>>> >
>>> >
>>> >> > > > > > ------------------------------
>>> >
>>> >
>>> >> > > > > > PR #10563 — [Feature][Connector-V2] Add checkpoint-gated
>>> > >> > > > post_sync_action
>>> >
>>> >
>>> >> > > > > > and retention for file source continuous discovery
>>> > >> > > > > >
>>> > >> > > > > >
>>> > https://github.com/apache/seatunnel/pull/10563
>>> > >> > > > > >
>>> > <
>>> https://github.com/apache/seatunnel/pull/10563>> > > > > >
>>> >
>>> > ;
>>> >
>>> >
>>> >> > > > > > *What it adds / solves*
>>> > >> > > > > >
>>> >
>>> >
>>> >> > > > > > This PR adds *checkpoint-gated source-side lifecycle management*
>>> > >> > for
>>> > >> > > > file
>>> >
>>> >
>>> >> > > > > > source continuous discovery, including:
>>> > >> > > > > >
>>> >
>>> >
>>> >> > > > > > - post_sync_action = delete
>>> >
>>> >
>>> >> > > > > > Delete source files only *after a checkpoint has completed*
>>> >
>>> >
>>> >> > > > > > - post_sync_action = backup
>>> >
>>> >
>>> >> > > > > > Move source files to a backup_path after checkpoint
>>> > >> completion
>>> >
>>> >
>>> >> > > > > > - Optional backup retention via retention_max_age
>>> > >> > > > > >
>>> >
>>> >
>>> >> > > > > > All operations are *checkpoint-aware*, ensuring source files are
>>> > >> > never
>>> >
>>> >
>>> >> > > > > > deleted or moved before the data is durably processed.
>>> > >> > > > > >
>>> >
>>> >
>>> >> > > > > > *Why it should be included in 3.0*
>>> > >> > > > > >
>>> >
>>> >
>>> >> > > > > > - *Correctness-first continuous ingestion*: Safe source
>>> > >> cleanup
>>> > >> > tied
>>> > >> > > > > to
>>> >
>>> >
>>> >> > > > > > checkpoint completion is essential for exactly-once and
>>> > >> > recoverable
>>> > >> > > > > file
>>> >
>>> >
>>> >> > > > > > ingestion, which aligns strongly with 3.0 quality goals.
>>> >
>>> >
>>> >> > > > > > - *Production-grade operational semantics*: Without
>>> > >> > checkpoint-gated
>>> >
>>> >
>>> >> > > > > > cleanup, continuous file ingestion is risky in real
>>> > >> production
>>> > >> > > > > environments.
>>> >
>>> >
>>> >> > > > > > - *Extends Connector-V2 maturity*: This PR deepens the
>>> > >> > operational
>>> >
>>> >
>>> >> > > > > > completeness of Connector-V2, especially for file-based
>>> > >> sources.
>>> >
>>> >
>>> >> > > > > > - *Well-tested and documented*: Includes unit tests, E2E
>>> > >> tests
>>> > >> > > > (Local
>>> > >> > > > > +
>>> >
>>> >
>>> >> > > > > > HDFS), and documentation updates (EN + ZH).
>>> > >> > > > > >
>>> >
>>> >
>>> >> > > > > > ------------------------------
>>> > >> > > > > >
>>> >
>>> >
>>> >> > > > > > Both PRs are *opt-in*, backward-compatible, and focused on
>>> > >> making
>>> > >> > > > > SeaTunnel
>>> >
>>> >
>>> >> > > > > > more robust, predictable, and safe in real-world deployments. I
>>> > >> > believe
>>> >
>>> >
>>> >> > > > > > they fit naturally into the SeaTunnel 3.0 scope, especially
>>> > >> given
>>> > >> > the
>>> >
>>> >
>>> >> > > > > > emphasis on stability, correctness, and large-scale usage.
>>> > >> > > > > >
>>> >
>>> >
>>> >> > > > > > Happy to discuss details or answer questions.
>>> > >> > > > > >
>>> >
>>> >
>>> >> > > > > > Zhang Donghao <
>>> > zhangdonghao@apache.org> 于2026年4月29日周三 14:38写道：
>>> > >> > > > > >
>>> >
>>> >
>>> >> > > > > > > Hi everyone,
>>> > >> > > > > > >
>>> >
>>> >
>>> >> > > > > > > I'd like to propose including PR #9993 in the SeaTunnel 3.0
>>> > >> > roadmap,
>>> > >> > > > > here
>>> >
>>> >
>>> >> > > > > > > are the details with additional packaging time analysis:
>>> > >> > > > > > >
>>> >
>>> >
>>> >> > > > > > > ### PR link
>>> >
>>> >> > > > > > >
>>> > https://github.com/apache/seatunnel/pull/9993
>>> > >> > > > > > >
>>> > <
>>> https://github.com/apache/seatunnel/pull/9993>> > > > > > >
>>> >
>>> > ;
>>> >
>>> >
>>> >> > > > > > > ### Short description
>>> >
>>> >
>>> >> > > > > > > This PR extracts the shade modules into an independent
>>> > >> > > > seatunnel-shade
>>> >
>>> >
>>> >> > > > > > > project, enables uploading shade-jars to the central
>>> > >> repository,
>>> > >> > and
>>> >
>>> >
>>> >> > > > > > > optimizes the overall packaging efficiency of SeaTunnel.
>>> > >> > > > > > >
>>> >
>>> >
>>> >> > > > > > > ### Packaging time analysis (current state before PR
>>> > >> > optimization)
>>> >
>>> >
>>> >> > > > > > > The current build time breakdown of individual shade modules
>>> > >> is
>>> > >> > as
>>> >
>>> >
>>> >> > > > > > > follows, highlighting the inefficiency of the integrated build
>>> > >> > > > > approach:
>>> >
>>> >
>>> >> > > > > > > ```
>>> >
>>> >
>>> >> > > > > > > [INFO] SeaTunnel : Shade : 3.0.0 ..........................
>>> > >> > SUCCESS [
>>> >
>>> >
>>> >> > > > > > > 12.661 s]
>>> >
>>> >
>>> >> > > > > > > [INFO] SeaTunnel : Shade : Hazelcast 5.1-3.0.0 ............
>>> > >> > SUCCESS [
>>> >
>>> >
>>> >> > > > > > > 8.192 s]
>>> >
>>> >
>>> >> > > > > > > [INFO] SeaTunnel : Shade : Hadoop3 Uber 3.1.4-3.0.0 .......
>>> > >> > SUCCESS
>>> > >> > > > > [02:29
>>> >
>>> >
>>> >> > > > > > > min]
>>> >
>>> >
>>> >> > > > > > > [INFO] SeaTunnel : Shade : Jackson 2.13.3-3.0.0 ...........
>>> > >> > SUCCESS [
>>> >
>>> >
>>> >> > > > > > > 3.289 s]
>>> >
>>> >
>>> >> > > > > > > [INFO] SeaTunnel : Shade : Guava 27.0-jre-3.0.0 ...........
>>> > >> > SUCCESS [
>>> >
>>> >
>>> >> > > > > > > 2.960 s]
>>> >
>>> >
>>> >> > > > > > > [INFO] SeaTunnel : Shade : Hadoop : AWS 3.1.4-3.0.0 .......
>>> > >> > SUCCESS
>>> > >> > > > > [05:05
>>> >
>>> >
>>> >> > > > > > > min]
>>> >
>>> >
>>> >> > > > > > > [INFO] SeaTunnel : Shade : Arrow 15.0.1-3.0.0 .............
>>> > >> > SUCCESS [
>>> >
>>> >
>>> >> > > > > > > 5.880 s]
>>> >
>>> >
>>> >> > > > > > > [INFO] SeaTunnel : Shade : Hikari 4.0.3-3.0.0 .............
>>> > >> > SUCCESS [
>>> >
>>> >
>>> >> > > > > > > 1.085 s]
>>> >
>>> >
>>> >> > > > > > > [INFO] SeaTunnel : Shade : Thrift-Service 1.0.0-3.0.0 .....
>>> > >> > SUCCESS [
>>> >
>>> >
>>> >> > > > > > > 1.572 s]
>>> >
>>> >
>>> >> > > > > > > [INFO] SeaTunnel : Shade : Janino 3.0.11-3.0.0 ............
>>> > >> > SUCCESS [
>>> >
>>> >
>>> >> > > > > > > 1.235 s]
>>> >
>>> >
>>> >> > > > > > > [INFO] SeaTunnel : Shade : Scala 2.13.11-3.0.0 ............
>>> > >> > SUCCESS [
>>> >
>>> >
>>> >> > > > > > > 6.586 s]
>>> >
>>> >
>>> >> > > > > > > [INFO] SeaTunnel : Shade : Jetty 9.4.56.v20240826-3.0.0 ...
>>> > >> > SUCCESS [
>>> >
>>> >
>>> >> > > > > > > 2.421 s]
>>> >
>>> >
>>> >> > > > > > > [INFO] SeaTunnel : Shade : Commons Lang3 3.18.0-3.0.0 .....
>>> > >> > SUCCESS [
>>> >
>>> >
>>> >> > > > > > > 2.044 s]
>>> >
>>> >
>>> >> > > > > > > ```
>>> >
>>> >
>>> >> > > > > > > Key observations from the time breakdown:
>>> >
>>> >
>>> >> > > > > > > 1. **Long-tail modules dominate build time**: Two modules
>>> > >> > account for
>>> > >> > > > > the
>>> >
>>> >
>>> >> > > > > > > vast majority of shade build time — `Hadoop3 Uber` (2 minutes
>>> > >> 29
>>> > >> > > > > seconds)
>>> >
>>> >
>>> >> > > > > > > and `Hadoop : AWS` (5 minutes 5 seconds), totaling ~7.5
>>> > >> minutes,
>>> > >> > > > while
>>> >
>>> >
>>> >> > > > > > > other modules take only a few seconds each.
>>> >
>>> >
>>> >> > > > > > > 2. **Integrated build waste**: In the current structure, these
>>> >
>>> >
>>> >> > > > > > > time-consuming shade modules are built alongside core modules
>>> > >> > every
>>> > >> > > > > time
>>> >
>>> >
>>> >> > > > > > > SeaTunnel is packaged, even when there are no changes to shade
>>> >
>>> >
>>> >> > > > > > > dependencies. This results in repeated long build cycles for
>>> >
>>> >> > > > > developers and
>>> >
>>> >
>>> >> > > > > > > release workflows.
>>> >
>>> >
>>> >> > > > > > > 3. **Total shade build time**: The cumulative build time of
>>> > >> all
>>> > >> > shade
>>> >
>>> >
>>> >> > > > > > > modules is approximately 8 minutes (excluding
>>> > >> > overlapping/parallel
>>> > >> > > > > build
>>> >
>>> >
>>> >> > > > > > > optimizations), which is a significant portion of the overall
>>> > >> > > > SeaTunnel
>>> >
>>> >
>>> >> > > > > > > packaging process.
>>> > >> > > > > > >
>>> >
>>> >
>>> >> > > > > > > ### Reason for including in 3.0
>>> >
>>> >
>>> >> > > > > > > 1. **Build efficiency optimization (quantifiable gains)**:
>>> >
>>> >
>>> >> > > > > > > - By isolating shade modules into an independent project
>>> > >> and
>>> > >> > > > > uploading
>>> >
>>> >
>>> >> > > > > > > shade-jars to the central repository, we eliminate the need to
>>> > >> > > > rebuild
>>> >
>>> >
>>> >> > > > > > > these modules in every SeaTunnel packaging cycle. For daily
>>> > >> > > > > development,
>>> >
>>> >
>>> >> > > > > > > this reduces the total build time by ~8 minutes per build (for
>>> > >> > > > > developers
>>> >
>>> >
>>> >> > > > > > > not modifying shade code).
>>> >
>>> >
>>> >> > > > > > > - For the 3.0 release process, we can pre-build and publish
>>> > >> > > > > shade-jars
>>> >
>>> >
>>> >> > > > > > > to the central repository once, then reference them as
>>> > >> > dependencies —
>>> >
>>> >
>>> >> > > > > > > cutting the release packaging time by a substantial margin and
>>> > >> > > > > reducing the
>>> >
>>> >
>>> >> > > > > > > risk of build failures caused by long-running shade module
>>> > >> > builds.
>>> >
>>> >
>>> >> > > > > > > 2. **Dependency management enhancement**: Uploading shade-jars
>>> > >> > to the
>>> >
>>> >
>>> >> > > > > > > central repository standardizes dependency distribution,
>>> > >> avoiding
>>> > >> > > > > conflicts
>>> >
>>> >
>>> >> > > > > > > caused by internal shade module duplication across
>>> > >> subprojects,
>>> > >> > and
>>> >
>>> >
>>> >> > > > > > > improving the stability of the 3.0 version's dependency
>>> > >> system.
>>> >
>>> >
>>> >> > > > > > > 3. **Project structure rationalization**: Isolating shade
>>> > >> modules
>>> > >> > > > > aligns
>>> >
>>> >
>>> >> > > > > > > with the modular design concept of 3.0, making the project
>>> > >> > structure
>>> >
>>> >
>>> >> > > > > > > clearer, easier to maintain, and facilitating future
>>> > >> expansion of
>>> >
>>> >
>>> >> > > > > > > shade-related capabilities (e.g., adding new shade modules for
>>> > >> > > > > additional
>>> >
>>> >
>>> >> > > > > > > dependencies).
>>> > >> > > > > > >
>>> >
>>> >
>>> >> > > > > > > This optimization directly addresses the identified packaging
>>> > >> > > > > bottlenecks
>>> >
>>> >
>>> >> > > > > > > (long-tail shade modules) and delivers measurable time savings
>>> > >> > for
>>> > >> > > > both
>>> >
>>> >
>>> >> > > > > > > development and release workflows — a critical improvement for
>>> > >> > the
>>> > >> > > > > smooth
>>> >
>>> >
>>> >> > > > > > > delivery and maintainability of SeaTunnel 3.0.
>>> > >> > > > > > >
>>> >
>>> >
>>> >> > > > > > > Best regards,
>>> >
>>> >
>>> >> > > > > > > zhangdonghao
>>> > >> > > > > > >
>>> > >> > > > >
>>> > >> > > >
>>> > >> >
>>> > >>
>>> > >
>>>
>>