You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@seatunnel.apache.org by David Zollo <da...@gmail.com> on 2026/09/01 14:22:23 UTC
Re: [DISCUSS] Apache SeaTunnel 3.0 Roadmap
Hi guys,
Thanks for all your effort.Now we have merged all the feature PRs as
planned in the 3.0.0 Roadmap.
and there are 7 bug fix PRs left to be merged before the 3.0.0 release:
- https://github.com/apache/seatunnel/pull/11503 @nielifeng
- https://github.com/apache/seatunnel/pull/11757 @waterWang
- https://github.com/apache/seatunnel/pull/11458 @zhangshenghang
- https://github.com/apache/seatunnel/pull/11727 @abdessalems
- https://github.com/apache/seatunnel/pull/11077 @hesam-oxe
- https://github.com/apache/seatunnel/pull/11569 @DanielLeens
- https://github.com/apache/seatunnel/pull/11271 @hutiefang76
Best Regards,
David
Linkedin: https://www.linkedin.com/in/davidzollo
On Wed, Jul 22, 2026 at 10:56 PM David Zollo <da...@gmail.com>
wrote:
> Hi guys,
>
> Guys, please leave a message if you have any PR that needs to be merged in
> the 3.0.0 version.
>
> Firstly, thanks for your hard work and making great progress. Now we only
> have 3 PRs left to update and review in the 3.0.0 Roadmap list[1].
> Below is the list of remaining PRs pending merge in the Roadmap:
>
> 1. [#10563](https://github.com/apache/seatunnel/pull/10563)
> [Feature][Connector-V2] Add checkpoint-gated post_sync_action and retention
> for file source continuous discovery @yzeng1618
> 2. [#9993](https://github.com/apache/seatunnel/pull/9993)
> [Feature][shade]Refactor the seatunnel-shade module @hawk9821
> 3. [#10306](https://github.com/apache/seatunnel/pull/10306)
> [Feature][Core] Error Data Bypass Capability for Transform & Sink (Dead
> Letter Queue) @zhangshenghang
>
>
> Again, please leave a message if you have any PR need to be merged in
> 3.0.0 version.
>
>
> [1]. https://github.com/apache/seatunnel/issues/11021
>
>
>
> Best Regards,
> David
> Linkedin: https://www.linkedin.com/in/davidzollo
>
>
>
> On Wed, Jul 15, 2026 at 12:36 PM David Zollo <da...@gmail.com>
> wrote:
>
>> Hi guys,
>>
>> Thanks for your hard work and make a great progress. Now we only have
>> four PRs left to be reviewed and merged.
>> Below is the list of remaining PRs pending merge in the Roadmap:
>>
>> 1. [#10563](https://github.com/apache/seatunnel/pull/10563)
>> [Feature][Connector-V2] Add checkpoint-gated post_sync_action and retention
>> for file source continuous discovery @yzeng1618
>> 2. [#9993](https://github.com/apache/seatunnel/pull/9993)
>> [Feature][shade]Refactor the seatunnel-shade module. @hawk9821
>> 3. [#10879](https://github.com/apache/seatunnel/pull/10879)
>> [Feature][Zeta] STIP-24 Phase 2: Introduce standalone SeaTunnel Edge Agent
>> MVP and dist integration @nzw921rx
>> 4. [#10306](https://github.com/apache/seatunnel/pull/10306)
>> [Feature][Core] Error Data Bypass Capability for Transform & Sink (Dead
>> Letter Queue) @zhangshenghang
>>
>>
>> Guys, please help review the PR for the 3.0 Roadmap when you have time.
>>
>>
>> Best Regards,
>> David
>> Linkedin: https://www.linkedin.com/in/davidzollo
>>
>>
>>
>>
>> On Thu, Jul 9, 2026 at 10:56 PM zhiwei niu <zn...@gmail.com> wrote:
>>
>>> Hi community,
>>>
>>> I would like to propose including STIP-28 in the SeaTunnel 3.0 roadmap.
>>>
>>> STIP-28 enhances SeaTunnel's OptionRule framework from basic
>>> presence/equality validation to richer declarative configuration
>>> validation. This allows connector and transform configuration constraints
>>> to be expressed as structured metadata instead of being hidden in
>>> scattered
>>> runtime if/throw logic.
>>>
>>> Related PRs:
>>>
>>> - Main PR: apache/seatunnel#10977
>>> https://github.com/apache/seatunnel/pull/10977
>>>
>>> Completes the rich OptionRule validation framework.
>>>
>>> - Follow-up PR: apache/seatunnel#11228
>>> https://github.com/apache/seatunnel/pull/11228
>>>
>>> Preserves extension validation error context.
>>>
>>> - Follow-up PR: apache/seatunnel#11121
>>> https://github.com/apache/seatunnel/pull/11121
>>>
>>> Fixes validation error aggregation.
>>>
>>> - Follow-up PR: apache/seatunnel#11010
>>> https://github.com/apache/seatunnel/pull/11010
>>>
>>> Adds Map-type condition validation.
>>>
>>> - Follow-up PR: apache/seatunnel#11022
>>> https://github.com/apache/seatunnel/pull/11022
>>>
>>> Supports combining structural and value rules.
>>>
>>> - Follow-up PR: apache/seatunnel#11048
>>> https://github.com/apache/seatunnel/pull/11048
>>>
>>> Adds custom extension validation support.
>>>
>>> Why this should be included in 3.0:
>>>
>>> - It makes SeaTunnel's configuration contract clearer and more explicit.
>>> - Users can catch more configuration errors earlier through --check.
>>> - REST API, Web UI, CLI metadata export, and future AI-assisted config
>>> generation can consume richer validation metadata.
>>> - Connector and transform developers can reduce duplicated imperative
>>> validation logic.
>>> - It provides a stronger foundation for future configuration generation,
>>> validation, explanation, and repair capabilities.
>>>
>>> STIP-28 framework capabilities have already been merged, and some
>>> connectors have already completed migration. Therefore, I believe STIP-28
>>> is ready to be included in the SeaTunnel 3.0 roadmap.
>>>
>>> Best Regards,
>>> zhiwei.niu
>>>
>>> David Zollo <da...@gmail.com> 于2026年7月9日周四 21:19写道：
>>>
>>> > Hi guys,
>>> >
>>> > Thanks for your hard work and make a great progress. Now we only have
>>> the
>>> > following PRs left to be reviewed and merged.
>>> > Below is the list of remaining PRs pending merge in the Roadmap:
>>> >
>>> > 1. [#10563](https://github.com/apache/seatunnel/pull/10563)
>>> > [Feature][Connector-V2] Add checkpoint-gated post_sync_action and
>>> retention
>>> > for file source continuous discovery @yzeng1618 PR conflicts and need
>>> to
>>> > solve
>>> > 2. [#9993](https://github.com/apache/seatunnel/pull/9993)
>>> > [Feature][shade]Refactor the seatunnel-shade module. @hawk9821 PR
>>> > conflicts and need to solve
>>> > 3. [#10879](https://github.com/apache/seatunnel/pull/10879)
>>> > [Feature][Zeta]
>>> > STIP-24 Phase 2: Introduce standalone SeaTunnel Edge Agent MVP and dist
>>> > integration @nzw921rx
>>> > 4. [#10800](https://github.com/apache/seatunnel/pull/10800)
>>> > [Feature][Zeta]
>>> > STIP-23 Phase 1: Add engine timer flush core flow and FlushSignal
>>> handling
>>> > @nzw921rx
>>> > 5. [#10801](https://github.com/apache/seatunnel/pull/10801)
>>> > [Feature][Connector-V2] STIP-23 Phase 2: JDBC connector adoption for
>>> engine
>>> > timer flush @nzw921rx
>>> > 6. [#10306](https://github.com/apache/seatunnel/pull/10306)
>>> > [Feature][Core]
>>> > Error Data Bypass Capability for Transform & Sink (Dead Letter Queue)
>>> > @zhangshenghang
>>> >
>>> >
>>> > Guys, please help review the PR for the 3.0 Roadmap when you have time.
>>> >
>>> >
>>> > Best Regards,
>>> > David
>>> > Linkedin: https://www.linkedin.com/in/davidzollo
>>> >
>>> >
>>> >
>>> > On Wed, Jul 1, 2026 at 2:54 PM David Zollo <da...@gmail.com>
>>> > wrote:
>>> >
>>> > > Hi guys,
>>> > >
>>> > > I created an issue for the 3.0 roadmap to track progress[1]. Given
>>> that
>>> > > most PRs have been merged so far, we plan to finish merging the
>>> remaining
>>> > > PRs by the end of July and then kick off the release process!
>>> > >
>>> > > 1. https://github.com/apache/seatunnel/issues/11021
>>> > >
>>> > >
>>> > >
>>> > >
>>> > > Best Regards,
>>> > > David
>>> > > Linkedin: https://www.linkedin.com/in/davidzollo
>>> > >
>>> > >
>>> > >
>>> > > On Sat, May 16, 2026 at 12:32 PM David Zollo <
>>> davidzollo365@gmail.com>
>>> > > wrote:
>>> > >
>>> > >> Hi SeaTunnel community,
>>> > >>
>>> > >> Thanks everyone for the active discussion around the Apache
>>> SeaTunnel
>>> > 3.0
>>> > >> roadmap.
>>> > >>
>>> > >> To make the follow-up review easier, I summarized the proposals
>>> > collected
>>> > >> in this thread so far. This is only a discussion summary, not a
>>> final
>>> > >> decision. Please feel free to correct me if I missed anything or
>>> > >> misunderstood any proposal.
>>> > >>
>>> > >> 1.
>>> > >>
>>> > >> SeaTunnel 3.0 roadmap discussion
>>> > >>
>>> > >> The discussion was started to collect proposals for Apache
>>> SeaTunnel 3.0
>>> > >> after PR #10510, which updated the version to 3.0.0, was merged
>>> into the
>>> > >> dev branch on March 11, 2026.
>>> > >>
>>> > >> Several already-merged PRs were mentioned as possible parts of the
>>> 3.0
>>> > >> scope:
>>> > >>
>>> > >> -
>>> > >>
>>> > >> PR #10586: [Feature][Zeta] Add datasource spi
>>> > >> https://github.com/apache/seatunnel/pull/10586
>>> > >> -
>>> > >>
>>> > >> PR #10585: [Feature][Zeta] Add /option-rules REST API for runtime
>>> > >> connector metadata
>>> > >> https://github.com/apache/seatunnel/pull/10585
>>> > >> -
>>> > >>
>>> > >> PR #10489: [Feature][Transform] Add Zeta SQL UDF context and
>>> > >> lifecycle hooks with e2e/docs updates
>>> > >> https://github.com/apache/seatunnel/pull/10489
>>> > >>
>>> > >> The purpose of this thread is to collect PRs, features,
>>> improvements,
>>> > and
>>> > >> ideas that the community believes should be considered for SeaTunnel
>>> > 3.0.
>>> > >>
>>> > >> 2.
>>> > >>
>>> > >> STIP-23: Engine-level timer flush for Sink connectors
>>> > >>
>>> > >> zhiwei niu proposed including STIP-23 in the SeaTunnel 3.0 roadmap.
>>> The
>>> > >> proposal focuses on engine-level timer flush support for Sink
>>> > connectors.
>>> > >>
>>> > >> Related PRs:
>>> > >>
>>> > >> -
>>> > >>
>>> > >> PR #10800: [Feature][Zeta] Phase 1: Engine timer flush core
>>> > >> https://github.com/apache/seatunnel/pull/10800
>>> > >>
>>> > >> This PR adds sink.flush.interval and introduces a FlushSignal
>>> > >> control-plane path through the Zeta engine. It allows sinks to
>>> flush
>>> > based
>>> > >> on a wall-clock interval without each sink managing its own timer
>>> > thread.
>>> > >> -
>>> > >>
>>> > >> PR #10801: [Feature][Connector-V2] Phase 2: JDBC connector
>>> adoption
>>> > >> https://github.com/apache/seatunnel/pull/10801
>>> > >>
>>> > >> This PR wires registerFlushAction into the JDBC non-XA sink
>>> writer
>>> > >> through a new enable_timer_flush option. It also includes
>>> multi-table
>>> > >> aggregated flush and MySQL CDC E2E tests.
>>> > >> -
>>> > >>
>>> > >> PR #10802: [Docs][Connector-V2] Phase 3: Timer flush feature
>>> matrix
>>> > >> https://github.com/apache/seatunnel/pull/10802
>>> > >>
>>> > >> This PR adds a timer flush support marker to sink connector
>>> > >> documentation and documents the sink.flush.interval and
>>> > registerFlushAction
>>> > >> contract.
>>> > >>
>>> > >> The main value is to make end-to-end latency more predictable for
>>> > >> buffered sinks, especially in idle or low-throughput streaming jobs.
>>> > >>
>>> > >> 3.
>>> > >>
>>> > >> Independent seatunnel-shade project and packaging efficiency
>>> > >> optimization
>>> > >>
>>> > >> Zhang Donghao proposed including PR #9993 in the SeaTunnel 3.0
>>> roadmap.
>>> > >>
>>> > >> -
>>> > >>
>>> > >> PR #9993:
>>> > >> https://github.com/apache/seatunnel/pull/9993
>>> > >>
>>> > >> This PR extracts the shade modules into an independent
>>> seatunnel-shade
>>> > >> project, supports uploading shade jars to Maven Central, and
>>> optimizes
>>> > the
>>> > >> overall packaging efficiency of SeaTunnel.
>>> > >>
>>> > >> The proposal points out that current shade module builds, especially
>>> > >> Hadoop3 Uber and Hadoop AWS, add significant build time. By
>>> publishing
>>> > >> shade jars separately and reusing them as dependencies, SeaTunnel
>>> can
>>> > >> reduce repeated build cost and improve release stability.
>>> > >>
>>> > >> 4.
>>> > >>
>>> > >> Table-level fault isolation for multi-table synchronization
>>> > >>
>>> > >> yi zeng proposed including PR #10600 in the SeaTunnel 3.0 roadmap.
>>> > >>
>>> > >> -
>>> > >>
>>> > >> PR #10600: [Feature][Core] Support table-level fault isolation
>>> for
>>> > >> multi-table synchronization
>>> > >> https://github.com/apache/seatunnel/pull/10600
>>> > >>
>>> > >> This PR introduces table-level fault isolation for multi-table
>>> > >> synchronization through a new failure policy:
>>> > >>
>>> > >> env {
>>> > >> multi_table {
>>> > >> failure_policy = "CONTINUE_OTHER_TABLES"
>>> > >> }
>>> > >> }
>>> > >>
>>> > >> With this enabled, failures can be isolated at table granularity.
>>> > Healthy
>>> > >> tables can continue processing even if one table fails. Failed
>>> tables
>>> > are
>>> > >> collected and reported with metadata such as table path, failure
>>> phase,
>>> > >> plugin, and error.
>>> > >>
>>> > >> The proposal is backward-compatible by default because the existing
>>> > >> FAIL_FAST behavior remains unchanged unless users explicitly enable
>>> the
>>> > new
>>> > >> policy.
>>> > >>
>>> > >> 5.
>>> > >>
>>> > >> Checkpoint-gated post_sync_action and retention for file source
>>> > >> continuous discovery
>>> > >>
>>> > >> yi zeng also proposed including PR #10563.
>>> > >>
>>> > >> -
>>> > >>
>>> > >> PR #10563: [Feature][Connector-V2] Add checkpoint-gated
>>> > >> post_sync_action and retention for file source continuous
>>> discovery
>>> > >> https://github.com/apache/seatunnel/pull/10563
>>> > >>
>>> > >> This PR adds checkpoint-aware source-side lifecycle management for
>>> file
>>> > >> source continuous discovery, including:
>>> > >>
>>> > >> -
>>> > >>
>>> > >> post_sync_action = delete
>>> > >> -
>>> > >>
>>> > >> post_sync_action = backup
>>> > >> -
>>> > >>
>>> > >> retention_max_age
>>> > >>
>>> > >> The key idea is that source files should only be deleted or moved
>>> after
>>> > a
>>> > >> checkpoint has completed, so that continuous file ingestion remains
>>> safe
>>> > >> and recoverable in production environments.
>>> > >>
>>> > >> 6.
>>> > >>
>>> > >> STIP-21 / StainTrace: End-to-end data lineage tracing and
>>> performance
>>> > >> analysis for Zeta
>>> > >>
>>> > >> David proposed including PR #10491 in the SeaTunnel 3.0 roadmap.
>>> > >>
>>> > >> -
>>> > >>
>>> > >> PR #10491:
>>> > >> https://github.com/apache/seatunnel/pull/10491
>>> > >>
>>> > >> StainTrace is the implementation of STIP-21. It provides end-to-end
>>> data
>>> > >> lineage tracing and performance analysis for the Zeta engine.
>>> > >>
>>> > >> It adds lightweight timestamp sampling points at six key pipeline
>>> > stages:
>>> > >>
>>> > >> -
>>> > >>
>>> > >> SOURCE_EMIT
>>> > >> -
>>> > >>
>>> > >> QUEUE_IN
>>> > >> -
>>> > >>
>>> > >> QUEUE_OUT
>>> > >> -
>>> > >>
>>> > >> TRANSFORM_IN
>>> > >> -
>>> > >>
>>> > >> TRANSFORM_OUT
>>> > >> -
>>> > >>
>>> > >> SINK_WRITE_DONE
>>> > >>
>>> > >> The goal is to track the full journey of sampled rows from source
>>> > >> emission to sink write completion, making it easier to measure
>>> > end-to-end
>>> > >> latency and identify bottlenecks.
>>> > >>
>>> > >> The proposal emphasizes production safety: the feature is disabled
>>> by
>>> > >> default, supports configurable sampling, has a per-worker event
>>> budget,
>>> > >> writes trace data locally as JSONL in OpenTelemetry OTLP JSON
>>> format,
>>> > and
>>> > >> includes an offline analyzer for P95/P99 latency and HTML
>>> visualization
>>> > >> reports.
>>> > >>
>>> > >> 7.
>>> > >>
>>> > >> StateStore abstraction for Zeta engine
>>> > >>
>>> > >> Doyeon Kim proposed including PR #10812 in the SeaTunnel 3.0
>>> roadmap.
>>> > >>
>>> > >> -
>>> > >>
>>> > >> PR #10812:
>>> > >> https://github.com/apache/seatunnel/pull/10812
>>> > >>
>>> > >> This PR introduces an initial StateStore abstraction layer to reduce
>>> > >> direct Hazelcast IMap dependencies in the Zeta engine. It makes
>>> state
>>> > >> responsibilities more explicit and prepares the engine for future
>>> > evolution
>>> > >> around state storage, node communication, and coordination layers.
>>> > >>
>>> > >> 8.
>>> > >>
>>> > >> IMap external storage compaction
>>> > >>
>>> > >> Doyeon Kim also proposed including PR #10399.
>>> > >>
>>> > >> -
>>> > >>
>>> > >> PR #10399:
>>> > >> https://github.com/apache/seatunnel/pull/10399
>>> > >>
>>> > >> This PR introduces LSM-style write and compaction logic for IMap
>>> > external
>>> > >> storage. It helps control storage growth by flushing sorted
>>> immutable
>>> > files
>>> > >> and compacting persisted files while keeping only the latest record
>>> per
>>> > key.
>>> > >>
>>> > >> The proposal is intended to improve stability for long-running Zeta
>>> jobs
>>> > >> by reducing duplicated records and tombstones in external storage.
>>> > >>
>>> > >> 9.
>>> > >>
>>> > >> AI-Native CLI and AI for Data Engineering
>>> > >>
>>> > >> sean zhang proposed including the AI-Native CLI capability in
>>> SeaTunnel
>>> > >> 3.0.
>>> > >>
>>> > >> -
>>> > >>
>>> > >> PR #10789:
>>> > >> https://github.com/apache/seatunnel/pull/10789
>>> > >>
>>> > >> This PR introduces seatunnel-cli, an AI-powered command-line tool
>>> that
>>> > >> can generate production-ready pipeline configurations from natural
>>> > language.
>>> > >>
>>> > >> Current capabilities include:
>>> > >>
>>> > >> -
>>> > >>
>>> > >> Multi-LLM provider support, including Anthropic, OpenAI, and AWS
>>> > >> Bedrock
>>> > >> -
>>> > >>
>>> > >> Runtime connector metadata extraction via Java OptionRule
>>> reflection
>>> > >> -
>>> > >>
>>> > >> Source/Sink type-aware validation
>>> > >> -
>>> > >>
>>> > >> Auto-fix loop for generated configurations
>>> > >> -
>>> > >>
>>> > >> Sensitive value redaction before persistence and LLM transmission
>>> > >> -
>>> > >>
>>> > >> Interactive mode with session memory
>>> > >> -
>>> > >>
>>> > >> One-shot mode for CI/CD
>>> > >>
>>> > >> The proposal also describes a broader “AI for Data Engineering”
>>> roadmap,
>>> > >> including intelligent schema mapping, natural language transforms,
>>> data
>>> > >> profiling, failure diagnosis, self-healing pipelines, anomaly
>>> detection,
>>> > >> performance tuning, cost-aware planning, intelligent scheduling, and
>>> > >> eventually autonomous data infrastructure management.
>>> > >>
>>> > >> 10.
>>> > >>
>>> > >> Extension of the roadmap discussion window
>>> > >>
>>> > >> David proposed extending the SeaTunnel 3.0 roadmap discussion
>>> window to
>>> > >> May 15, 2026, considering the upcoming holiday period and to give
>>> > >> contributors more time to share ideas.
>>> > >>
>>> > >> This message also recognized the quality and breadth of the
>>> proposals so
>>> > >> far, including StainTrace, AI-Native CLI, StateStore abstraction,
>>> and
>>> > >> table-level fault isolation.
>>> > >>
>>> > >> 11.
>>> > >>
>>> > >> STIP-27: Apache Arrow as an optional internal data model
>>> > >>
>>> > >> xuepeng wang proposed adding STIP-27 to the SeaTunnel 3.0 roadmap
>>> > >> discussion.
>>> > >>
>>> > >> -
>>> > >>
>>> > >> Issue #10806:
>>> > >> https://github.com/apache/seatunnel/issues/10806
>>> > >>
>>> > >> This proposal introduces Apache Arrow as an optional internal data
>>> > model,
>>> > >> not as a replacement for the existing row model.
>>> > >>
>>> > >> The goal is to improve performance for columnar scenarios such as
>>> > >> Parquet, ORC, ClickHouse, Doris, and batch transforms such as
>>> > projection,
>>> > >> filter, and cast.
>>> > >>
>>> > >> Expected benefits include better columnar I/O performance,
>>> vectorized
>>> > >> execution, and lower JVM overhead.
>>> > >>
>>> > >> 12.
>>> > >>
>>> > >> STIP-24: Edge collection, EdgeSocket ingress, and SeaTunnel Edge
>>> > Agent
>>> > >>
>>> > >> zhiwei niu proposed adding STIP-24 edge-collection work to the
>>> SeaTunnel
>>> > >> 3.0 roadmap.
>>> > >>
>>> > >> The original PRs were:
>>> > >>
>>> > >> -
>>> > >>
>>> > >> PR #10871: [Feature][Zeta] STIP-24 Phase 1: Support lightweight
>>> > >> EdgeSocket ingress for edge collector MVP
>>> > >> https://github.com/apache/seatunnel/pull/10871
>>> > >> -
>>> > >>
>>> > >> PR #10876: [Feature][Zeta] STIP-24 Phase 2: Introduce standalone
>>> > >> SeaTunnel Edge Agent MVP and dist integration
>>> > >> https://github.com/apache/seatunnel/pull/10876
>>> > >>
>>> > >> Phase 1 introduces the engine-side ingress foundation for edge
>>> > collection
>>> > >> through EdgeSocket, including authenticated ingress, source-side
>>> > protocol
>>> > >> handling, and task-group address discovery support in Zeta.
>>> > >>
>>> > >> Phase 2 introduces a standalone SeaTunnel Edge Agent runtime,
>>> including
>>> > >> core, transport, connector, local file/log/event inputs, SQLite WAL
>>> > >> durability, EdgeSocket transport interoperability, and distribution
>>> > >> integration.
>>> > >>
>>> > >> The goal is to support an end-to-end edge collection workflow:
>>> > >>
>>> > >> edge host -> Zeta pipeline
>>> > >>
>>> > >> zhiwei later updated the PR references for STIP-24:
>>> > >>
>>> > >> -
>>> > >>
>>> > >> PR #10871 was updated to PR #10878
>>> > >> https://github.com/apache/seatunnel/pull/10878
>>> > >> -
>>> > >>
>>> > >> PR #10876 was updated to PR #10879
>>> > >> https://github.com/apache/seatunnel/pull/10879
>>> > >>
>>> > >>
>>> > >> 13.
>>> > >>
>>> > >> Prometheus metrics for Zeta Engine IMap state stores
>>> > >>
>>> > >> JeremyXin proposed including PR #10860 in the SeaTunnel 3.0 roadmap.
>>> > >>
>>> > >> -
>>> > >>
>>> > >> PR #10860:
>>> > >> https://github.com/apache/seatunnel/pull/10860
>>> > >>
>>> > >> This PR improves Zeta Engine observability by exposing Prometheus
>>> > metrics
>>> > >> for core engine state stores backed by Hazelcast IMap.
>>> > >>
>>> > >> It covers internal stores such as running job information, job
>>> states,
>>> > >> checkpoints, and slot ownership. It exposes metrics such as local
>>> entry
>>> > >> counts, backup entry counts, and heap cost.
>>> > >>
>>> > >> The proposal aims to help users detect state growth, data skew, and
>>> > >> memory pressure earlier, improving troubleshooting, capacity
>>> planning,
>>> > and
>>> > >> production stability.
>>> > >>
>>> > >> 14.
>>> > >>
>>> > >> Real-time observability for Zeta engine tasks
>>> > >>
>>> > >> zhangshenghang proposed improving real-time observability for Zeta
>>> > engine
>>> > >> tasks.
>>> > >>
>>> > >> This direction adds the ability to monitor the runtime busyness
>>> level of
>>> > >> Zeta engine tasks, including real-time task metrics for Source,
>>> > Transform,
>>> > >> and Sink execution. It also includes queue and backpressure-related
>>> > >> information exposed through REST API and Web UI.
>>> > >>
>>> > >> The goal is to help users understand whether a job is blocked by
>>> source
>>> > >> reading, transform processing, sink writing, or intermediate queues.
>>> > >>
>>> > >> 15.
>>> > >>
>>> > >> Error data bypass and dead-letter queue capability for Transform
>>> and
>>> > >> Sink
>>> > >>
>>> > >> zhangshenghang also proposed row-level error handling for Transform
>>> and
>>> > >> Sink stages.
>>> > >>
>>> > >> When configured, SeaTunnel can collect failed records with useful
>>> > >> context, including:
>>> > >>
>>> > >> -
>>> > >>
>>> > >> error stage
>>> > >> -
>>> > >>
>>> > >> plugin type
>>> > >> -
>>> > >>
>>> > >> plugin name
>>> > >> -
>>> > >>
>>> > >> source table
>>> > >> -
>>> > >>
>>> > >> error message
>>> > >> -
>>> > >>
>>> > >> exception class
>>> > >> -
>>> > >>
>>> > >> original data
>>> > >> -
>>> > >>
>>> > >> occurrence time
>>> > >>
>>> > >> This allows jobs to bypass selected dirty or error records instead
>>> of
>>> > >> always failing fast, while keeping the default behavior unchanged
>>> when
>>> > the
>>> > >> feature is not enabled.
>>> > >>
>>> > >> This is useful for long-running synchronization jobs where users
>>> need
>>> > >> better fault tolerance and later inspection of bad data.
>>> > >>
>>> > >> 16.
>>> > >>
>>> > >> Improve cloud-native deployment experience
>>> > >>
>>> > >> zhangshenghang also suggested improving the cloud-native deployment
>>> > >> experience for SeaTunnel 3.0.
>>> > >>
>>> > >> The proposal mentions Kubernetes and Helm support as important
>>> areas,
>>> > >> including better guidance and behavior around:
>>> > >>
>>> > >> -
>>> > >>
>>> > >> deployment
>>> > >> -
>>> > >>
>>> > >> restart
>>> > >> -
>>> > >>
>>> > >> persistent checkpoint storage
>>> > >> -
>>> > >>
>>> > >> configuration management
>>> > >> -
>>> > >>
>>> > >> secret management
>>> > >> -
>>> > >>
>>> > >> rolling upgrade
>>> > >> -
>>> > >>
>>> > >> resource isolation
>>> > >>
>>> > >>
>>> > >> 17.
>>> > >>
>>> > >> Clean up deprecated behavior carefully for 3.0
>>> > >>
>>> > >> zhangshenghang also suggested that SeaTunnel 3.0 is a good
>>> opportunity
>>> > to
>>> > >> clean up deprecated options and old APIs.
>>> > >>
>>> > >> At the same time, any incompatible change should be handled
>>> carefully,
>>> > >> with clear migration documentation, examples, and release notes.
>>> > >>
>>> > >> Possible discussion groups
>>> > >>
>>> > >> Based on the above proposals, I think the roadmap discussion can be
>>> > >> grouped into the following areas:
>>> > >>
>>> > >> 1.
>>> > >>
>>> > >> Zeta engine observability
>>> > >>
>>> > >>
>>> > >> -
>>> > >>
>>> > >> STIP-21 / StainTrace
>>> > >> -
>>> > >>
>>> > >> Prometheus metrics for IMap state stores
>>> > >> -
>>> > >>
>>> > >> Real-time task metrics
>>> > >> -
>>> > >>
>>> > >> Queue and backpressure visibility
>>> > >> -
>>> > >>
>>> > >> REST API and Web UI exposure
>>> > >>
>>> > >>
>>> > >> 2.
>>> > >>
>>> > >> Zeta state management and storage evolution
>>> > >>
>>> > >>
>>> > >> -
>>> > >>
>>> > >> StateStore abstraction
>>> > >> -
>>> > >>
>>> > >> IMap external storage compaction
>>> > >> -
>>> > >>
>>> > >> IMap state metrics
>>> > >> -
>>> > >>
>>> > >> Future storage backend evolution
>>> > >>
>>> > >>
>>> > >> 3.
>>> > >>
>>> > >> Runtime and connector semantics
>>> > >>
>>> > >>
>>> > >> -
>>> > >>
>>> > >> Engine-level timer flush for Sink connectors
>>> > >> -
>>> > >>
>>> > >> JDBC adoption of timer flush
>>> > >> -
>>> > >>
>>> > >> Checkpoint-gated file source lifecycle management
>>> > >> -
>>> > >>
>>> > >> Table-level fault isolation
>>> > >> -
>>> > >>
>>> > >> Row-level error bypass and dead-letter queue
>>> > >>
>>> > >>
>>> > >> 4.
>>> > >>
>>> > >> AI for Data Engineering
>>> > >>
>>> > >>
>>> > >> -
>>> > >>
>>> > >> AI-Native CLI
>>> > >> -
>>> > >>
>>> > >> Natural language pipeline generation
>>> > >> -
>>> > >>
>>> > >> Connector metadata reflection
>>> > >> -
>>> > >>
>>> > >> Auto-fix loop
>>> > >> -
>>> > >>
>>> > >> Security and sensitive value redaction
>>> > >> -
>>> > >>
>>> > >> Future AI-assisted schema mapping, diagnosis, optimization, and
>>> > >> autonomous data management
>>> > >>
>>> > >>
>>> > >> 5.
>>> > >>
>>> > >> Edge collection
>>> > >>
>>> > >>
>>> > >> -
>>> > >>
>>> > >> EdgeSocket ingress
>>> > >> -
>>> > >>
>>> > >> SeaTunnel Edge Agent
>>> > >> -
>>> > >>
>>> > >> Edge host to Zeta pipeline workflow
>>> > >> -
>>> > >>
>>> > >> Authentication, local durability, and transport interoperability
>>> > >>
>>> > >>
>>> > >> 6.
>>> > >>
>>> > >> Performance and internal data model
>>> > >>
>>> > >>
>>> > >> -
>>> > >>
>>> > >> Apache Arrow as an optional internal data model
>>> > >> -
>>> > >>
>>> > >> Columnar I/O
>>> > >> -
>>> > >>
>>> > >> Vectorized execution
>>> > >> -
>>> > >>
>>> > >> Lower JVM overhead
>>> > >>
>>> > >>
>>> > >> 7.
>>> > >>
>>> > >> Build, release, and project structure
>>> > >>
>>> > >>
>>> > >> -
>>> > >>
>>> > >> Independent seatunnel-shade project
>>> > >> -
>>> > >>
>>> > >> Maven Central distribution of shade jars
>>> > >> -
>>> > >>
>>> > >> Reduced build and release time
>>> > >>
>>> > >>
>>> > >> 8.
>>> > >>
>>> > >> Cloud-native and compatibility
>>> > >>
>>> > >>
>>> > >> -
>>> > >>
>>> > >> Kubernetes and Helm deployment experience
>>> > >> -
>>> > >>
>>> > >> Persistent checkpoint storage
>>> > >> -
>>> > >>
>>> > >> Secret management
>>> > >> -
>>> > >>
>>> > >> Rolling upgrade
>>> > >> -
>>> > >>
>>> > >> Resource isolation
>>> > >> -
>>> > >>
>>> > >> Deprecated behavior cleanup
>>> > >> -
>>> > >>
>>> > >> Migration documentation and release notes
>>> > >>
>>> > >> Next steps
>>> > >>
>>> > >> I suggest we continue to review these proposals and decide:
>>> > >>
>>> > >> 1.
>>> > >>
>>> > >> Which items should be included in the SeaTunnel 3.0 roadmap.
>>> > >> 2.
>>> > >>
>>> > >> Which items should be marked as experimental.
>>> > >> 3.
>>> > >>
>>> > >> Which items require an STIP or further design discussion.
>>> > >> 4.
>>> > >>
>>> > >> Which items can be merged independently before the 3.0 release.
>>> > >> 5.
>>> > >>
>>> > >> Which items need migration documentation, compatibility notes, or
>>> > >> release notes.
>>> > >>
>>> > >> Again, this is only a summary of the current discussion thread.
>>> Please
>>> > >> help correct or supplement anything I missed.
>>> > >>
>>> > >>
>>> > >>
>>> > >> Best Regards,
>>> > >> David
>>> > >> Linkedin: https://www.linkedin.com/in/davidzollo
>>> > >>
>>> > >>
>>> > >>
>>> > >> On Thu, May 14, 2026 at 2:59 PM zhangshenghang <
>>> shenghang@apache.org>
>>> > >> wrote:
>>> > >>
>>> > >>> Hi community,
>>> > >>>
>>> > >>> For the SeaTunnel 3.0 roadmap
>>> > >>>
>>> > >>> I have already submitted two related PRs, and besides those, I
>>> would
>>> > like
>>> > >>> to propose the following directions for 3.0:
>>> > >>>
>>> > >>> 1. Real-time observability for Zeta engine tasks
>>> > >>>
>>> > >>> This work adds the ability to monitor the runtime busyness level of
>>> > Zeta
>>> > >>> engine tasks. It introduces real-time task metrics for Source,
>>> > Transform,
>>> > >>> and Sink execution, including queue/backpressure-related
>>> information,
>>> > and
>>> > >>> exposes these metrics through REST API and Web UI. This can help
>>> users
>>> > >>> understand whether a job is blocked by source reading, transform
>>> > >>> processing, sink writing, or intermediate queues, which is very
>>> > important
>>> > >>> for production troubleshooting and performance tuning.
>>> > >>>
>>> > >>> 2. Error data bypass and dead-letter queue capability for
>>> Transform and
>>> > >>> Sink
>>> > >>>
>>> > >>> This work introduces row-level error handling for Transform and
>>> Sink
>>> > >>> stages. When configured, SeaTunnel can collect failed records with
>>> > useful
>>> > >>> context such as error stage, plugin type, plugin name, source
>>> table,
>>> > >>> error
>>> > >>> message, exception class, original data, and occurrence time. This
>>> > allows
>>> > >>> jobs to bypass selected dirty/error records instead of always
>>> failing
>>> > >>> fast,
>>> > >>> while keeping the default behavior unchanged when the feature is
>>> not
>>> > >>> enabled. This is useful for long-running synchronization jobs where
>>> > users
>>> > >>> need better fault tolerance and later inspection of bad data.
>>> > >>>
>>> > >>> 3. Improve cloud-native deployment experience
>>> > >>>
>>> > >>> Kubernetes and Helm support are important for new users. We should
>>> > >>> improve
>>> > >>> guidance and behavior around deployment, restart, persistent
>>> checkpoint
>>> > >>> storage, configuration management, secret management, rolling
>>> upgrade,
>>> > >>> and
>>> > >>> resource isolation in cloud-native environments.
>>> > >>>
>>> > >>> 4. Clean up deprecated behavior carefully for 3.0
>>> > >>>
>>> > >>> Since 3.0 is a major version, it is a good opportunity to clean up
>>> > >>> deprecated options and old APIs, but we should do this carefully.
>>> Any
>>> > >>> incompatible change should have clear migration documentation,
>>> > examples,
>>> > >>> and release notes.
>>> > >>>
>>> > >>>
>>> > >>> Regards,
>>> > >>> Jast (Shenghang)
>>> > >>>
>>> > >>>
>>> > >>> JeremyXin <73...@qq.com> 于2026年5月14日周四 14:27写道：
>>> > >>>
>>> > >>> > Hi everyone, I’d like to propose adding this PR to the SeaTunnel
>>> 3.0
>>> > >>> > roadmap.
>>> > >>> >
>>> > >>> >
>>> > >>> > - PR#10860
>>> > >>> > - link: https://github.com/apache/seatunnel/pull/10860
>>> > >>> >
>>> > >>> >
>>> > >>> > This PR improves Zeta Engine observability by exposing Prometheus
>>> > >>> metrics
>>> > >>> > for core engine state stores backed by Hazelcast IMap.
>>> > >>> >
>>> > >>> >
>>> > >>> > It covers important internal stores such as running job
>>> information,
>>> > >>> job
>>> > >>> > states, checkpoints, slot ownership and provides visibility into
>>> > their
>>> > >>> > local entry counts, backup entry counts, and heap cost.
>>> > >>> >
>>> > >>> >
>>> > >>> > Including this in SeaTunnel 3.0 would make the engine easier to
>>> > monitor
>>> > >>> > and operate in production. It helps users detect state growth,
>>> data
>>> > >>> skew,
>>> > >>> > and memory pressure earlier, which improves troubleshooting
>>> > efficiency,
>>> > >>> > cluster capacity planning, and overall operational stability.
>>> > >>> >
>>> > >>> >
>>> > >>> > Best regards,
>>> > >>> > JeremyXin
>>> > >>> >
>>> > >>> >
>>> > >>> > 原始邮件
>>> > >>> >
>>> > >>> >
>>> > >>> > 发件人：zhiwei niu <zniu70696@gmail.com>
>>> > >>> > 发件时间：2026年5月12日 10:27
>>> > >>> > 收件人：dev <dev@seatunnel.apache.org>
>>> > >>> > 主题：Re: [DISCUSS] Apache SeaTunnel 3.0 Roadmap
>>> > >>> >
>>> > >>> >
>>> > >>> >
>>> > >>> > Hi everyone,
>>> > >>> >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> As part of Roadmap 3.0 progress tracking, the STIP-24 PR links have been
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> updated to keep references consistent and easier to follow.
>>> > >>> >
>>> > >>> >
>>> Please use the updated PR links below:
>>> > >>> >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> [Feature][Zeta] STIP-24 Phase 1: Support lightweight EdgeSocket ingress
>>> > >>> > for edge collector MVP
>>> > >>> > - PR #10871(
>>> > >>> > https://github.com/apache/seatunnel/pull/10871)
>>> > >>> > —> PR#10878(
>>> > >>> > https://github.com/apache/seatunnel/pull/10878)
>>> > >>> >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> [Feature][Zeta] STIP-24 Phase 2: Introduce standalone SeaTunnel Edge Agent
>>> > >>> > MVP and dist integration
>>> > >>> > - PR #10876(
>>> > >>> > https://github.com/apache/seatunnel/pull/10876)
>>> > >>> > — PR#10879(
>>> > >>> > https://github.com/apache/seatunnel/pull/10879)
>>> > >>> >
>>> > >>> >
>>> > >>> > zhiwei niu <zniu70696@gmail.com
>>> > >>> > > 于2026年5月12日周二 01:31写道：
>>> > >>> >
>>> > >>> > > Hi everyone,
>>> > >>> > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> > In addition to STIP-23, I’d like to propose the STIP-24 edge-collection
>>> > >>> > > work for SeaTunnel 3.0.
>>> > >>> > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> > - PR #10871 — [Feature][Zeta] STIP-24 Phase 1: Support lightweight
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> > EdgeSocket ingress for edge collector MVP
>>> > >>> > > https://github.com/apache/seatunnel/pull/10871
>>> > >>> > > What it adds:
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> > Introduces the engine-side ingress foundation for edge collection via
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> > EdgeSocket, including authenticated ingress, source-side protocol handling,
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> > and task-group address discovery support in Zeta.
>>> > >>> > > Why 3.0:
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> > This is the foundational engine capability for remote edge data collection
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> > scenarios. It is backward-compatible and establishes the protocol/runtime
>>> > >>> >
>>> > >>>
>>> >
>>> > path required for the edge-side agent.
>>> > >>> > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> > - PR #10876 — [Feature][Zeta] STIP-24 Phase 2: Introduce standalone
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> > SeaTunnel Edge Agent MVP and dist integration
>>> > >>> > > https://github.com/apache/seatunnel/pull/10876
>>> > >>> > > What it adds:
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> > Introduces a standalone SeaTunnel Edge Agent runtime
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> > (core/transport/connector), local file/log/event inputs, SQLite WAL
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> > durability, EdgeSocket transport interop, and distribution integration.
>>> > >>> > > Why 3.0:
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> > This delivers the first practical end-to-end edge collection workflow
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> > (edge host -> Zeta pipeline), which significantly expands SeaTunnel’s
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> > deployment topology and real-world ingestion coverage.
>>> > >>> > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> > PR #10876 depends on #10871; both are under active review.
>>> > >>> > >
>>> > >>> > > Best regards,
>>> > >>> > > zhiwei.niu
>>> > >>> > >
>>> > >>> > > xuepeng wang <wxuepeng5@gmail.com
>>> > >>> > > 于2026年5月8日周五 14:03写道：
>>> > >>> > >
>>> > >>> > >> Hi team,
>>> > >>> > >>
>>> > >>> > >>
>>> > >>> > >>
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> I'd like to add STIP-27 [1] to the SeaTunnel 3.0 Roadmap discussion.
>>> > >>> > >>
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> This proposal introduces Apache Arrow as an optional internal data model
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> (not replacing the existing row model). It aims to improve performance for
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> columnar scenarios (Parquet/ORC, ClickHouse/Doris) and batch transforms
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> (projection/filter/cast), while keeping the current architecture stable.
>>> > >>> > >>
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> Key benefits: better columnar I/O performance, vectorized execution,
>>> > >>> > >> lower JVM overhead.
>>> > >>> > >>
>>> > >>> > >> [1]
>>> > >>> > https://github.com/apache/seatunnel/issues/10806
>>> > >>> > >> <https://github.com/apache/seatunnel/issues/10806>>
>>> >;
>>> > >>> > >> Thoughts?
>>> > >>> > >>
>>> > >>> > >> Regards,
>>> > >>> > >> xuepeng(github id:chl-wxp)
>>> > >>> > >>
>>> > >>> > >> David Zollo <davidzollo365@gmail.com
>>> > >>> > > 于2026年4月30日周四 21:17写道：
>>> > >>> > >>
>>> > >>> > >> > Hi everyone,
>>> > >>> > >> >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > First of all, thank you all for the fantastic proposals so far — the
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > quality and breadth of ideas in this thread have been genuinely
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > exciting to read. From StainTrace to the AI-Native CLI, from
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > StateStore abstraction to table-level fault isolation, it's clear that
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > this community has a bold and thoughtful vision for what SeaTunnel 3.0
>>> > >>> > >> > can become.
>>> > >>> > >> >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > Given that the Golden Week holiday is just around the corner, and many
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > of our contributors are based in China or taking time off during this
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > period, I'd like to extend the discussion window to May 15th. This
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > gives everyone enough breathing room to enjoy the holiday, come back
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > refreshed, and still have time to share their thoughts without feeling
>>> > >>> > >> > rushed.
>>> > >>> > >> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > Happy Labor Day to everyone!
>>> > >>> > >> >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > We highly welcome everyone's suggestions for version 3.0! Feel free to
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > leave a message if you have any ideas.
>>> > >>> > >> >
>>> > >>> > >> >
>>> > >>> > >> >
>>> > >>> > >> > Best Regards,
>>> > >>> > >> > David
>>> > >>> > >> > Linkedin:
>>> > >>> > https://www.linkedin.com/in/davidzollo
>>> > >>> > >> >
>>> > >>> > <https://www.linkedin.com/in/davidzollo>> >>;
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > On Wed, Apr 29, 2026 at 8:50 PM sean zhang <
>>> > >>> > seanzhang92@gmail.com>
>>> > >>> > >> wrote:
>>> > >>> > >> > >
>>> > >>> >
>>> >> > > Hi SeaTunnel community,
>>> > >>> > >> > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > I'd like to propose including the AI-Native CLI capability in the
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > SeaTunnel 3.0 release, and share a broader vision for the "AI for Data
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > Engineering" direction. I believe this aligns well with the 3.0
>>> > >>> > >> milestone
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > as a significant user-experience leap.
>>> > >>> > >> > > This PR
>>> > >>> > https://github.com/apache/seatunnel/pull/10789 introduces
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > `seatunnel-cli`, an AI-powered command-line tool that
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > generates production-ready pipeline configurations from natural
>>> > >>> > >> language
>>> > >>> > <
>>> > >>>
>>> >
>>> https://github.com/apache/seatunnel/pull/10789 introduces>> > > seatunnel-cli, an AI-powered command-line tool that>> > > generates production-ready pipeline configurations from natural>> language
>>> > >>> >
>>> > >>> > :
>>> > >>> > >> > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > $ seatunnel "Sync the users table from MySQL to S3 in parquet
>>> > >>> > >> format"
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > ✅ Generated config: mysql_users_to_s3.conf
>>> > >>> > >> > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > Key capabilities in the current implementation:
>>> > >>> > >> > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > - Multi-LLM provider support (Anthropic / OpenAI / AWS Bedrock)
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > with auto-detection
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > - Runtime connector metadata extraction via Java OptionRule
>>> > >>> > >> reflection,
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > ensuring 100% accurate parameter knowledge for all 100+ connectors
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > - Source/Sink type-aware validation (Jdbc source options ≠ Jdbc sink
>>> > >>> > >> > > options)
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > - Auto-fix loop: generated configs are validated and self-corrected
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > (up to 3 iterations)
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > - Security hardening: sensitive values (passwords, tokens, keys) are
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > automatically redacted to ${ENV_VAR} placeholders before
>>> > >>> > >> persistence
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > and LLM transmission
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > - Interactive mode with session memory and one-shot mode for CI/CD
>>> > >>> > >> > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > Future Roadmap — AI for Data Engineering
>>> > >>> > >> > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > Looking beyond config generation, I see a multi-level roadmap where
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > AI becomes deeply integrated into SeaTunnel's core value proposition:
>>> > >>> > >> > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > ### Level 2: AI Understands Data (Next phase)
>>> > >>> > >> > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > - Intelligent Schema Mapping: AI reads source schema, infers target
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > schema, and auto-generates type conversions and DDL
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > - Natural Language Transforms: "mask phone numbers, merge first and
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > last name, filter last 7 days" → SQL Transform auto-generation
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > - Data Profiling: AI samples source data, detects anomalies (NULLs,
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > outliers, format violations), and suggests data quality transforms
>>> > >>> > >> > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > ### Level 3: AI Operates Pipelines
>>> > >>> > >> > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > - Intelligent Failure Diagnosis: AI analyzes error logs, pinpoints
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > root cause (connection refused → MySQL down vs firewall), suggests
>>> > >>> > >> > fix
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > - Self-Healing Pipelines: auto-detect DDL changes in CDC sources,
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > auto-evolve sink schema, zero-downtime recovery
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > - Anomaly Detection: monitor data flow volume/latency, alert when
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > throughput drops 99% with correlated root cause
>>> > >>> > >> > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > ### Level 4: AI Optimizes Everything
>>> > >>> > >> > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > - Performance Tuning: AI analyzes bottlenecks (batch_size too small,
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > wrong file format, insufficient parallelism), suggests
>>> > >>> > >> optimization
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > - Cost-Aware Planning: given N tables to sync, AI proposes pipeline
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > topology that balances cost vs latency vs resource usage
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > - Intelligent Scheduling: learn from historical patterns to optimize
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > sync frequency per table
>>> > >>> > >> > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > ### Level 5: Autonomous Data Platform (Long-term vision)
>>> > >>> > >> > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > - Conversational data infrastructure management: "onboard the
>>> > >>> > >> payment
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > system to the data lake, ensure GDPR compliance" → AI
>>> > >>> > >> auto-discovers
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > sources, identifies PII, generates pipelines with masking
>>> > >>> > >> transforms,
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > deploys with monitoring and alerting
>>> > >>> > >> > >
>>> > >>> > >> > >
>>> > >>> > >> > > Doyeon Kim <
>>> > >>> > ekagnlek2@gmail.com> 于2026年4月29日周三 18:17写道：
>>> > >>> > >> > >
>>> > >>> > >> > > > Hi everyone,
>>> > >>> > >> > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > I’d like to propose the following Zeta engine improvements for
>>> > >>> > >> > SeaTunnel
>>> > >>> > >> > > > 3.0.
>>> > >>> > >> > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > 1. StateStore abstraction
>>> > >>> > >> > > > PR:
>>> > >>> > https://github.com/apache/seatunnel/pull/10812
>>> > >>> > >> > > >
>>> > >>> > <
>>> > >>>
>>> >
>>> https://github.com/apache/seatunnel/pull/10812>> > > >
>>> > >>> >
>>> > >>> > ;
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > This introduces an initial StateStore abstraction layer to reduce
>>> > >>> > >> > direct
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > Hazelcast IMap dependencies in the engine. It makes different state
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > responsibilities more explicit and prepares the engine for future
>>> > >>> > >> > backend
>>> > >>> > >> > > > evolution.
>>> > >>> > >> > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > I think this is relevant for 3.0 because it is a foundational step
>>> > >>> > >> > toward
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > decoupling the engine from Hazelcast-specific APIs and enabling
>>> > >>> > >> future
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > improvements around state storage, node communication, and
>>> > >>> > >> coordination
>>> > >>> > >> > > > layers.
>>> > >>> > >> > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > 2. IMap external storage compaction
>>> > >>> > >> > > > PR:
>>> > >>> > https://github.com/apache/seatunnel/pull/10399
>>> > >>> > >> > > >
>>> > >>> > <
>>> > >>>
>>> >
>>> https://github.com/apache/seatunnel/pull/10399>> > > >
>>> > >>> >
>>> > >>> > ;
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > This introduces LSM-style write and compaction logic for IMap
>>> > >>> > >> external
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > storage. It helps control storage growth by flushing sorted
>>> > >>> > >> immutable
>>> > >>> > >> > files
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > and compacting persisted files while keeping only the latest record
>>> > >>> > >> per
>>> > >>> > >> > > > key.
>>> > >>> > >> > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > I think this should be considered for 3.0 because long-running Zeta
>>> > >>> > >> > jobs
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > may accumulate duplicated records and tombstones in external storage
>>> > >>> > >> > over
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > time. Compaction can improve stability and reduce the risk of
>>> > >>> > >> unbounded
>>> > >>> > >> > > > storage growth.
>>> > >>> > >> > > >
>>> > >>> > >> > > > Best Regards,
>>> > >>> > >> > > > Doyeon
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > (Github ID: dybyte)
>>> > >>> > >> > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > On Wed, Apr 29, 2026 at 7:13 PM David Zollo <
>>> > >>> > >> davidzollo365@gmail.com>
>>> > >>> > >> > > > wrote:
>>> > >>> > >> > > >
>>> > >>> >
>>> >> > > > > Hi guys,
>>> > >>> > >> > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > I'd like to propose including PR #10491 in the SeaTunnel 3.0
>>> > >>> > >> roadmap,
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > here are the details with additional packaging time analysis:
>>> > >>> > >> > > > >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > ### PR link
>>> > >>> > >> > > > >
>>> > >>> > https://github.com/apache/seatunnel/pull/10491
>>> > >>> > >> > > > >
>>> > >>> > <
>>> > >>>
>>> >
>>> https://github.com/apache/seatunnel/pull/10491>> > > > >
>>> > >>> >
>>> > >>> > ;
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > StainTrace (PR #10491) is the concrete implementation of STIP-21,
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > delivering end-to-end data lineage tracing and performance
>>> > >>> > >> analysis
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > capabilities for the Apache SeaTunnel Zeta engine. It injects
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > lightweight timestamp sampling points at 6 key pipeline stages —
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > SOURCE_EMIT → QUEUE_IN → QUEUE_OUT → TRANSFORM_IN → TRANSFORM_OUT
>>> > >>> > >> →
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > SINK_WRITE_DONE — tracking each sampled row's complete journey
>>> > >>> > >> from
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > source emission to sink write completion, enabling precise
>>> > >>> > >> end-to-end
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > latency measurement and bottleneck identification.
>>> > >>> > >> > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > The system is designed with production safety as a first-class
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > concern: it is disabled by default with very low overhead, and
>>> > >>> > >> when
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > enabled it operates at a configurable sampling rate with a strict
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > per-worker per-second event budget, achieving approximately 0.1~1%
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > overhead at 1% sampling. Trace data is written locally as JSONL
>>> > >>> > >> files
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > in OpenTelemetry OTLP JSON format with no external dependencies.
>>> > >>> > >> An
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > accompanying offline analyzer reads these files to compute P95/P99
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > latency statistics, identify bottleneck stages, and generate an
>>> > >>> > >> HTML
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > visualization report with Chinese/English language toggle support.
>>> > >>> > >> > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > The value to SeaTunnel lies in closing a critical observability
>>> > >>> > >> gap
>>> > >>> > >> > at
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > the engine level. Before this feature, users facing latency
>>> > >>> > >> spikes or
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > throughput bottlenecks had no structured way to pinpoint whether
>>> > >>> > >> the
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > problem resided in queue backpressure, transform processing, or
>>> > >>> > >> sink
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > write time — they could only rely on logs and guesswork.
>>> > >>> > >> StainTrace
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > makes per-stage timing transparent and actionable, which is
>>> > >>> > >> > especially
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > impactful for large-scale, multi-table, long-chain production
>>> > >>> > >> > > > > pipelines.
>>> > >>> > >> > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > The case for including it in the 3.0 roadmap is equally strong. As
>>> > >>> > >> > the
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > implementation of STIP-21, StainTrace is a framework-level
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > observability capability — connector-agnostic and immediately
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > effective across all jobs once merged. One of SeaTunnel 3.0's
>>> > >>> > >> central
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > goals is to become a more mature, production-ready data
>>> > >>> > >> integration
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > engine, and observability is an indispensable foundation for that
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > goal. Establishing it at a major version boundary like 3.0
>>> > >>> > >> delivers
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > far greater long-term value to users.
>>> > >>> > >> > > > >
>>> > >>> > >> > > > >
>>> > >>> > >> > > > >
>>> > >>> > >> > > > >
>>> > >>> > >> > > > >
>>> > >>> > >> > > > >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > Best Regards,
>>> > >>> > >> > > > > David
>>> > >>> >
>>> >> > > > > Linkedin:
>>> > >>> > https://www.linkedin.com/in/davidzollo
>>> > >>> > >> > > > >
>>> > >>> > <
>>> > >>>
>>> >
>>> https://www.linkedin.com/in/davidzollo>> > > > >
>>> > >>> >
>>> > >>> > ;
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > On Wed, Apr 29, 2026 at 2:48 PM yi zeng <
>>> > >>> > yzeng1618@gmail.com>
>>> > >>> > >> wrote:
>>> > >>> > >> > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > Hi everyone,
>>> > >>> > >> > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > I’d like to propose including the following two PRs in the
>>> > >>> > >> > *SeaTunnel
>>> > >>> > >> > > > 3.0
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > roadmap*. Both focus on strengthening *correctness, reliability,
>>> > >>> > >> > and
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > operational safety* for core multi-table and continuous
>>> > >>> > >> ingestion
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > scenarios, which I believe are key pillars for 3.0.
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > ------------------------------
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > PR #10600 — [Feature][Core] Support table-level fault isolation
>>> > >>> > >> for
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > multi-table synchronization
>>> > >>> > >> > > > > >
>>> > >>> > >> > > > > >
>>> > >>> > https://github.com/apache/seatunnel/pull/10600
>>> > >>> > >> > > > > >
>>> > >>> > <
>>> > >>>
>>> >
>>> https://github.com/apache/seatunnel/pull/10600>> > > > > >
>>> > >>> >
>>> > >>> > ;
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > *What it adds / solves*
>>> > >>> > >> > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > This PR introduces *table-level fault isolation* for multi-table
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > synchronization by adding a new failure policy:
>>> > >>> >
>>> >> > > > > > code
>>> > >>> >
>>> >> > > > > > 复制
>>> > >>> > >> > > > > >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > env {
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > multi_table {
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > failure_policy = "CONTINUE_OTHER_TABLES"
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > }
>>> > >>> > >> > > > > > }
>>> > >>> > >> > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > With this enabled:
>>> > >>> > >> > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > - Failures are isolated at *table granularity*, instead of
>>> > >>> > >> > failing
>>> > >>> > >> > > > the
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > entire job.
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > - Healthy tables continue processing even if one table fails.
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > - Failed tables are collected and reported with clear
>>> > >>> > >> metadata
>>> > >>> > >> > > > (table
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > path, failure phase, plugin, error).
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > - Batch jobs report failed tables in final results; streaming
>>> > >>> > >> > jobs
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > continue running with healthy tables.
>>> > >>> > >> > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > This addresses a long-standing pain point where a single
>>> > >>> > >> > problematic
>>> > >>> > >> > > > > table
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > can block progress for all others.
>>> > >>> > >> > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > *Why it should be included in 3.0*
>>> > >>> > >> > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > - *Improved fault tolerance model*: 3.0 aims to be more
>>> > >>> > >> > resilient
>>> > >>> > >> > > > and
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > production-ready; table-level isolation is a foundational
>>> > >>> > >> > capability
>>> > >>> > >> > > > > for
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > large-scale, multi-table pipelines.
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > - *Backward-compatible by default*: The default behavior
>>> > >>> > >> remains
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > FAIL_FAST, so existing jobs are unaffected unless users
>>> > >>> > >> > explicitly
>>> > >>> > >> > > > opt
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > in.
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > - *Framework-level improvement*: This is not
>>> > >>> > >> > connector-specific; it
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > enhances the core execution and failure-handling semantics of
>>> > >>> > >> > > > > SeaTunnel.
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > - *Mature design & coverage*: Includes core logic changes,
>>> > >>> > >> > runtime
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > handling, reporting, and E2E tests, making it suitable for a
>>> > >>> > >> > major
>>> > >>> > >> > > > > version
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > milestone.
>>> > >>> > >> > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > ------------------------------
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > PR #10563 — [Feature][Connector-V2] Add checkpoint-gated
>>> > >>> > >> > > > post_sync_action
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > and retention for file source continuous discovery
>>> > >>> > >> > > > > >
>>> > >>> > >> > > > > >
>>> > >>> > https://github.com/apache/seatunnel/pull/10563
>>> > >>> > >> > > > > >
>>> > >>> > <
>>> > >>>
>>> >
>>> https://github.com/apache/seatunnel/pull/10563>> > > > > >
>>> > >>> >
>>> > >>> > ;
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > *What it adds / solves*
>>> > >>> > >> > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > This PR adds *checkpoint-gated source-side lifecycle management*
>>> > >>> > >> > for
>>> > >>> > >> > > > file
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > source continuous discovery, including:
>>> > >>> > >> > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > - post_sync_action = delete
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > Delete source files only *after a checkpoint has completed*
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > - post_sync_action = backup
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > Move source files to a backup_path after checkpoint
>>> > >>> > >> completion
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > - Optional backup retention via retention_max_age
>>> > >>> > >> > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > All operations are *checkpoint-aware*, ensuring source files are
>>> > >>> > >> > never
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > deleted or moved before the data is durably processed.
>>> > >>> > >> > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > *Why it should be included in 3.0*
>>> > >>> > >> > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > - *Correctness-first continuous ingestion*: Safe source
>>> > >>> > >> cleanup
>>> > >>> > >> > tied
>>> > >>> > >> > > > > to
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > checkpoint completion is essential for exactly-once and
>>> > >>> > >> > recoverable
>>> > >>> > >> > > > > file
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > ingestion, which aligns strongly with 3.0 quality goals.
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > - *Production-grade operational semantics*: Without
>>> > >>> > >> > checkpoint-gated
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > cleanup, continuous file ingestion is risky in real
>>> > >>> > >> production
>>> > >>> >
>>> >> > > > > environments.
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > - *Extends Connector-V2 maturity*: This PR deepens the
>>> > >>> > >> > operational
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > completeness of Connector-V2, especially for file-based
>>> > >>> > >> sources.
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > - *Well-tested and documented*: Includes unit tests, E2E
>>> > >>> > >> tests
>>> > >>> > >> > > > (Local
>>> > >>> > >> > > > > +
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > HDFS), and documentation updates (EN + ZH).
>>> > >>> > >> > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > ------------------------------
>>> > >>> > >> > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > Both PRs are *opt-in*, backward-compatible, and focused on
>>> > >>> > >> making
>>> > >>> > >> > > > > SeaTunnel
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > more robust, predictable, and safe in real-world deployments. I
>>> > >>> > >> > believe
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > they fit naturally into the SeaTunnel 3.0 scope, especially
>>> > >>> > >> given
>>> > >>> > >> > the
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > emphasis on stability, correctness, and large-scale usage.
>>> > >>> > >> > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > Happy to discuss details or answer questions.
>>> > >>> > >> > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > Zhang Donghao <
>>> > >>> > zhangdonghao@apache.org> 于2026年4月29日周三 14:38写道：
>>> > >>> > >> > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > Hi everyone,
>>> > >>> >
>>> >> > > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > I'd like to propose including PR #9993 in the SeaTunnel 3.0
>>> > >>> > >> > roadmap,
>>> > >>> > >> > > > > here
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > are the details with additional packaging time analysis:
>>> > >>> >
>>> >> > > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > ### PR link
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > >
>>> > >>> > https://github.com/apache/seatunnel/pull/9993
>>> > >>> >
>>> >> > > > > > >
>>> > >>> > <
>>> > >>>
>>> >
>>> https://github.com/apache/seatunnel/pull/9993>> > > > > > >
>>> > >>> >
>>> > >>> > ;
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > ### Short description
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > This PR extracts the shade modules into an independent
>>> > >>> > >> > > > seatunnel-shade
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > project, enables uploading shade-jars to the central
>>> > >>> > >> repository,
>>> > >>> > >> > and
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > optimizes the overall packaging efficiency of SeaTunnel.
>>> > >>> >
>>> >> > > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > ### Packaging time analysis (current state before PR
>>> > >>> > >> > optimization)
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > The current build time breakdown of individual shade modules
>>> > >>> > >> is
>>> > >>> > >> > as
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > follows, highlighting the inefficiency of the integrated build
>>> > >>> > >> > > > > approach:
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > ```
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > [INFO] SeaTunnel : Shade : 3.0.0 ..........................
>>> > >>> > >> > SUCCESS [
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > 12.661 s]
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > [INFO] SeaTunnel : Shade : Hazelcast 5.1-3.0.0 ............
>>> > >>> > >> > SUCCESS [
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > 8.192 s]
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > [INFO] SeaTunnel : Shade : Hadoop3 Uber 3.1.4-3.0.0 .......
>>> > >>> > >> > SUCCESS
>>> > >>> > >> > > > > [02:29
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > min]
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > [INFO] SeaTunnel : Shade : Jackson 2.13.3-3.0.0 ...........
>>> > >>> > >> > SUCCESS [
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > 3.289 s]
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > [INFO] SeaTunnel : Shade : Guava 27.0-jre-3.0.0 ...........
>>> > >>> > >> > SUCCESS [
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > 2.960 s]
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > [INFO] SeaTunnel : Shade : Hadoop : AWS 3.1.4-3.0.0 .......
>>> > >>> > >> > SUCCESS
>>> > >>> > >> > > > > [05:05
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > min]
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > [INFO] SeaTunnel : Shade : Arrow 15.0.1-3.0.0 .............
>>> > >>> > >> > SUCCESS [
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > 5.880 s]
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > [INFO] SeaTunnel : Shade : Hikari 4.0.3-3.0.0 .............
>>> > >>> > >> > SUCCESS [
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > 1.085 s]
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > [INFO] SeaTunnel : Shade : Thrift-Service 1.0.0-3.0.0 .....
>>> > >>> > >> > SUCCESS [
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > 1.572 s]
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > [INFO] SeaTunnel : Shade : Janino 3.0.11-3.0.0 ............
>>> > >>> > >> > SUCCESS [
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > 1.235 s]
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > [INFO] SeaTunnel : Shade : Scala 2.13.11-3.0.0 ............
>>> > >>> > >> > SUCCESS [
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > 6.586 s]
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > [INFO] SeaTunnel : Shade : Jetty 9.4.56.v20240826-3.0.0 ...
>>> > >>> > >> > SUCCESS [
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > 2.421 s]
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > [INFO] SeaTunnel : Shade : Commons Lang3 3.18.0-3.0.0 .....
>>> > >>> > >> > SUCCESS [
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > 2.044 s]
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > ```
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > Key observations from the time breakdown:
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > 1. **Long-tail modules dominate build time**: Two modules
>>> > >>> > >> > account for
>>> > >>> > >> > > > > the
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > vast majority of shade build time — `Hadoop3 Uber` (2 minutes
>>> > >>> > >> 29
>>> > >>> > >> > > > > seconds)
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > and `Hadoop : AWS` (5 minutes 5 seconds), totaling ~7.5
>>> > >>> > >> minutes,
>>> > >>> > >> > > > while
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > other modules take only a few seconds each.
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > 2. **Integrated build waste**: In the current structure, these
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > time-consuming shade modules are built alongside core modules
>>> > >>> > >> > every
>>> > >>> > >> > > > > time
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > SeaTunnel is packaged, even when there are no changes to shade
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > dependencies. This results in repeated long build cycles for
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > developers and
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > release workflows.
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > 3. **Total shade build time**: The cumulative build time of
>>> > >>> > >> all
>>> > >>> > >> > shade
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > modules is approximately 8 minutes (excluding
>>> > >>> > >> > overlapping/parallel
>>> > >>> > >> > > > > build
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > optimizations), which is a significant portion of the overall
>>> > >>> > >> > > > SeaTunnel
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > packaging process.
>>> > >>> >
>>> >> > > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > ### Reason for including in 3.0
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > 1. **Build efficiency optimization (quantifiable gains)**:
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > - By isolating shade modules into an independent project
>>> > >>> > >> and
>>> > >>> > >> > > > > uploading
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > shade-jars to the central repository, we eliminate the need to
>>> > >>> > >> > > > rebuild
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > these modules in every SeaTunnel packaging cycle. For daily
>>> > >>> >
>>> >> > > > > development,
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > this reduces the total build time by ~8 minutes per build (for
>>> > >>> > >> > > > > developers
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > not modifying shade code).
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > - For the 3.0 release process, we can pre-build and publish
>>> > >>> > >> > > > > shade-jars
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > to the central repository once, then reference them as
>>> > >>> > >> > dependencies —
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > cutting the release packaging time by a substantial margin and
>>> > >>> >
>>> > >> > > > > reducing the
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > risk of build failures caused by long-running shade module
>>> > >>> > >> > builds.
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > 2. **Dependency management enhancement**: Uploading shade-jars
>>> > >>> > >> > to the
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > central repository standardizes dependency distribution,
>>> > >>> > >> avoiding
>>> > >>> > >> > > > > conflicts
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > caused by internal shade module duplication across
>>> > >>> > >> subprojects,
>>> > >>> > >> > and
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > improving the stability of the 3.0 version's dependency
>>> > >>> > >> system.
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > 3. **Project structure rationalization**: Isolating shade
>>> > >>> > >> modules
>>> > >>> > >> > > > > aligns
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > with the modular design concept of 3.0, making the project
>>> > >>> > >> > structure
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > clearer, easier to maintain, and facilitating future
>>> > >>> > >> expansion of
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > shade-related capabilities (e.g., adding new shade modules for
>>> > >>> > >> > > > > additional
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > dependencies).
>>> > >>> >
>>> >> > > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > This optimization directly addresses the identified packaging
>>> > >>> > >> > > > > bottlenecks
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > (long-tail shade modules) and delivers measurable time savings
>>> > >>> > >> > for
>>> > >>> > >> > > > both
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > development and release workflows — a critical improvement for
>>> > >>> > >> > the
>>> > >>> > >> > > > > smooth
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > delivery and maintainability of SeaTunnel 3.0.
>>> > >>> >
>>> >> > > > > > >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > Best regards,
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> >
>>> >> > > > > > > zhangdonghao
>>> > >>> >
>>> >> > > > > > >
>>> > >>> > >> > > > >
>>> > >>> > >> > > >
>>> > >>> > >> >
>>> > >>> > >>
>>> > >>> > >
>>> > >>>
>>> > >>
>>> >
>>>
>>