You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@seatunnel.apache.org by zhiwei niu <zn...@gmail.com> on 2026/09/23 16:41:28 UTC
[DISCUSS][STIP-41] Native Zeta Application Mode on YARN and Kubernetes
Hi community,
I would like to start a discussion about STIP-41: introducing a native Zeta
Application Mode on YARN and Kubernetes.
Proposal: https://github.com/apache/seatunnel/issues/12457
Today, SeaTunnel users can submit jobs to a long-running Zeta cluster. This
model works well for shared deployments, but it also means that multiple
jobs share the same cluster lifecycle, resources, dependencies, and failure
boundary.
Many organizations already use YARN or Kubernetes as their resource
management platform and expect each submitted job to run as an independent
application. Without native support, users must maintain platform-specific
scripts to start the Zeta Master and Workers, configure cluster discovery,
submit the job, monitor failures, and clean up resources. These custom
solutions are difficult to keep reliable and consistent.
Therefore, I propose adding a native Application Mode for the Zeta engine.
The first phase will provide one isolated Zeta cluster for one job, with
one Master and a configurable fixed number of Workers. It will support
platform-native submission, status, cancellation, resource cleanup,
multi-Worker parallel execution, standard SeaTunnel distribution
integration, and recovery in a new application from a durable checkpoint
stored in HDFS, object storage, or a Kubernetes persistent volume.
Looking forward to your feedback.
Best regards,
Zhiwei Niu
STIP-41: Native Zeta Application Mode on YARN and Kubernetes
<https://drive.google.com/open?id=1JMzeT9PBaKkVDgfsN5E9VW67GdMufGHgWTe52LvSwFs>
Re: [DISCUSS][STIP-41] Native Zeta Application Mode on YARN and Kubernetes
Posted by zhiwei niu <zn...@gmail.com>.
Hi all,
The MVP version of Zeta application mode support for Yarn and Kubernetes
has been completed.
We look forward to everyone trying it out and reviewing it. Let's expand
Seatunnel's use cases together and make it easier for users to use.
1. The service scenarios of STIP-41 are:
1.1 One-time batch jobs that are destroyed immediately after use.
1.2 Teams with large-scale batch jobs that require resource isolation.
2. Why STIP-41 supports both Yarn and Kubernetes:
The original intention was to abstract out a complete boundary so that
we can later enhance Yarn and Kubernetes in parallel, thereby reducing
architectural debt. With the support of AI,
our coding capability is no longer a burden, so advancing Yarn and
Kubernetes support is no longer a manpower bottleneck.
3. Scenarios that can be explored later:
3.1 Airflow and DolphinScheduler support application mode.
3.2 Support PyPI distribution and directly run native Zeta applications
through a Python Launcher, lowering the barrier to entry for Python data
engineers and conveniently handling data backfills and daily one-time tasks.
We look forward to everyone's feedback and review!
Best regards,
Zhiwei Niu
zhiwei niu <zn...@gmail.com> 于2026年9月30日周三 15:52写道：
> Hi community,
>
> STIP 41 has been implemented and completed. If you are interested, you can
> compile and try it out. If you have any questions, please feel free to give
> feedback.
>
> PR: https://github.com/apache/seatunnel/pull/12460
> Issue: https://github.com/apache/seatunnel/issues/12457
>
> Best regards,
> Zhiwei Niu
>
>
> David Zollo <da...@gmail.com> 于2026年9月28日周一 14:03写道：
>
>> Sounds Good. Thanks
>>
>>
>>
>> Best Regards,
>> David
>> Linkedin: https://www.linkedin.com/in/davidzollo
>>
>>
>>
>> On Mon, Sep 28, 2026 at 12:43 AM zhiwei niu <zn...@gmail.com> wrote:
>>
>> > Hi david
>> >
>> > The STI-41 MVP stage supports yarn and k8s to abstract and stabilize API
>> > boundaries. The subsequent plan is to first fully enhance k8s and
>> > supplement boundary conditions.
>> > I think our goal is consistent.
>> >
>> > Best regards,
>> > Zhiwei Niu
>> >
>> >
>> > David Zollo <da...@gmail.com> 于2026年9月24日周四 23:15写道：
>> >
>> > > Good proposal. I'd prefer to Kubernetes first.
>> > >
>> > >
>> > >
>> > > Best Regards,
>> > > David
>> > > Linkedin: https://www.linkedin.com/in/davidzollo
>> > >
>> > >
>> > >
>> > > On Wed, Sep 23, 2026 at 12:43 PM zhiwei niu <zn...@gmail.com>
>> wrote:
>> > >
>> > > > Hi community,
>> > > >
>> > > > I would like to start a discussion about STIP-41: introducing a
>> native
>> > > Zeta
>> > > > Application Mode on YARN and Kubernetes.
>> > > >
>> > > > Proposal: https://github.com/apache/seatunnel/issues/12457
>> > > >
>> > > > Today, SeaTunnel users can submit jobs to a long-running Zeta
>> cluster.
>> > > This
>> > > > model works well for shared deployments, but it also means that
>> > multiple
>> > > > jobs share the same cluster lifecycle, resources, dependencies, and
>> > > failure
>> > > > boundary.
>> > > >
>> > > > Many organizations already use YARN or Kubernetes as their resource
>> > > > management platform and expect each submitted job to run as an
>> > > independent
>> > > > application. Without native support, users must maintain
>> > > platform-specific
>> > > > scripts to start the Zeta Master and Workers, configure cluster
>> > > discovery,
>> > > > submit the job, monitor failures, and clean up resources. These
>> custom
>> > > > solutions are difficult to keep reliable and consistent.
>> > > >
>> > > > Therefore, I propose adding a native Application Mode for the Zeta
>> > > engine.
>> > > >
>> > > > The first phase will provide one isolated Zeta cluster for one job,
>> > with
>> > > > one Master and a configurable fixed number of Workers. It will
>> support
>> > > > platform-native submission, status, cancellation, resource cleanup,
>> > > > multi-Worker parallel execution, standard SeaTunnel distribution
>> > > > integration, and recovery in a new application from a durable
>> > checkpoint
>> > > > stored in HDFS, object storage, or a Kubernetes persistent volume.
>> > > >
>> > > > Looking forward to your feedback.
>> > > >
>> > > >
>> > > > Best regards,
>> > > > Zhiwei Niu
>> > > >
>> > > > STIP-41: Native Zeta Application Mode on YARN and Kubernetes
>> > > > <
>> > > >
>> > >
>> >
>> https://drive.google.com/open?id=1JMzeT9PBaKkVDgfsN5E9VW67GdMufGHgWTe52LvSwFs
>> > > > >
>> > > >
>> > >
>> >
>>
>
Re: [DISCUSS][STIP-41] Native Zeta Application Mode on YARN and Kubernetes
Posted by zhiwei niu <zn...@gmail.com>.
Hi community,
STIP 41 has been implemented and completed. If you are interested, you can
compile and try it out. If you have any questions, please feel free to give
feedback.
PR: https://github.com/apache/seatunnel/pull/12460
Issue: https://github.com/apache/seatunnel/issues/12457
Best regards,
Zhiwei Niu
David Zollo <da...@gmail.com> 于2026年9月28日周一 14:03写道：
> Sounds Good. Thanks
>
>
>
> Best Regards,
> David
> Linkedin: https://www.linkedin.com/in/davidzollo
>
>
>
> On Mon, Sep 28, 2026 at 12:43 AM zhiwei niu <zn...@gmail.com> wrote:
>
> > Hi david
> >
> > The STI-41 MVP stage supports yarn and k8s to abstract and stabilize API
> > boundaries. The subsequent plan is to first fully enhance k8s and
> > supplement boundary conditions.
> > I think our goal is consistent.
> >
> > Best regards,
> > Zhiwei Niu
> >
> >
> > David Zollo <da...@gmail.com> 于2026年9月24日周四 23:15写道：
> >
> > > Good proposal. I'd prefer to Kubernetes first.
> > >
> > >
> > >
> > > Best Regards,
> > > David
> > > Linkedin: https://www.linkedin.com/in/davidzollo
> > >
> > >
> > >
> > > On Wed, Sep 23, 2026 at 12:43 PM zhiwei niu <zn...@gmail.com>
> wrote:
> > >
> > > > Hi community,
> > > >
> > > > I would like to start a discussion about STIP-41: introducing a
> native
> > > Zeta
> > > > Application Mode on YARN and Kubernetes.
> > > >
> > > > Proposal: https://github.com/apache/seatunnel/issues/12457
> > > >
> > > > Today, SeaTunnel users can submit jobs to a long-running Zeta
> cluster.
> > > This
> > > > model works well for shared deployments, but it also means that
> > multiple
> > > > jobs share the same cluster lifecycle, resources, dependencies, and
> > > failure
> > > > boundary.
> > > >
> > > > Many organizations already use YARN or Kubernetes as their resource
> > > > management platform and expect each submitted job to run as an
> > > independent
> > > > application. Without native support, users must maintain
> > > platform-specific
> > > > scripts to start the Zeta Master and Workers, configure cluster
> > > discovery,
> > > > submit the job, monitor failures, and clean up resources. These
> custom
> > > > solutions are difficult to keep reliable and consistent.
> > > >
> > > > Therefore, I propose adding a native Application Mode for the Zeta
> > > engine.
> > > >
> > > > The first phase will provide one isolated Zeta cluster for one job,
> > with
> > > > one Master and a configurable fixed number of Workers. It will
> support
> > > > platform-native submission, status, cancellation, resource cleanup,
> > > > multi-Worker parallel execution, standard SeaTunnel distribution
> > > > integration, and recovery in a new application from a durable
> > checkpoint
> > > > stored in HDFS, object storage, or a Kubernetes persistent volume.
> > > >
> > > > Looking forward to your feedback.
> > > >
> > > >
> > > > Best regards,
> > > > Zhiwei Niu
> > > >
> > > > STIP-41: Native Zeta Application Mode on YARN and Kubernetes
> > > > <
> > > >
> > >
> >
> https://drive.google.com/open?id=1JMzeT9PBaKkVDgfsN5E9VW67GdMufGHgWTe52LvSwFs
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS][STIP-41] Native Zeta Application Mode on YARN and Kubernetes
Posted by David Zollo <da...@gmail.com>.
Sounds Good. Thanks
Best Regards,
David
Linkedin: https://www.linkedin.com/in/davidzollo
On Mon, Sep 28, 2026 at 12:43 AM zhiwei niu <zn...@gmail.com> wrote:
> Hi david
>
> The STI-41 MVP stage supports yarn and k8s to abstract and stabilize API
> boundaries. The subsequent plan is to first fully enhance k8s and
> supplement boundary conditions.
> I think our goal is consistent.
>
> Best regards,
> Zhiwei Niu
>
>
> David Zollo <da...@gmail.com> 于2026年9月24日周四 23:15写道：
>
> > Good proposal. I'd prefer to Kubernetes first.
> >
> >
> >
> > Best Regards,
> > David
> > Linkedin: https://www.linkedin.com/in/davidzollo
> >
> >
> >
> > On Wed, Sep 23, 2026 at 12:43 PM zhiwei niu <zn...@gmail.com> wrote:
> >
> > > Hi community,
> > >
> > > I would like to start a discussion about STIP-41: introducing a native
> > Zeta
> > > Application Mode on YARN and Kubernetes.
> > >
> > > Proposal: https://github.com/apache/seatunnel/issues/12457
> > >
> > > Today, SeaTunnel users can submit jobs to a long-running Zeta cluster.
> > This
> > > model works well for shared deployments, but it also means that
> multiple
> > > jobs share the same cluster lifecycle, resources, dependencies, and
> > failure
> > > boundary.
> > >
> > > Many organizations already use YARN or Kubernetes as their resource
> > > management platform and expect each submitted job to run as an
> > independent
> > > application. Without native support, users must maintain
> > platform-specific
> > > scripts to start the Zeta Master and Workers, configure cluster
> > discovery,
> > > submit the job, monitor failures, and clean up resources. These custom
> > > solutions are difficult to keep reliable and consistent.
> > >
> > > Therefore, I propose adding a native Application Mode for the Zeta
> > engine.
> > >
> > > The first phase will provide one isolated Zeta cluster for one job,
> with
> > > one Master and a configurable fixed number of Workers. It will support
> > > platform-native submission, status, cancellation, resource cleanup,
> > > multi-Worker parallel execution, standard SeaTunnel distribution
> > > integration, and recovery in a new application from a durable
> checkpoint
> > > stored in HDFS, object storage, or a Kubernetes persistent volume.
> > >
> > > Looking forward to your feedback.
> > >
> > >
> > > Best regards,
> > > Zhiwei Niu
> > >
> > > STIP-41: Native Zeta Application Mode on YARN and Kubernetes
> > > <
> > >
> >
> https://drive.google.com/open?id=1JMzeT9PBaKkVDgfsN5E9VW67GdMufGHgWTe52LvSwFs
> > > >
> > >
> >
>
Re: [DISCUSS][STIP-41] Native Zeta Application Mode on YARN and Kubernetes
Posted by zhiwei niu <zn...@gmail.com>.
Hi david
The STI-41 MVP stage supports yarn and k8s to abstract and stabilize API
boundaries. The subsequent plan is to first fully enhance k8s and
supplement boundary conditions.
I think our goal is consistent.
Best regards,
Zhiwei Niu
David Zollo <da...@gmail.com> 于2026年9月24日周四 23:15写道：
> Good proposal. I'd prefer to Kubernetes first.
>
>
>
> Best Regards,
> David
> Linkedin: https://www.linkedin.com/in/davidzollo
>
>
>
> On Wed, Sep 23, 2026 at 12:43 PM zhiwei niu <zn...@gmail.com> wrote:
>
> > Hi community,
> >
> > I would like to start a discussion about STIP-41: introducing a native
> Zeta
> > Application Mode on YARN and Kubernetes.
> >
> > Proposal: https://github.com/apache/seatunnel/issues/12457
> >
> > Today, SeaTunnel users can submit jobs to a long-running Zeta cluster.
> This
> > model works well for shared deployments, but it also means that multiple
> > jobs share the same cluster lifecycle, resources, dependencies, and
> failure
> > boundary.
> >
> > Many organizations already use YARN or Kubernetes as their resource
> > management platform and expect each submitted job to run as an
> independent
> > application. Without native support, users must maintain
> platform-specific
> > scripts to start the Zeta Master and Workers, configure cluster
> discovery,
> > submit the job, monitor failures, and clean up resources. These custom
> > solutions are difficult to keep reliable and consistent.
> >
> > Therefore, I propose adding a native Application Mode for the Zeta
> engine.
> >
> > The first phase will provide one isolated Zeta cluster for one job, with
> > one Master and a configurable fixed number of Workers. It will support
> > platform-native submission, status, cancellation, resource cleanup,
> > multi-Worker parallel execution, standard SeaTunnel distribution
> > integration, and recovery in a new application from a durable checkpoint
> > stored in HDFS, object storage, or a Kubernetes persistent volume.
> >
> > Looking forward to your feedback.
> >
> >
> > Best regards,
> > Zhiwei Niu
> >
> > STIP-41: Native Zeta Application Mode on YARN and Kubernetes
> > <
> >
> https://drive.google.com/open?id=1JMzeT9PBaKkVDgfsN5E9VW67GdMufGHgWTe52LvSwFs
> > >
> >
>
Re: [DISCUSS][STIP-41] Native Zeta Application Mode on YARN and Kubernetes
Posted by David Zollo <da...@gmail.com>.
Good proposal. I'd prefer to Kubernetes first.
Best Regards,
David
Linkedin: https://www.linkedin.com/in/davidzollo
On Wed, Sep 23, 2026 at 12:43 PM zhiwei niu <zn...@gmail.com> wrote:
> Hi community,
>
> I would like to start a discussion about STIP-41: introducing a native Zeta
> Application Mode on YARN and Kubernetes.
>
> Proposal: https://github.com/apache/seatunnel/issues/12457
>
> Today, SeaTunnel users can submit jobs to a long-running Zeta cluster. This
> model works well for shared deployments, but it also means that multiple
> jobs share the same cluster lifecycle, resources, dependencies, and failure
> boundary.
>
> Many organizations already use YARN or Kubernetes as their resource
> management platform and expect each submitted job to run as an independent
> application. Without native support, users must maintain platform-specific
> scripts to start the Zeta Master and Workers, configure cluster discovery,
> submit the job, monitor failures, and clean up resources. These custom
> solutions are difficult to keep reliable and consistent.
>
> Therefore, I propose adding a native Application Mode for the Zeta engine.
>
> The first phase will provide one isolated Zeta cluster for one job, with
> one Master and a configurable fixed number of Workers. It will support
> platform-native submission, status, cancellation, resource cleanup,
> multi-Worker parallel execution, standard SeaTunnel distribution
> integration, and recovery in a new application from a durable checkpoint
> stored in HDFS, object storage, or a Kubernetes persistent volume.
>
> Looking forward to your feedback.
>
>
> Best regards,
> Zhiwei Niu
>
> STIP-41: Native Zeta Application Mode on YARN and Kubernetes
> <
> https://drive.google.com/open?id=1JMzeT9PBaKkVDgfsN5E9VW67GdMufGHgWTe52LvSwFs
> >
>