You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@tez.apache.org by YifengLu <no...@163.com> on 2024/01/24 05:50:36 UTC

tez on k8s

Hi, Tez Team：
We are from the bytedance EMR team，and we would like to inquire about the plans for Tez on Kubernetes (K8S) and express our interest in Tez on K8S. We hope to have the opportunity to collaborate with you on this project.
We understand that Tez is a powerful distributed computing framework, while K8S is a popular container orchestration platform. We believe that deploying Tez on K8S can provide better scalability and manageability. We are very excited about the potential of this project and would like to be involved.
If you have any plans or ideas regarding Tez on K8S, we would be eager to listen and provide assistance. We have experience with K8S and can support the development and deployment of the project. Additionally, we would like to discuss how we can better integrate Tez with K8S to meet the needs of users.
We believe that through joint efforts, we can make Tez on K8S an excellent open-source project and bring greater value to the community.
Finally, thank you very much for taking the time to read this email. If you have any questions or would like to further discuss, please feel free to contact me. Looking forward to your response.


Best regards,


Yifeng Lu

Re: Re: tez on k8s

Posted by Butao Zhang <bu...@163.com>.

Thanks Yifeng for raising this thread!

Very glad to see the topic about tez on k8s. Tez on k8s would give tez more possibilities & elasticity. I think many users include me want to be involved in the development of tez on k8s. I am also thinking what Hive should to do to launch the tez on k8s job. :)
Anyway, Let's move forward this significant feature to make tez cloud-native!

Actually, we have made some attempts before. The scenario at that time may have been somewhat unique as we had our own client-side, used Tez AM and our custom Worker. We have completed the Kubernetes adaption of Tez AM and our custom Worker, Tez AM utilizes the K8S default scheduler for resource allocation and manages the startup and shutdown of the custom Worker. We have used it in production and served a significant number of batch processing jobs.

However, this differs significantly from the community's scenario, so we will carefully consider how to contribute back to the community.

Best regards,
Yifeng Lu

At 2024-01-29 16:52:50, "László Bodor" <bo...@gmail.com> wrote:
Hi Yifeng!

I can tell that the most spectacular step toward tez on Kubernetes was the
unmanaged sessions umbrella: https://issues.apache.org/jira/browse/TEZ-3991
I can see some of those tasks are still not committed, however, unmanaged
sessions have been working for years in a Cloudera product, so we would be
happy to review these if someone picked them up and created a PR.

Once unmanaged sessions are done, we can explore other ways. Tez is still
tightly coupled with Yarn I believe, so on the road to k8s, we have to
explore the following:
1. *Tez AM* to be started by another resource manager than Yarn (which is
slightly different from the unmanaged sessions above, as unmanaged sessions
start independently of the client (e.g. Hive's hiveserver2)) <- this is not
a must-have as long as the unmanaged sessions approach is fine for the K8s
use-cases.
2. *Tez containers: t*his is a huge gap: the product I know works by Hive
LLAP as executors, so similarly to the sessions, the Hive LLAP containers
start independently of Tez, so if we want to make this work in Tez out of
the box, we need to implement new things like start a TezChild container or
rather a group of TezChild containers as a k8s pod for instance, just
thinking aloud.

Thanks for coming up with this topic. Let us get back to this later if any
plans, in the meantime, finishing the upstream contribution of TEZ-3991 is
more than welcome.

Regards,
Laszlo Bodor

YifengLu <no...@163.com> ezt írta (időpont: 2024. jan. 29., H, 3:52):

Thank you very much for your response. We will continue to explore Tez on
Kubernetes, and if there are any further developments, we will reach out to
the community. Once again, thank you.
Best regards,
Yifeng Lu

Re:Re: Re: tez on k8s

Posted by YifengLu <no...@163.com>.

Thank you very much for providing us with the PR and giving us insights into the implementation approach. It has been of great help to us.


Actually, we have made some attempts before. The scenario at that time may have been somewhat unique as we had our own client-side, used Tez AM and our custom Worker. We have completed the Kubernetes adaption of Tez AM and our custom Worker, Tez AM utilizes the K8S default scheduler for resource allocation and manages the startup and shutdown of the custom Worker. We have used it in production and served a significant number of batch processing jobs.


However, this differs significantly from the community's scenario, so we will carefully consider how to contribute back to the community.


Best regards,
Yifeng Lu


At 2024-01-29 16:52:50, "László Bodor" <bo...@gmail.com> wrote:
>Hi Yifeng!
>
>I can tell that the most spectacular step toward tez on Kubernetes was the
>unmanaged sessions umbrella: https://issues.apache.org/jira/browse/TEZ-3991
>I can see some of those tasks are still not committed, however, unmanaged
>sessions have been working for years in a Cloudera product, so we would be
>happy to review these if someone picked them up and created a PR.
>
>Once unmanaged sessions are done, we can explore other ways. Tez is still
>tightly coupled with Yarn I believe, so on the road to k8s, we have to
>explore the following:
>1. *Tez AM* to be started by another resource manager than Yarn (which is
>slightly different from the unmanaged sessions above, as unmanaged sessions
>start independently of the client (e.g. Hive's hiveserver2)) <- this is not
>a must-have as long as the unmanaged sessions approach is fine for the K8s
>use-cases.
>2. *Tez containers: t*his is a huge gap: the product I know works by Hive
>LLAP as executors, so similarly to the sessions, the Hive LLAP containers
>start independently of Tez, so if we want to make this work in Tez out of
>the box, we need to implement new things like start a TezChild container or
>rather a group of TezChild containers as a k8s pod for instance, just
>thinking aloud.
>
>Thanks for coming up with this topic. Let us get back to this later if any
>plans, in the meantime, finishing the upstream contribution of TEZ-3991 is
>more than welcome.
>
>Regards,
>Laszlo Bodor
>
>
>YifengLu <no...@163.com> ezt írta (időpont: 2024. jan. 29., H, 3:52):
>
>> Thank you very much for your response. We will continue to explore Tez on
>> Kubernetes, and if there are any further developments, we will reach out to
>> the community. Once again, thank you.
>> Best regards,
>> Yifeng Lu

Re: Re: tez on k8s

Posted by László Bodor <bo...@gmail.com>.

Hi Yifeng!

I can tell that the most spectacular step toward tez on Kubernetes was the
unmanaged sessions umbrella: https://issues.apache.org/jira/browse/TEZ-3991
I can see some of those tasks are still not committed, however, unmanaged
sessions have been working for years in a Cloudera product, so we would be
happy to review these if someone picked them up and created a PR.

Once unmanaged sessions are done, we can explore other ways. Tez is still
tightly coupled with Yarn I believe, so on the road to k8s, we have to
explore the following:
1. *Tez AM* to be started by another resource manager than Yarn (which is
slightly different from the unmanaged sessions above, as unmanaged sessions
start independently of the client (e.g. Hive's hiveserver2)) <- this is not
a must-have as long as the unmanaged sessions approach is fine for the K8s
use-cases.
2. *Tez containers: t*his is a huge gap: the product I know works by Hive
LLAP as executors, so similarly to the sessions, the Hive LLAP containers
start independently of Tez, so if we want to make this work in Tez out of
the box, we need to implement new things like start a TezChild container or
rather a group of TezChild containers as a k8s pod for instance, just
thinking aloud.

Thanks for coming up with this topic. Let us get back to this later if any
plans, in the meantime, finishing the upstream contribution of TEZ-3991 is
more than welcome.

Regards,
Laszlo Bodor


YifengLu <no...@163.com> ezt írta (időpont: 2024. jan. 29., H, 3:52):

> Thank you very much for your response. We will continue to explore Tez on
> Kubernetes, and if there are any further developments, we will reach out to
> the community. Once again, thank you.
> Best regards,
> Yifeng Lu

Re:Re: tez on k8s

Posted by YifengLu <no...@163.com>.

Thank you very much for your response. We will continue to explore Tez on Kubernetes, and if there are any further developments, we will reach out to the community. Once again, thank you.
Best regards,
Yifeng Lu

Re: tez on k8s

Posted by Okumin <ma...@okumin.com>.

Hi Yifeng,

I am not sure yet if it is practically feasible but let me make a
reply just to express my interest in this topic. It sounds nice if we
could give a choice of Hive or Tez on k8s.

Regards,
Okumin

On Wed, Jan 24, 2024 at 2:50 PM YifengLu <no...@163.com> wrote:
>
> Hi, Tez Team：
> We are from the bytedance EMR team，and we would like to inquire about the plans for Tez on Kubernetes (K8S) and express our interest in Tez on K8S. We hope to have the opportunity to collaborate with you on this project.
> We understand that Tez is a powerful distributed computing framework, while K8S is a popular container orchestration platform. We believe that deploying Tez on K8S can provide better scalability and manageability. We are very excited about the potential of this project and would like to be involved.
> If you have any plans or ideas regarding Tez on K8S, we would be eager to listen and provide assistance. We have experience with K8S and can support the development and deployment of the project. Additionally, we would like to discuss how we can better integrate Tez with K8S to meet the needs of users.
> We believe that through joint efforts, we can make Tez on K8S an excellent open-source project and bring greater value to the community.
> Finally, thank you very much for taking the time to read this email. If you have any questions or would like to further discuss, please feel free to contact me. Looking forward to your response.
>
>
> Best regards,
>
>
> Yifeng Lu