You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by "Driesprong, Fokko" <fo...@driesprong.frl> on 2018/08/21 14:14:53 UTC
Re: Regarding airflow 1.10
Hi Hemanth,
Thanks for the question. There is the KubernetesExecutor which just uses a
container with Airflow to schedule a task on the kubernetes cluster. This
container will then execute a single task, and then the Executor checks the
exitcode of the container.
There is also the KubernetesOperator which allows you to kick of a new
container on the kubernetes cluster. Hope this helps.
Cheers, Fokko
Op di 21 aug. 2018 om 15:54 schreef Musunuru, Hemanth <
Hemanth.Musunuru@nike.com>
> Hi Team,
>
> Does Kubernetes executor in 1.10 create new pod for every task ? (Which
> means won’t there be any active workers running except for new Tasks)
>
> We are trying to containerize our existing to airflow environment, your
> reply Will help us to understand it more.
>
> Thanks in Advance .
>
> Thanks
> Hemanth
Re: Regarding airflow 1.10
Posted by Daniel Imberman <da...@gmail.com>.
Hi Hemanth,
Fokko and Taylor have covered this question but just cc'ing myself in case
you have any further questions about the k8s executor :).
On Wed, Aug 22, 2018 at 12:10 PM Taylor Edmiston <te...@gmail.com>
wrote:
> Hemanth -
>
> To add to what Fokko mentioned...
>
> You can find the KubernetesExecutor implementation in KubernetesExecutor
> <
> https://github.com/apache/incubator-airflow/blob/7ea7dd2031c32a5dd14e4fd74f415f938a69e3ad/airflow/contrib/executors/kubernetes_executor.py#L465
> >
> and AirflowKubernetesScheduler
> <
> https://github.com/apache/incubator-airflow/blob/7ea7dd2031c32a5dd14e4fd74f415f938a69e3ad/airflow/contrib/executors/kubernetes_executor.py#L292
> >.
> There's also a usage example
> <
> https://github.com/apache/incubator-airflow/blob/272952a9dce932cb2c648f82c9f9f2cafd572ff1/airflow/example_dags/example_kubernetes_executor.py
> >
> .
>
> And here's the KubernetesPodOperator
> <
> https://github.com/apache/incubator-airflow/blob/9d516c7134eb22a3c2fc63cf96626ef6e8b247f2/airflow/contrib/operators/kubernetes_pod_operator.py#L33
> >
> implementation
> which has an example
> <
> https://github.com/apache/incubator-airflow/blob/9d516c7134eb22a3c2fc63cf96626ef6e8b247f2/airflow/example_dags/example_kubernetes_operator.py
> >
> as
> well.
>
> There are brief docs
> <https://airflow.readthedocs.io/en/latest/kubernetes.html> available for
> both as well.
>
> And a shameless plug — we've been building an open source Airflow on
> Kubernetes stack at Astronomer (Helm charts
> <https://github.com/astronomerio/helm.astronomer.io>, Docker images
> <https://github.com/astronomerio/astronomer>). Some people use this to
> avoid having to rebuild all of the containerization. Currently we use
> CeleryExecutor but are already migrating toward the KubernetesExecutor.
> Our goal is to provide the best experience for running Airflow on
> Kubernetes. Feel free to message me off list if this is interesting.
>
> Best,
> Taylor
>
> *Taylor Edmiston*
> Blog <https://blog.tedmiston.com/> | CV
> <https://stackoverflow.com/cv/taylor> | LinkedIn
> <https://www.linkedin.com/in/tedmiston/> | AngelList
> <https://angel.co/taylor> | Stack Overflow
> <https://stackoverflow.com/users/149428/taylor-edmiston>
>
>
>
> On Tue, Aug 21, 2018 at 10:15 AM Driesprong, Fokko <fo...@driesprong.frl>
> wrote:
>
> > Hi Hemanth,
> >
> > Thanks for the question. There is the KubernetesExecutor which just uses
> a
> > container with Airflow to schedule a task on the kubernetes cluster. This
> > container will then execute a single task, and then the Executor checks
> the
> > exitcode of the container.
> >
> > There is also the KubernetesOperator which allows you to kick of a new
> > container on the kubernetes cluster. Hope this helps.
> >
> > Cheers, Fokko
> >
> > Op di 21 aug. 2018 om 15:54 schreef Musunuru, Hemanth <
> > Hemanth.Musunuru@nike.com>
> >
> > > Hi Team,
> > >
> > > Does Kubernetes executor in 1.10 create new pod for every task ?
> (Which
> > > means won’t there be any active workers running except for new Tasks)
> > >
> > > We are trying to containerize our existing to airflow environment, your
> > > reply Will help us to understand it more.
> > >
> > > Thanks in Advance .
> > >
> > > Thanks
> > > Hemanth
> >
>
Re: Regarding airflow 1.10
Posted by Taylor Edmiston <te...@gmail.com>.
Hemanth -
To add to what Fokko mentioned...
You can find the KubernetesExecutor implementation in KubernetesExecutor
<https://github.com/apache/incubator-airflow/blob/7ea7dd2031c32a5dd14e4fd74f415f938a69e3ad/airflow/contrib/executors/kubernetes_executor.py#L465>
and AirflowKubernetesScheduler
<https://github.com/apache/incubator-airflow/blob/7ea7dd2031c32a5dd14e4fd74f415f938a69e3ad/airflow/contrib/executors/kubernetes_executor.py#L292>.
There's also a usage example
<https://github.com/apache/incubator-airflow/blob/272952a9dce932cb2c648f82c9f9f2cafd572ff1/airflow/example_dags/example_kubernetes_executor.py>
.
And here's the KubernetesPodOperator
<https://github.com/apache/incubator-airflow/blob/9d516c7134eb22a3c2fc63cf96626ef6e8b247f2/airflow/contrib/operators/kubernetes_pod_operator.py#L33>
implementation
which has an example
<https://github.com/apache/incubator-airflow/blob/9d516c7134eb22a3c2fc63cf96626ef6e8b247f2/airflow/example_dags/example_kubernetes_operator.py>
as
well.
There are brief docs
<https://airflow.readthedocs.io/en/latest/kubernetes.html> available for
both as well.
And a shameless plug — we've been building an open source Airflow on
Kubernetes stack at Astronomer (Helm charts
<https://github.com/astronomerio/helm.astronomer.io>, Docker images
<https://github.com/astronomerio/astronomer>). Some people use this to
avoid having to rebuild all of the containerization. Currently we use
CeleryExecutor but are already migrating toward the KubernetesExecutor.
Our goal is to provide the best experience for running Airflow on
Kubernetes. Feel free to message me off list if this is interesting.
Best,
Taylor
*Taylor Edmiston*
Blog <https://blog.tedmiston.com/> | CV
<https://stackoverflow.com/cv/taylor> | LinkedIn
<https://www.linkedin.com/in/tedmiston/> | AngelList
<https://angel.co/taylor> | Stack Overflow
<https://stackoverflow.com/users/149428/taylor-edmiston>
On Tue, Aug 21, 2018 at 10:15 AM Driesprong, Fokko <fo...@driesprong.frl>
wrote:
> Hi Hemanth,
>
> Thanks for the question. There is the KubernetesExecutor which just uses a
> container with Airflow to schedule a task on the kubernetes cluster. This
> container will then execute a single task, and then the Executor checks the
> exitcode of the container.
>
> There is also the KubernetesOperator which allows you to kick of a new
> container on the kubernetes cluster. Hope this helps.
>
> Cheers, Fokko
>
> Op di 21 aug. 2018 om 15:54 schreef Musunuru, Hemanth <
> Hemanth.Musunuru@nike.com>
>
> > Hi Team,
> >
> > Does Kubernetes executor in 1.10 create new pod for every task ? (Which
> > means won’t there be any active workers running except for new Tasks)
> >
> > We are trying to containerize our existing to airflow environment, your
> > reply Will help us to understand it more.
> >
> > Thanks in Advance .
> >
> > Thanks
> > Hemanth
>