You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Ashwin Sai Shankar <as...@slack-corp.com.INVALID> on 2019/04/09 00:56:02 UTC
Difference between Kubernetes Executor vs PodOperator
Hi Airflow users,
What is the difference between Kube executor vs pod operator?
http://airflow.apache.org/kubernetes.html
Thanks,
Ash
Re: Difference between Kubernetes Executor vs PodOperator
Posted by James Meickle <jm...@quantopian.com.INVALID>.
We use a fork of the KubernetesPodOperator in house (and we don't use the
Executor). The steps to use it are roughly:
1) Set up your Kubernetes cluster (can be EKS, GKE, etc.)
2) Make sure your Airflow cluster has network access to the Kubernetes API
servers
3) Create a ServiceAccount for the Airflow user, and if using RBAC, grant
it access to whatever you'll be creating with it (for instance, we run all
Airflow pods within an Airflow-specific namespace)
4) Create a ~/.kube/config file to hold your Kubernetes credentials and
define which clusters are available:
https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/#define-clusters-users-and-contexts
5) Create a DAG that imports the operator and provides necessary arguments.
It will use the kubectl config file's credentials when submitting to the
cluster.
You may also need a pipeline for building the images that the operator uses.
On Wed, Apr 10, 2019 at 2:41 AM Ashwin Sai Shankar
<as...@slack-corp.com.invalid> wrote:
> Thanks, James and Kamil! Please let me know if you have any examples of
> setting up Kubernetes Executor and Operator.
>
> On Tue, Apr 9, 2019 at 8:03 AM James Meickle
> <jm...@quantopian.com.invalid> wrote:
>
> > Yes, that summary is correct - the Executor is using Kubernetes to
> execute
> > all Airflow tasks (each wrapped by a temporary Airflow process), while
> the
> > PodOperator is using Kubernetes only for that task, to execute one Pod
> > (which likely won't run any Airflow code at all).
> >
> > On Tue, Apr 9, 2019 at 3:17 AM Kamil Gałuszka <ka...@flyrlabs.com>
> wrote:
> >
> > > Hi Ashwin,
> > >
> > > I had exactly same question couple of days ago. Let me try to explain.
> If
> > > I'm wrong please someone correct me.
> > >
> > > Kubernetes Executor is used to execute TaskInstance, which means that
> Pod
> > > is created of that TaskInstance that for ex. could be BashOperator or
> > > SlackAPIOperator and after execution of this Task is finished, executor
> > pod
> > > is cleaned up/removed.
> > >
> > > If you use KubernetesPodOperator, then there could be used whatever
> > > language/library from docker image that will be scheduled by airflow.
> For
> > > example You parse CSV files with XSV library in Rust, then
> > > KubernetesPodOperator is perfect for You as you don't have to spend to
> > much
> > > time figuring out how to write custom operator, You just have an image
> > with
> > > executable that is simply used with KubernetesPodOperator.
> > >
> > > If You use KubernetesPodOperator with KubernetesExecutor then following
> > > thing will happen:
> > >
> > > AirflowScheduler -> will create KubernetesExecutor Pod in k8s -> which
> > will
> > > create KubernetesPodOperator Pod in k8s.
> > >
> > > KubernetesPodOperator can be used with any other executors to my
> > knowledge.
> > >
> > > Thanks
> > > Kamil
> > >
> > > On Tue, Apr 9, 2019 at 2:56 AM Ashwin Sai Shankar
> > > <as...@slack-corp.com.invalid> wrote:
> > >
> > > > Hi Airflow users,
> > > > What is the difference between Kube executor vs pod operator?
> > > > http://airflow.apache.org/kubernetes.html
> > > >
> > > > Thanks,
> > > > Ash
> > > >
> > >
> >
>
Re: Difference between Kubernetes Executor vs PodOperator
Posted by Ashwin Sai Shankar <as...@slack-corp.com.INVALID>.
Thanks, James and Kamil! Please let me know if you have any examples of
setting up Kubernetes Executor and Operator.
On Tue, Apr 9, 2019 at 8:03 AM James Meickle
<jm...@quantopian.com.invalid> wrote:
> Yes, that summary is correct - the Executor is using Kubernetes to execute
> all Airflow tasks (each wrapped by a temporary Airflow process), while the
> PodOperator is using Kubernetes only for that task, to execute one Pod
> (which likely won't run any Airflow code at all).
>
> On Tue, Apr 9, 2019 at 3:17 AM Kamil Gałuszka <ka...@flyrlabs.com> wrote:
>
> > Hi Ashwin,
> >
> > I had exactly same question couple of days ago. Let me try to explain. If
> > I'm wrong please someone correct me.
> >
> > Kubernetes Executor is used to execute TaskInstance, which means that Pod
> > is created of that TaskInstance that for ex. could be BashOperator or
> > SlackAPIOperator and after execution of this Task is finished, executor
> pod
> > is cleaned up/removed.
> >
> > If you use KubernetesPodOperator, then there could be used whatever
> > language/library from docker image that will be scheduled by airflow. For
> > example You parse CSV files with XSV library in Rust, then
> > KubernetesPodOperator is perfect for You as you don't have to spend to
> much
> > time figuring out how to write custom operator, You just have an image
> with
> > executable that is simply used with KubernetesPodOperator.
> >
> > If You use KubernetesPodOperator with KubernetesExecutor then following
> > thing will happen:
> >
> > AirflowScheduler -> will create KubernetesExecutor Pod in k8s -> which
> will
> > create KubernetesPodOperator Pod in k8s.
> >
> > KubernetesPodOperator can be used with any other executors to my
> knowledge.
> >
> > Thanks
> > Kamil
> >
> > On Tue, Apr 9, 2019 at 2:56 AM Ashwin Sai Shankar
> > <as...@slack-corp.com.invalid> wrote:
> >
> > > Hi Airflow users,
> > > What is the difference between Kube executor vs pod operator?
> > > http://airflow.apache.org/kubernetes.html
> > >
> > > Thanks,
> > > Ash
> > >
> >
>
Re: Difference between Kubernetes Executor vs PodOperator
Posted by James Meickle <jm...@quantopian.com.INVALID>.
Yes, that summary is correct - the Executor is using Kubernetes to execute
all Airflow tasks (each wrapped by a temporary Airflow process), while the
PodOperator is using Kubernetes only for that task, to execute one Pod
(which likely won't run any Airflow code at all).
On Tue, Apr 9, 2019 at 3:17 AM Kamil Gałuszka <ka...@flyrlabs.com> wrote:
> Hi Ashwin,
>
> I had exactly same question couple of days ago. Let me try to explain. If
> I'm wrong please someone correct me.
>
> Kubernetes Executor is used to execute TaskInstance, which means that Pod
> is created of that TaskInstance that for ex. could be BashOperator or
> SlackAPIOperator and after execution of this Task is finished, executor pod
> is cleaned up/removed.
>
> If you use KubernetesPodOperator, then there could be used whatever
> language/library from docker image that will be scheduled by airflow. For
> example You parse CSV files with XSV library in Rust, then
> KubernetesPodOperator is perfect for You as you don't have to spend to much
> time figuring out how to write custom operator, You just have an image with
> executable that is simply used with KubernetesPodOperator.
>
> If You use KubernetesPodOperator with KubernetesExecutor then following
> thing will happen:
>
> AirflowScheduler -> will create KubernetesExecutor Pod in k8s -> which will
> create KubernetesPodOperator Pod in k8s.
>
> KubernetesPodOperator can be used with any other executors to my knowledge.
>
> Thanks
> Kamil
>
> On Tue, Apr 9, 2019 at 2:56 AM Ashwin Sai Shankar
> <as...@slack-corp.com.invalid> wrote:
>
> > Hi Airflow users,
> > What is the difference between Kube executor vs pod operator?
> > http://airflow.apache.org/kubernetes.html
> >
> > Thanks,
> > Ash
> >
>
Re: Difference between Kubernetes Executor vs PodOperator
Posted by Kamil Gałuszka <ka...@flyrlabs.com>.
Hi Ashwin,
I had exactly same question couple of days ago. Let me try to explain. If
I'm wrong please someone correct me.
Kubernetes Executor is used to execute TaskInstance, which means that Pod
is created of that TaskInstance that for ex. could be BashOperator or
SlackAPIOperator and after execution of this Task is finished, executor pod
is cleaned up/removed.
If you use KubernetesPodOperator, then there could be used whatever
language/library from docker image that will be scheduled by airflow. For
example You parse CSV files with XSV library in Rust, then
KubernetesPodOperator is perfect for You as you don't have to spend to much
time figuring out how to write custom operator, You just have an image with
executable that is simply used with KubernetesPodOperator.
If You use KubernetesPodOperator with KubernetesExecutor then following
thing will happen:
AirflowScheduler -> will create KubernetesExecutor Pod in k8s -> which will
create KubernetesPodOperator Pod in k8s.
KubernetesPodOperator can be used with any other executors to my knowledge.
Thanks
Kamil
On Tue, Apr 9, 2019 at 2:56 AM Ashwin Sai Shankar
<as...@slack-corp.com.invalid> wrote:
> Hi Airflow users,
> What is the difference between Kube executor vs pod operator?
> http://airflow.apache.org/kubernetes.html
>
> Thanks,
> Ash
>