You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/02/12 14:58:39 UTC

[GitHub] [airflow] dimberman opened a new pull request #7406: Improve k8s executor documentation

dimberman opened a new pull request #7406: Improve k8s executor documentation
URL: https://github.com/apache/airflow/pull/7406
 
 
   ---
   Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [ ] Description above provides context of the change
   - [ ] Commit message/PR title starts with `[AIRFLOW-NNNN]`. AIRFLOW-NNNN = JIRA ID<sup>*</sup>
   - [ ] Unit tests coverage for changes (not needed for documentation changes)
   - [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
   - [ ] Relevant documentation is updated including usage instructions.
   - [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   <sup>*</sup> For document-only changes commit message can start with `[AIRFLOW-XXXX]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #7406: [AIRFLOW-XXXX] Add architecture section to k8sexec docs

Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #7406: [AIRFLOW-XXXX] Add architecture section to k8sexec docs
URL: https://github.com/apache/airflow/pull/7406#discussion_r382744565
 
 

 ##########
 File path: docs/executor/kubernetes.rst
 ##########
 @@ -34,3 +34,71 @@ The volumes are optional and depend on your configuration. There are two volumes
   - By storing logs onto a persistent disk, the files are accessible by workers and the webserver. If you don't configure this, the logs will be lost after the worker pods shuts down
 
   - Another option is to use S3/GCS/etc to store logs
+
+KubernetesExecutor Architecture
+################################
+
+The KubernetesExecutor runs as a process in the Scheduler that only requires access to the Kubernetes API (it does *not* need to run inside of a Kubernetes cluster). The KubernetesExecutor requires a non-sqlite database in the backend, but there are no external brokers or persistent workers needed.
+For these reasons, we recommend the KubernetesExecutor for deployments have long periods of dormancy between DAG execution.
+
+
+.. image:: ../img/k8s-0-worker.jpeg
+
+
+When a DAG submits a task, the KubernetesExecutor requests a worker pod from the Kubernetes API. The worker pod then runs the task, reports the result, and terminates.
+
+
+
+.. image:: ../img/k8s-3-worker.jpeg
+
+.. @startuml
+.. Airflow_Scheduler -> Kubernetes: Request a new pod with command "airflow run..."
+.. Kubernetes -> Airflow_Worker: Create Airflow worker with command "airflow run..."
+.. Airflow_Worker -> Airflow_DB: Report task passing or failure to DB
+.. Airflow_Worker -> Kubernetes: Pod completes with state "Succeeded" and k8s records in ETCD
+.. Kubernetes -> Airflow_Scheduler: Airflow scheduler reads "Succeeded" from k8s watcher thread
+.. @enduml
+.. image:: ../img/k8s-happy-path.png
+
+
+***************
+Fault Tolerance
+***************
+
+===========================
+Handling Worker Pod Crashes
+===========================
+
+When dealing with distributed systems, we need a system that assumes that any component can crash at any moment for reasons ranging from OOM errors to node upgrades.
+
+In the case where a worker dies before it can report its status to the backend DB, the executor can use a Kubernetes watcher thread to discover the failed pod.
+
+.. image:: ../img/k8s-watcher-2.jpeg
 
 Review comment:
   We don't have this image !

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] dimberman commented on a change in pull request #7406: [AIRFLOW-XXXX] Add architecture section to k8sexec docs

Posted by GitBox <gi...@apache.org>.
dimberman commented on a change in pull request #7406: [AIRFLOW-XXXX] Add architecture section to k8sexec docs
URL: https://github.com/apache/airflow/pull/7406#discussion_r378420805
 
 

 ##########
 File path: docs/executor/kubernetes.rst
 ##########
 @@ -34,3 +34,71 @@ The volumes are optional and depend on your configuration. There are two volumes
   - By storing logs onto a persistent disk, the files are accessible by workers and the webserver. If you don't configure this, the logs will be lost after the worker pods shuts down
 
   - Another option is to use S3/GCS/etc to store logs
+
+KubernetesExecutor Architecture
+################################
+
+The KubernetesExecutor runs as a single pod that only requires access to the Kubernetes API. There are no external brokers or consistent workers needed.
 
 Review comment:
   These are all good points. Will fix now.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #7406: [AIRFLOW-XXXX] Add architecture section to k8sexec docs

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #7406: [AIRFLOW-XXXX] Add architecture section to k8sexec docs
URL: https://github.com/apache/airflow/pull/7406#discussion_r378420245
 
 

 ##########
 File path: docs/executor/kubernetes.rst
 ##########
 @@ -34,3 +34,71 @@ The volumes are optional and depend on your configuration. There are two volumes
   - By storing logs onto a persistent disk, the files are accessible by workers and the webserver. If you don't configure this, the logs will be lost after the worker pods shuts down
 
   - Another option is to use S3/GCS/etc to store logs
+
+KubernetesExecutor Architecture
+################################
+
+The KubernetesExecutor runs as a single pod that only requires access to the Kubernetes API. There are no external brokers or consistent workers needed.
 
 Review comment:
   It is possible to run KubeExecutor from outside of a cluster too. Perhaps we can ignore that here though.
   
   Also the KubeExecutor doesn't run as a pod, but runs inside the scheduler.
   
   No external brokers, but still need the (existing) metadata DB, which can't be sqlite (let's be explicit.)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on issue #7406: [AIRFLOW-XXXX] Add architecture section to k8sexec docs

Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #7406: [AIRFLOW-XXXX] Add architecture section to k8sexec docs
URL: https://github.com/apache/airflow/pull/7406#issuecomment-585283178
 
 
   Can you add some labels to the diagram, please? The diagram uses the Airflow logo to describe many different components.  It is not clear what the role of each logo is. Lines appear in the diagram, but they are not described. I think that it is also worth numbering the next steps in the diagram, which allows for easier tracking of the message. If possible, it is worth using generally accepted conventions, e.g. sequence diagram. This makes the drawings simpler to understand and easier to draw.  You could use online editor e.g. https://www.planttext.com/  
   Example: https://github.com/apache/airflow/pull/6961/files 
   Old preview: https://user-images.githubusercontent.com/12058428/71550253-ebd62000-29cb-11ea-827f-d6d66cf25256.png
   What do you think about Did you have any reason to draw this diagram in a graphics program?  I am afraid of such diagram, because then they are difficult to edit.
   
   
   I have one more comment. Can you maintain uniform naming between text and diagram also, please? You use the term "the backend DB" in the text, but it does not appear in the image, but instead is "PostgresSQL". 
   
   I also prepared preview of your documentation:
   Preview:  https://acceptable-goat.surge.sh/executor/kubernetes.html
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil merged pull request #7406: [AIRFLOW-XXXX] Add architecture section to k8sexec docs

Posted by GitBox <gi...@apache.org>.
kaxil merged pull request #7406: [AIRFLOW-XXXX] Add architecture section to k8sexec docs
URL: https://github.com/apache/airflow/pull/7406
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services