You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@liminal.apache.org by GitBox <gi...@apache.org> on 2020/12/09 18:55:40 UTC
[GitHub] [incubator-liminal] naturalett opened a new pull request #11: liminal installation doc
naturalett opened a new pull request #11:
URL: https://github.com/apache/incubator-liminal/pull/11
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] naturalett commented on a change in pull request #11: liminal installation doc
Posted by GitBox <gi...@apache.org>.
naturalett commented on a change in pull request #11:
URL: https://github.com/apache/incubator-liminal/pull/11#discussion_r604239387
##########
File path: docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md
##########
@@ -0,0 +1,160 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required bgit y applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Install Liminal in Airflow
+* [Workflow](#workflow)
+* [Prerequisites](#prerequisites)
+ * [Supported Distributions](#supported-distributions)
+* [Get Started](#Get-Started)
+ * [Scripted Installation](#scripted-installation)
+ * [Manual Installation](#manual-installation)
+* [References and other resources](#references-and-other-resources)
+
+## Workflow
+Liminal deploys into the Airflow components and the Yamls home folder is mounted by EFS.
+The process of liminal is emphasized on the following diagram:
+
+
+
+## Prerequisites
+Before you begin, ensure you have met the following requirements:
+
+* You have a Kubernetes cluster running with [Helm][homebrew-helm] (and Tiller if using Helm v2.x) installed
+* You have the kubectl command line [(kubectl CLI)][homebrew-kubectl] installed
+* You have the current [context][cluster-access-kubeconfig] in kubernetes' kubeconfig file
+* You have [Airflow on Kubernetes with AWS EFS][airflowInstallation]
Review comment:
I guess it will be the best to do so
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] naturalett commented on a change in pull request #11: liminal installation doc
Posted by GitBox <gi...@apache.org>.
naturalett commented on a change in pull request #11:
URL: https://github.com/apache/incubator-liminal/pull/11#discussion_r604238128
##########
File path: docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md
##########
@@ -0,0 +1,122 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required bgit y applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Install Liminal in Airflow
+* [Prerequisites](#prerequisites)
+ * [Supported Distributions](#supported-distributions)
+* [Get Started](#Get-Started)
+ * [Scripted Installation](#scripted-installation)
+ * [Manual Installation](#manual-installation)
+* [References and other resources](#references-and-other-resources)
+
+## Prerequisites
+Before you begin, ensure you have met the following requirements:
+
+* You have a Kubernetes cluster running with [Helm][homebrew-helm] (and Tiller if using Helm v2.x) installed
+* You have the kubectl command line [(kubectl CLI)][homebrew-kubectl] installed
+* You have the current [context][cluster-access-kubeconfig] in kubernetes' kubeconfig file
+* You have Airflow on Kubernetes with AWS EFS
+* You have already created [custom DAGs][custom-dag] following the liminal [Getting Started Documentation][liminalGetStarted-doc]
+* Make sure that the [example repository][liminal-getting-started-project] is your workspace
+
+### Supported Distributions
+
+|Distribution | Versions |
+|-|-|
+|[Airflow][airflowImage] | apache/airflow:1.10.12-python3.6 |
+|[Airflow Helm Chart][airflowChart] | 7.14.3 |
+
+#### You may need to add the Repo
+
+```sh
+helm repo add airflow-stable https://airflow-helm.github.io/charts
+helm repo update
+```
+
+## Get Started
+
+### Scripted Installation
+
+[Use `install_liminal_in_airflow_on_kubernetes.sh`][liminal-installation-script] to install liminal in Airflow.
+* Move to the [example repository project][liminal-getting-started-project]
+* Copy the [script][liminal-installation-script] to the [example repository project][liminal-getting-started-project] and run it
+
+
+
+### Manual Installation
+
+Here are a couple of steps of how to install liminal in Airflow on Kubernetes:
+* [Find Airflow components](#Find-Airflow-pod-names)
+* [Install liminal in the components](#Install-liminal-in-Airflow-pods)
+* [Setting up liminal](#Setting-up-liminal)
+
+#### Find Airflow pod names:
+
+```sh
+echo -n "Please enter a namespace: "
+read namespace
+components=("web" "scheduler" "worker")
+namespace=${namespace}
+podNames=()
+for component in ${components[@]}
+do
+ podNames+="$(kubectl get pod -n ${namespace} -l "app=airflow,component=${component}" --no-headers -o custom-columns=":metadata.name")\n"
+
+done
+
+echo -e "The following Airflow pods are:\n${podNames}"
+```
+#### Install liminal in Airflow pods:
Review comment:
No.
I will change it to have a requirements txt in the DAGs folder. It will promise that Airflow will install it on every restart.
In the supported distributions I assume that you deploy Airflow Helm Chart 7.16.0.
There are other ways to install python packages. (init containers and etc.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] naturalett commented on a change in pull request #11: liminal installation doc
Posted by GitBox <gi...@apache.org>.
naturalett commented on a change in pull request #11:
URL: https://github.com/apache/incubator-liminal/pull/11#discussion_r594713637
##########
File path: docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md
##########
@@ -0,0 +1,122 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required bgit y applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Install Liminal in Airflow
+* [Prerequisites](#prerequisites)
+ * [Supported Distributions](#supported-distributions)
+* [Get Started](#Get-Started)
+ * [Scripted Installation](#scripted-installation)
+ * [Manual Installation](#manual-installation)
+* [References and other resources](#references-and-other-resources)
+
+## Prerequisites
+Before you begin, ensure you have met the following requirements:
+
+* You have a Kubernetes cluster running with [Helm][homebrew-helm] (and Tiller if using Helm v2.x) installed
+* You have the kubectl command line [(kubectl CLI)][homebrew-kubectl] installed
+* You have the current [context][cluster-access-kubeconfig] in kubernetes' kubeconfig file
+* You have Airflow on Kubernetes with AWS EFS
+* You have already created [custom DAGs][custom-dag] following the liminal [Getting Started Documentation][liminalGetStarted-doc]
+* Make sure that the [example repository][liminal-getting-started-project] is your workspace
+
+### Supported Distributions
+
+|Distribution | Versions |
+|-|-|
+|[Airflow][airflowImage] | apache/airflow:1.10.12-python3.6 |
+|[Airflow Helm Chart][airflowChart] | 7.14.3 |
+
+#### You may need to add the Repo
+
+```sh
+helm repo add airflow-stable https://airflow-helm.github.io/charts
+helm repo update
+```
+
+## Get Started
+
+### Scripted Installation
+
+[Use `install_liminal_in_airflow_on_kubernetes.sh`][liminal-installation-script] to install liminal in Airflow.
+* Move to the [example repository project][liminal-getting-started-project]
+* Copy the [script][liminal-installation-script] to the [example repository project][liminal-getting-started-project] and run it
+
+
+
+### Manual Installation
+
+Here are a couple of steps of how to install liminal in Airflow on Kubernetes:
+* [Find Airflow components](#Find-Airflow-pod-names)
+* [Install liminal in the components](#Install-liminal-in-Airflow-pods)
+* [Setting up liminal](#Setting-up-liminal)
+
+#### Find Airflow pod names:
+
+```sh
+echo -n "Please enter a namespace: "
+read namespace
+components=("web" "scheduler" "worker")
+namespace=${namespace}
+podNames=()
+for component in ${components[@]}
+do
+ podNames+="$(kubectl get pod -n ${namespace} -l "app=airflow,component=${component}" --no-headers -o custom-columns=":metadata.name")\n"
+
+done
+
+echo -e "The following Airflow pods are:\n${podNames}"
+```
+#### Install liminal in Airflow pods:
Review comment:
No. I would launch Airflow with a requirements.txt in the mounted DAGs folder which defines the packages to install and this is how I can assure that the packages will be installed once Airflow will get restarted as well.
However, here we have a running Airflow that we want to install a liminal on.
We assume the client has its own running Airflow with EFS based on the **Prerequisites**
WDYT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] naturalett commented on a change in pull request #11: liminal installation doc
Posted by GitBox <gi...@apache.org>.
naturalett commented on a change in pull request #11:
URL: https://github.com/apache/incubator-liminal/pull/11#discussion_r604240434
##########
File path: docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md
##########
@@ -0,0 +1,160 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required bgit y applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Install Liminal in Airflow
+* [Workflow](#workflow)
+* [Prerequisites](#prerequisites)
+ * [Supported Distributions](#supported-distributions)
+* [Get Started](#Get-Started)
+ * [Scripted Installation](#scripted-installation)
+ * [Manual Installation](#manual-installation)
+* [References and other resources](#references-and-other-resources)
+
+## Workflow
+Liminal deploys into the Airflow components and the Yamls home folder is mounted by EFS.
Review comment:
Ok. I will this description. Thx.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] naturalett commented on a change in pull request #11: liminal installation doc
Posted by GitBox <gi...@apache.org>.
naturalett commented on a change in pull request #11:
URL: https://github.com/apache/incubator-liminal/pull/11#discussion_r604267380
##########
File path: docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md
##########
@@ -0,0 +1,160 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required bgit y applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Install Liminal in Airflow
+* [Workflow](#workflow)
+* [Prerequisites](#prerequisites)
+ * [Supported Distributions](#supported-distributions)
+* [Get Started](#Get-Started)
+ * [Scripted Installation](#scripted-installation)
+ * [Manual Installation](#manual-installation)
+* [References and other resources](#references-and-other-resources)
+
+## Workflow
+Liminal deploys into the Airflow components and the Yamls home folder is mounted by EFS.
+The process of liminal is emphasized on the following diagram:
+
+
+
+## Prerequisites
+Before you begin, ensure you have met the following requirements:
+
+* You have a Kubernetes cluster running with [Helm][homebrew-helm] (and Tiller if using Helm v2.x) installed
+* You have the kubectl command line [(kubectl CLI)][homebrew-kubectl] installed
+* You have the current [context][cluster-access-kubeconfig] in kubernetes' kubeconfig file
+* You have [Airflow on Kubernetes with AWS EFS][airflowInstallation]
+* You have already created [Liminal Yaml file][liminal-yaml-file] following the liminal [Getting Started Documentation][liminalGetStarted-doc]
+* Make sure that the [example repository][liminal-getting-started-project] is your workspace
+
+### Supported Distributions
+
+|Distribution | Versions |
+|-|-|
+|[Airflow][airflowImage] | apache/airflow:1.10.12-python3.6 |
+|[Airflow Helm Chart][airflowChart] | 7.14.3 |
+
+#### You may need to add the Repo
+
+```sh
+helm repo add airflow-stable https://airflow-helm.github.io/charts
+helm repo update
+```
+
+## Get Started
+
+### Scripted Installation
+
+[Use `install_liminal_in_airflow_on_kubernetes.sh`][liminal-installation-script] to install liminal in Airflow.
+* Move to the [example repository project][liminal-getting-started-project]
+* Copy the [script][liminal-installation-script] to the [example repository project][liminal-getting-started-project] and run it
+
+
+
+### Manual Installation
+
+Here are a couple of steps of how to install liminal in Airflow on Kubernetes:
+* [Find Airflow components](#Find-Airflow-pod-names)
+* [Install liminal in the components](#Install-liminal-in-Airflow-pods)
+* [Setting up liminal](#Setting-up-liminal)
+
+#### Find Airflow pod names:
+
+```sh
+echo -n "Please enter a namespace: "
+read namespace
+components=("web" "scheduler" "worker")
+namespace=${namespace}
+podNames=()
+for component in ${components[@]}
+do
+ podNames+="$(kubectl get pod -n ${namespace} -l "app=airflow,component=${component}" --no-headers -o custom-columns=":metadata.name")\n"
+
+done
+
+echo -e "The following Airflow pods are:\n${podNames}"
+```
+#### Install liminal in Airflow pods:
+
+```sh
+IFS=$'\n'|''; set -f; podNames=( ${podNames} )
+webPodName=${podNames[${#podNames[@]} - 1]}
+
+for podName in `echo -e ${podNames[@]}`
+do
+ echo "Installing liminal in pod: ${podName}"
+ kubectl exec -it -n ${namespace} ${podName} -- bash -c 'pip install --user apache-liminal'
+done
+```
+
+#### Setting up liminal:
+There are a couple of ways to deploy Yamls in Airflow.
+1. Mount your Amazon EFS file system on a Linux instance:
+ ```
+ sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport <EFS_ID>:/ /opt/airflow/dags
+ ```
+2. Use gitsync function in order to sync Yamls in the Amazon EFS file system. In this case, we chose to use a kubernetes job:
Review comment:
I re-write the all doc.
I added the gitSync as a second option. I will create a video that shows about the mount and will adjust it once again.
If still the gitSync isn't much recommended so I can remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] naturalett commented on a change in pull request #11: liminal installation doc
Posted by GitBox <gi...@apache.org>.
naturalett commented on a change in pull request #11:
URL: https://github.com/apache/incubator-liminal/pull/11#discussion_r604240099
##########
File path: docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md
##########
@@ -0,0 +1,160 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required bgit y applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Install Liminal in Airflow
+* [Workflow](#workflow)
+* [Prerequisites](#prerequisites)
+ * [Supported Distributions](#supported-distributions)
+* [Get Started](#Get-Started)
+ * [Scripted Installation](#scripted-installation)
+ * [Manual Installation](#manual-installation)
+* [References and other resources](#references-and-other-resources)
+
+## Workflow
+Liminal deploys into the Airflow components and the Yamls home folder is mounted by EFS.
+The process of liminal is emphasized on the following diagram:
+
+
Review comment:
Sure thx.
Meanwhile, I created one. Let me know if it needs improvement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] naturalett commented on a change in pull request #11: liminal installation doc
Posted by GitBox <gi...@apache.org>.
naturalett commented on a change in pull request #11:
URL: https://github.com/apache/incubator-liminal/pull/11#discussion_r594714521
##########
File path: docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md
##########
@@ -0,0 +1,103 @@
+# Install Liminal in Airflow
+* [Prerequisites](#prerequisites)
+ * [Supported Distributions](#supported-distributions)
+* [Get Started](#Get-Started)
+ * [Scripted Installation](#scripted-installation)
+ * [Manual Installation](#manual-installation)
+* [References and other resources](#references-and-other-resources)
+
+## Prerequisites
+Before you begin, ensure you have met the following requirements:
+
+* You have a Kubernetes cluster running with [Helm][homebrew-helm] (and Tiller if using Helm v2.x) installed
+* You have the kubectl command line [(kubectl CLI)][homebrew-kubectl] installed
+* You have the current [context][cluster-access-kubeconfig] in kubernetes' kubeconfig file
+* You have Airflow on Kubernetes with AWS EFS
+* You have already created [custom DAGs][custom-dag] following the liminal [Getting Started Documentation][liminalGetStarted-doc]
+* Make sure that the [example repository][liminal-getting-started-project] is your workspace
+
+### Supported Distributions
+
+|Distribution | Versions |
+|-|-|
+|[Airflow][airflowImage] | apache/airflow:1.10.12-python3.6 |
+|[Airflow Helm Chart][airflowChart] | 7.14.3 |
+
+#### You may need to add the Repo
+
+```sh
+helm repo add airflow-stable https://airflow-helm.github.io/charts
+helm repo update
+```
+
+## Get Started
+
+### Scripted Installation
+
+[Use `install_liminal_in_airflow_on_kubernetes.sh`][liminal-installation-script] to install liminal in Airflow.
+* Move to the [example repository project][liminal-getting-started-project]
+* Copy the [script][liminal-installation-script] to the [example repository project][liminal-getting-started-project] and run it
+
+
+
+### Manual Installation
+
+Here are a couple of steps of how to install liminal in Airflow on Kubernetes:
+* [Find Airflow components](#Find-Airflow-pod-names)
+* [Install liminal in the components](#Install-liminal-in-Airflow-pods)
+* [Setting up liminal](Setting-up-liminal)
+
+#### Find Airflow pod names:
+
+```sh
+echo -n "Please enter a namespace: "
+read namespace
+components=("web" "scheduler" "worker")
+namespace=${namespace}
+podNames=()
+for component in ${components[@]}
+do
+ podNames+="$(kubectl get pod -n ${namespace} -l "app=airflow,component=${component}" --no-headers -o custom-columns=":metadata.name")\n"
+
+done
+
+echo -e "The following Airflow pods are:\n${podNames}"
+```
+#### Install liminal in Airflow pods:
+
+```sh
+IFS=$'\n'|''; set -f; podNames=( ${podNames} )
+webPodName=${podNames[${#podNames[@]} - 1]}
+
+for podName in `echo -e ${podNames[@]}`
+do
+ echo "Installing liminal in pod: ${podName}"
+ kubectl exec -it -n ${namespace} ${podName} -- bash -c 'pip install --user apache-liminal'
+done
+```
+
+#### Setting up liminal:
+
+```sh
+echo "Deploying liminal"
Review comment:
I emphasized both options now.
In this case, we chose to use kubectl to copy the Yamls.
We need EFS for all the components (scheduler, web, worker). We assume the user is using EFS for all three of these.
Then for the Yamls deployment we do **kubectl cp**
It is more clear to show how to copy to the mounted file system instead having a machine that mounted to the same EFS.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] assapin commented on a change in pull request #11: liminal installation doc
Posted by GitBox <gi...@apache.org>.
assapin commented on a change in pull request #11:
URL: https://github.com/apache/incubator-liminal/pull/11#discussion_r600206432
##########
File path: docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md
##########
@@ -0,0 +1,160 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required bgit y applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Install Liminal in Airflow
+* [Workflow](#workflow)
+* [Prerequisites](#prerequisites)
+ * [Supported Distributions](#supported-distributions)
+* [Get Started](#Get-Started)
+ * [Scripted Installation](#scripted-installation)
+ * [Manual Installation](#manual-installation)
+* [References and other resources](#references-and-other-resources)
+
+## Workflow
+Liminal deploys into the Airflow components and the Yamls home folder is mounted by EFS.
+The process of liminal is emphasized on the following diagram:
+
+
Review comment:
we shouldn't use this as -is, it contains many issues which are not related to the topic;
I can help you prepare a diagram with only the relevant parts
##########
File path: docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md
##########
@@ -0,0 +1,160 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required bgit y applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Install Liminal in Airflow
+* [Workflow](#workflow)
+* [Prerequisites](#prerequisites)
+ * [Supported Distributions](#supported-distributions)
+* [Get Started](#Get-Started)
+ * [Scripted Installation](#scripted-installation)
+ * [Manual Installation](#manual-installation)
+* [References and other resources](#references-and-other-resources)
+
+## Workflow
+Liminal deploys into the Airflow components and the Yamls home folder is mounted by EFS.
+The process of liminal is emphasized on the following diagram:
+
+
+
+## Prerequisites
+Before you begin, ensure you have met the following requirements:
+
+* You have a Kubernetes cluster running with [Helm][homebrew-helm] (and Tiller if using Helm v2.x) installed
+* You have the kubectl command line [(kubectl CLI)][homebrew-kubectl] installed
+* You have the current [context][cluster-access-kubeconfig] in kubernetes' kubeconfig file
+* You have [Airflow on Kubernetes with AWS EFS][airflowInstallation]
+* You have already created [Liminal Yaml file][liminal-yaml-file] following the liminal [Getting Started Documentation][liminalGetStarted-doc]
+* Make sure that the [example repository][liminal-getting-started-project] is your workspace
+
+### Supported Distributions
+
+|Distribution | Versions |
+|-|-|
+|[Airflow][airflowImage] | apache/airflow:1.10.12-python3.6 |
+|[Airflow Helm Chart][airflowChart] | 7.14.3 |
+
+#### You may need to add the Repo
+
+```sh
+helm repo add airflow-stable https://airflow-helm.github.io/charts
+helm repo update
+```
+
+## Get Started
+
+### Scripted Installation
+
+[Use `install_liminal_in_airflow_on_kubernetes.sh`][liminal-installation-script] to install liminal in Airflow.
+* Move to the [example repository project][liminal-getting-started-project]
+* Copy the [script][liminal-installation-script] to the [example repository project][liminal-getting-started-project] and run it
+
+
+
+### Manual Installation
+
+Here are a couple of steps of how to install liminal in Airflow on Kubernetes:
+* [Find Airflow components](#Find-Airflow-pod-names)
+* [Install liminal in the components](#Install-liminal-in-Airflow-pods)
+* [Setting up liminal](#Setting-up-liminal)
+
+#### Find Airflow pod names:
+
+```sh
+echo -n "Please enter a namespace: "
+read namespace
+components=("web" "scheduler" "worker")
+namespace=${namespace}
+podNames=()
+for component in ${components[@]}
+do
+ podNames+="$(kubectl get pod -n ${namespace} -l "app=airflow,component=${component}" --no-headers -o custom-columns=":metadata.name")\n"
+
+done
+
+echo -e "The following Airflow pods are:\n${podNames}"
+```
+#### Install liminal in Airflow pods:
+
+```sh
+IFS=$'\n'|''; set -f; podNames=( ${podNames} )
+webPodName=${podNames[${#podNames[@]} - 1]}
+
+for podName in `echo -e ${podNames[@]}`
+do
+ echo "Installing liminal in pod: ${podName}"
+ kubectl exec -it -n ${namespace} ${podName} -- bash -c 'pip install --user apache-liminal'
+done
+```
+
+#### Setting up liminal:
Review comment:
the title should be
"Deploying Liminal Yamls into Airflow"
##########
File path: docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md
##########
@@ -0,0 +1,160 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required bgit y applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Install Liminal in Airflow
+* [Workflow](#workflow)
+* [Prerequisites](#prerequisites)
+ * [Supported Distributions](#supported-distributions)
+* [Get Started](#Get-Started)
+ * [Scripted Installation](#scripted-installation)
+ * [Manual Installation](#manual-installation)
+* [References and other resources](#references-and-other-resources)
+
+## Workflow
+Liminal deploys into the Airflow components and the Yamls home folder is mounted by EFS.
+The process of liminal is emphasized on the following diagram:
+
+
+
+## Prerequisites
+Before you begin, ensure you have met the following requirements:
+
+* You have a Kubernetes cluster running with [Helm][homebrew-helm] (and Tiller if using Helm v2.x) installed
+* You have the kubectl command line [(kubectl CLI)][homebrew-kubectl] installed
+* You have the current [context][cluster-access-kubeconfig] in kubernetes' kubeconfig file
+* You have [Airflow on Kubernetes with AWS EFS][airflowInstallation]
+* You have already created [Liminal Yaml file][liminal-yaml-file] following the liminal [Getting Started Documentation][liminalGetStarted-doc]
+* Make sure that the [example repository][liminal-getting-started-project] is your workspace
+
+### Supported Distributions
+
+|Distribution | Versions |
+|-|-|
+|[Airflow][airflowImage] | apache/airflow:1.10.12-python3.6 |
+|[Airflow Helm Chart][airflowChart] | 7.14.3 |
+
+#### You may need to add the Repo
+
+```sh
+helm repo add airflow-stable https://airflow-helm.github.io/charts
+helm repo update
+```
+
+## Get Started
+
+### Scripted Installation
+
+[Use `install_liminal_in_airflow_on_kubernetes.sh`][liminal-installation-script] to install liminal in Airflow.
+* Move to the [example repository project][liminal-getting-started-project]
+* Copy the [script][liminal-installation-script] to the [example repository project][liminal-getting-started-project] and run it
+
+
+
+### Manual Installation
+
+Here are a couple of steps of how to install liminal in Airflow on Kubernetes:
+* [Find Airflow components](#Find-Airflow-pod-names)
+* [Install liminal in the components](#Install-liminal-in-Airflow-pods)
+* [Setting up liminal](#Setting-up-liminal)
+
+#### Find Airflow pod names:
+
+```sh
+echo -n "Please enter a namespace: "
+read namespace
+components=("web" "scheduler" "worker")
+namespace=${namespace}
+podNames=()
+for component in ${components[@]}
+do
+ podNames+="$(kubectl get pod -n ${namespace} -l "app=airflow,component=${component}" --no-headers -o custom-columns=":metadata.name")\n"
+
+done
+
+echo -e "The following Airflow pods are:\n${podNames}"
+```
+#### Install liminal in Airflow pods:
+
+```sh
+IFS=$'\n'|''; set -f; podNames=( ${podNames} )
+webPodName=${podNames[${#podNames[@]} - 1]}
+
+for podName in `echo -e ${podNames[@]}`
+do
+ echo "Installing liminal in pod: ${podName}"
+ kubectl exec -it -n ${namespace} ${podName} -- bash -c 'pip install --user apache-liminal'
+done
+```
+
+#### Setting up liminal:
+There are a couple of ways to deploy Yamls in Airflow.
+1. Mount your Amazon EFS file system on a Linux instance:
+ ```
+ sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport <EFS_ID>:/ /opt/airflow/dags
+ ```
+2. Use gitsync function in order to sync Yamls in the Amazon EFS file system. In this case, we chose to use a kubernetes job:
Review comment:
I am not sure it's necessary to get into this suggestion.
Since it does essentially what we describe the deployment box does just using a git action...
If you really want to contrast our proposal with a different approach, you can mention (without code) that
some people deploy into airflow in pulls from git from inside airflow etc - but that's not our recommended way.
##########
File path: docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md
##########
@@ -0,0 +1,122 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required bgit y applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Install Liminal in Airflow
+* [Prerequisites](#prerequisites)
+ * [Supported Distributions](#supported-distributions)
+* [Get Started](#Get-Started)
+ * [Scripted Installation](#scripted-installation)
+ * [Manual Installation](#manual-installation)
+* [References and other resources](#references-and-other-resources)
+
+## Prerequisites
+Before you begin, ensure you have met the following requirements:
+
+* You have a Kubernetes cluster running with [Helm][homebrew-helm] (and Tiller if using Helm v2.x) installed
+* You have the kubectl command line [(kubectl CLI)][homebrew-kubectl] installed
+* You have the current [context][cluster-access-kubeconfig] in kubernetes' kubeconfig file
+* You have Airflow on Kubernetes with AWS EFS
+* You have already created [custom DAGs][custom-dag] following the liminal [Getting Started Documentation][liminalGetStarted-doc]
+* Make sure that the [example repository][liminal-getting-started-project] is your workspace
+
+### Supported Distributions
+
+|Distribution | Versions |
+|-|-|
+|[Airflow][airflowImage] | apache/airflow:1.10.12-python3.6 |
+|[Airflow Helm Chart][airflowChart] | 7.14.3 |
+
+#### You may need to add the Repo
+
+```sh
+helm repo add airflow-stable https://airflow-helm.github.io/charts
+helm repo update
+```
+
+## Get Started
+
+### Scripted Installation
+
+[Use `install_liminal_in_airflow_on_kubernetes.sh`][liminal-installation-script] to install liminal in Airflow.
+* Move to the [example repository project][liminal-getting-started-project]
+* Copy the [script][liminal-installation-script] to the [example repository project][liminal-getting-started-project] and run it
+
+
+
+### Manual Installation
+
+Here are a couple of steps of how to install liminal in Airflow on Kubernetes:
+* [Find Airflow components](#Find-Airflow-pod-names)
+* [Install liminal in the components](#Install-liminal-in-Airflow-pods)
+* [Setting up liminal](#Setting-up-liminal)
+
+#### Find Airflow pod names:
+
+```sh
+echo -n "Please enter a namespace: "
+read namespace
+components=("web" "scheduler" "worker")
+namespace=${namespace}
+podNames=()
+for component in ${components[@]}
+do
+ podNames+="$(kubectl get pod -n ${namespace} -l "app=airflow,component=${component}" --no-headers -o custom-columns=":metadata.name")\n"
+
+done
+
+echo -e "The following Airflow pods are:\n${podNames}"
+```
+#### Install liminal in Airflow pods:
Review comment:
so in practice, given that we assume airflow is already installed, this is the best practice to add a new package later on?
##########
File path: docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md
##########
@@ -0,0 +1,160 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required bgit y applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Install Liminal in Airflow
+* [Workflow](#workflow)
+* [Prerequisites](#prerequisites)
+ * [Supported Distributions](#supported-distributions)
+* [Get Started](#Get-Started)
+ * [Scripted Installation](#scripted-installation)
+ * [Manual Installation](#manual-installation)
+* [References and other resources](#references-and-other-resources)
+
+## Workflow
+Liminal deploys into the Airflow components and the Yamls home folder is mounted by EFS.
Review comment:
To deploy and run liminal on airflow, we need to complete two tasks:
1. Ensure that the Liminal python package is installed inside each one of Airflow's pods (schedulers and Web Server)
2. Ensure that the user's Liminal code (i.e. Yamls) are present in each of the pods' DAGs folder (typically located in /opt/airflow/dags)
The first task is accomplished by iterating over Airflows' k8s pods and running a pip install on each of them.
The second task is accomplished by using a folder on a shared filesystem (EFS) to host the airflow DAGs (and Liminal Yamls). This folder is mounted into Airflow pods' DAGs folder - thus enables each of the pods to pick the files up and run the Liminal DAGs.
##########
File path: docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md
##########
@@ -0,0 +1,160 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required bgit y applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Install Liminal in Airflow
+* [Workflow](#workflow)
+* [Prerequisites](#prerequisites)
+ * [Supported Distributions](#supported-distributions)
+* [Get Started](#Get-Started)
+ * [Scripted Installation](#scripted-installation)
+ * [Manual Installation](#manual-installation)
+* [References and other resources](#references-and-other-resources)
+
+## Workflow
+Liminal deploys into the Airflow components and the Yamls home folder is mounted by EFS.
+The process of liminal is emphasized on the following diagram:
+
+
+
+## Prerequisites
+Before you begin, ensure you have met the following requirements:
+
+* You have a Kubernetes cluster running with [Helm][homebrew-helm] (and Tiller if using Helm v2.x) installed
+* You have the kubectl command line [(kubectl CLI)][homebrew-kubectl] installed
+* You have the current [context][cluster-access-kubeconfig] in kubernetes' kubeconfig file
+* You have [Airflow on Kubernetes with AWS EFS][airflowInstallation]
+* You have already created [Liminal Yaml file][liminal-yaml-file] following the liminal [Getting Started Documentation][liminalGetStarted-doc]
+* Make sure that the [example repository][liminal-getting-started-project] is your workspace
+
+### Supported Distributions
+
+|Distribution | Versions |
+|-|-|
+|[Airflow][airflowImage] | apache/airflow:1.10.12-python3.6 |
+|[Airflow Helm Chart][airflowChart] | 7.14.3 |
+
+#### You may need to add the Repo
+
+```sh
+helm repo add airflow-stable https://airflow-helm.github.io/charts
+helm repo update
+```
+
+## Get Started
+
+### Scripted Installation
+
+[Use `install_liminal_in_airflow_on_kubernetes.sh`][liminal-installation-script] to install liminal in Airflow.
+* Move to the [example repository project][liminal-getting-started-project]
+* Copy the [script][liminal-installation-script] to the [example repository project][liminal-getting-started-project] and run it
+
+
+
+### Manual Installation
+
+Here are a couple of steps of how to install liminal in Airflow on Kubernetes:
+* [Find Airflow components](#Find-Airflow-pod-names)
+* [Install liminal in the components](#Install-liminal-in-Airflow-pods)
+* [Setting up liminal](#Setting-up-liminal)
+
+#### Find Airflow pod names:
+
+```sh
+echo -n "Please enter a namespace: "
+read namespace
+components=("web" "scheduler" "worker")
+namespace=${namespace}
+podNames=()
+for component in ${components[@]}
+do
+ podNames+="$(kubectl get pod -n ${namespace} -l "app=airflow,component=${component}" --no-headers -o custom-columns=":metadata.name")\n"
+
+done
+
+echo -e "The following Airflow pods are:\n${podNames}"
+```
+#### Install liminal in Airflow pods:
+
+```sh
+IFS=$'\n'|''; set -f; podNames=( ${podNames} )
+webPodName=${podNames[${#podNames[@]} - 1]}
+
+for podName in `echo -e ${podNames[@]}`
+do
+ echo "Installing liminal in pod: ${podName}"
+ kubectl exec -it -n ${namespace} ${podName} -- bash -c 'pip install --user apache-liminal'
+done
+```
+
+#### Setting up liminal:
+There are a couple of ways to deploy Yamls in Airflow.
Review comment:
I suggest to repeat the message:
Now that Airflow contains the liminal package, we need to deploy the actual Yamls (DAGs) into airflow - by placing the Yaml files in each components` /opt/airflow/dags folder.
There are multiple ways to achieve this.
Our recommended approach is to use EFS as a shared folder containing the Liminal Yamls.
To this end, you will need to perform the deployment from a (separate) linux box, which can get hold of the Liminal Yamls and deploy them into the EFS - which we shall name "the deployment box".
One time setup:
1. Create an EFS file system
2. Mount the EFS file system onto a deployment box
```sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport <EFS_ID>:/ /mnt/efs```
2. Create a folder on an EFS drive. This will be our target for deployment
```mkdir -p /mnt/efs/opt/airflow/liminal_home```
3. set
```export LIMINAL_HOME=/mnt/efs/opt/airflow/liminal_home```
4. Follow the guide <<<<add a good link>>>> for deploying Airflow on K8S, with EFS mount.
Point the pods' PV to the right EFS folder, like so:
```
volumeMounts:
- name: efs-data
mountPath: /opt/airflow/dags
```
where efs-data is a PVC pointing to ```<EFS_ID>:/opt/airflow/liminal_home```
Once this setup is done (one time), you have:
1. A deployment box which can deploy Liminal to an EFS folder
2. Airflow pods which will pick up the Yamls from that folder automatically.
Each time you want to deploy the Liminal Yamls from the deployment box:
```liminal deploy <<<path to liminal user code>>>```
##########
File path: docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md
##########
@@ -0,0 +1,160 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required bgit y applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Install Liminal in Airflow
+* [Workflow](#workflow)
+* [Prerequisites](#prerequisites)
+ * [Supported Distributions](#supported-distributions)
+* [Get Started](#Get-Started)
+ * [Scripted Installation](#scripted-installation)
+ * [Manual Installation](#manual-installation)
+* [References and other resources](#references-and-other-resources)
+
+## Workflow
+Liminal deploys into the Airflow components and the Yamls home folder is mounted by EFS.
+The process of liminal is emphasized on the following diagram:
+
+
+
+## Prerequisites
+Before you begin, ensure you have met the following requirements:
+
+* You have a Kubernetes cluster running with [Helm][homebrew-helm] (and Tiller if using Helm v2.x) installed
+* You have the kubectl command line [(kubectl CLI)][homebrew-kubectl] installed
+* You have the current [context][cluster-access-kubeconfig] in kubernetes' kubeconfig file
+* You have [Airflow on Kubernetes with AWS EFS][airflowInstallation]
Review comment:
this post is actually not good in my opinion.
I believe the recommended way to deploy efs today is to use csi;
in this post they for some reason set up an NFS server pod, not sure what for...
Essentially I think we can write our own post in a matter of 2-3 hours which will be better.
##########
File path: docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md
##########
@@ -0,0 +1,160 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required bgit y applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Install Liminal in Airflow
+* [Workflow](#workflow)
+* [Prerequisites](#prerequisites)
+ * [Supported Distributions](#supported-distributions)
+* [Get Started](#Get-Started)
+ * [Scripted Installation](#scripted-installation)
+ * [Manual Installation](#manual-installation)
+* [References and other resources](#references-and-other-resources)
+
+## Workflow
+Liminal deploys into the Airflow components and the Yamls home folder is mounted by EFS.
+The process of liminal is emphasized on the following diagram:
+
+
+
+## Prerequisites
+Before you begin, ensure you have met the following requirements:
+
+* You have a Kubernetes cluster running with [Helm][homebrew-helm] (and Tiller if using Helm v2.x) installed
+* You have the kubectl command line [(kubectl CLI)][homebrew-kubectl] installed
+* You have the current [context][cluster-access-kubeconfig] in kubernetes' kubeconfig file
+* You have [Airflow on Kubernetes with AWS EFS][airflowInstallation]
+* You have already created [Liminal Yaml file][liminal-yaml-file] following the liminal [Getting Started Documentation][liminalGetStarted-doc]
+* Make sure that the [example repository][liminal-getting-started-project] is your workspace
+
+### Supported Distributions
+
+|Distribution | Versions |
+|-|-|
+|[Airflow][airflowImage] | apache/airflow:1.10.12-python3.6 |
+|[Airflow Helm Chart][airflowChart] | 7.14.3 |
+
+#### You may need to add the Repo
+
+```sh
+helm repo add airflow-stable https://airflow-helm.github.io/charts
+helm repo update
+```
+
+## Get Started
+
+### Scripted Installation
+
+[Use `install_liminal_in_airflow_on_kubernetes.sh`][liminal-installation-script] to install liminal in Airflow.
+* Move to the [example repository project][liminal-getting-started-project]
+* Copy the [script][liminal-installation-script] to the [example repository project][liminal-getting-started-project] and run it
+
+
+
+### Manual Installation
+
+Here are a couple of steps of how to install liminal in Airflow on Kubernetes:
+* [Find Airflow components](#Find-Airflow-pod-names)
+* [Install liminal in the components](#Install-liminal-in-Airflow-pods)
+* [Setting up liminal](#Setting-up-liminal)
+
+#### Find Airflow pod names:
+
+```sh
+echo -n "Please enter a namespace: "
+read namespace
+components=("web" "scheduler" "worker")
+namespace=${namespace}
+podNames=()
+for component in ${components[@]}
+do
+ podNames+="$(kubectl get pod -n ${namespace} -l "app=airflow,component=${component}" --no-headers -o custom-columns=":metadata.name")\n"
+
+done
+
+echo -e "The following Airflow pods are:\n${podNames}"
+```
+#### Install liminal in Airflow pods:
+
+```sh
+IFS=$'\n'|''; set -f; podNames=( ${podNames} )
+webPodName=${podNames[${#podNames[@]} - 1]}
+
+for podName in `echo -e ${podNames[@]}`
+do
+ echo "Installing liminal in pod: ${podName}"
+ kubectl exec -it -n ${namespace} ${podName} -- bash -c 'pip install --user apache-liminal'
+done
+```
+
+#### Setting up liminal:
+There are a couple of ways to deploy Yamls in Airflow.
+1. Mount your Amazon EFS file system on a Linux instance:
+ ```
+ sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport <EFS_ID>:/ /opt/airflow/dags
+ ```
+2. Use gitsync function in order to sync Yamls in the Amazon EFS file system. In this case, we chose to use a kubernetes job:
+ ```
+ apiVersion: batch/v1
+ kind: Job
+ metadata:
+ namespace: production
+ name: airflower-production-git
+ spec:
+ activeDeadlineSeconds: 60
+ backoffLimit: 1
+ template:
+ spec:
+ containers:
+ - name: git
+ image: alpine/git
+ command: ["/bin/sh","-c"]
+ args: ["cd /dags; git fetch && git reset --hard origin/master"]
+ volumeMounts:
+ - mountPath: /dags
+ name: dags
+ volumes:
+ - name: dags
+ persistentVolumeClaim:
+ claimName: airflow-production
+ restartPolicy: Never
+ ```
+3. Deploy the Yamls by using kubectl in order to copy the Yamls to the mounted file system. In this case, we chose to use kubectl to copy the Yamls to the mounted EFS:
Review comment:
Again, in terms of completeness - you are right, it can also be done in this way.
however we need to focus on our recommended way, for which we asked the user to set up EFS
you can mention it in the previous section in passing:
There are other ways to deploy into Aiflow.
You can poll a git repo from inside airflow and copy the files in pull.
Alternatively, you can do something similar to how we installed liminal on the pods, and iterate on them, copying the file into each.
However, to use native Liminal tools, we recommend the following approach....
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] aviemzur commented on pull request #11: liminal installation doc
Posted by GitBox <gi...@apache.org>.
aviemzur commented on pull request #11:
URL: https://github.com/apache/incubator-liminal/pull/11#issuecomment-750859384
R: @assapin
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] naturalett commented on a change in pull request #11: liminal installation doc
Posted by GitBox <gi...@apache.org>.
naturalett commented on a change in pull request #11:
URL: https://github.com/apache/incubator-liminal/pull/11#discussion_r594723598
##########
File path: docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md
##########
@@ -0,0 +1,103 @@
+# Install Liminal in Airflow
+* [Prerequisites](#prerequisites)
+ * [Supported Distributions](#supported-distributions)
+* [Get Started](#Get-Started)
+ * [Scripted Installation](#scripted-installation)
+ * [Manual Installation](#manual-installation)
+* [References and other resources](#references-and-other-resources)
+
+## Prerequisites
Review comment:
1. All the bullets below the prerequisites.
2. The main assumption here is the bullet: **You have Airflow on Kubernetes with AWS EFS**
3. Won't it be too much? Afterwards, I would have to explain the entire diagram which may be distractive.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] naturalett commented on a change in pull request #11: liminal installation doc
Posted by GitBox <gi...@apache.org>.
naturalett commented on a change in pull request #11:
URL: https://github.com/apache/incubator-liminal/pull/11#discussion_r604266201
##########
File path: docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md
##########
@@ -0,0 +1,160 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required bgit y applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Install Liminal in Airflow
+* [Workflow](#workflow)
+* [Prerequisites](#prerequisites)
+ * [Supported Distributions](#supported-distributions)
+* [Get Started](#Get-Started)
+ * [Scripted Installation](#scripted-installation)
+ * [Manual Installation](#manual-installation)
+* [References and other resources](#references-and-other-resources)
+
+## Workflow
+Liminal deploys into the Airflow components and the Yamls home folder is mounted by EFS.
+The process of liminal is emphasized on the following diagram:
+
+
+
+## Prerequisites
+Before you begin, ensure you have met the following requirements:
+
+* You have a Kubernetes cluster running with [Helm][homebrew-helm] (and Tiller if using Helm v2.x) installed
+* You have the kubectl command line [(kubectl CLI)][homebrew-kubectl] installed
+* You have the current [context][cluster-access-kubeconfig] in kubernetes' kubeconfig file
+* You have [Airflow on Kubernetes with AWS EFS][airflowInstallation]
+* You have already created [Liminal Yaml file][liminal-yaml-file] following the liminal [Getting Started Documentation][liminalGetStarted-doc]
+* Make sure that the [example repository][liminal-getting-started-project] is your workspace
+
+### Supported Distributions
+
+|Distribution | Versions |
+|-|-|
+|[Airflow][airflowImage] | apache/airflow:1.10.12-python3.6 |
+|[Airflow Helm Chart][airflowChart] | 7.14.3 |
+
+#### You may need to add the Repo
+
+```sh
+helm repo add airflow-stable https://airflow-helm.github.io/charts
+helm repo update
+```
+
+## Get Started
+
+### Scripted Installation
+
+[Use `install_liminal_in_airflow_on_kubernetes.sh`][liminal-installation-script] to install liminal in Airflow.
+* Move to the [example repository project][liminal-getting-started-project]
+* Copy the [script][liminal-installation-script] to the [example repository project][liminal-getting-started-project] and run it
+
+
+
+### Manual Installation
+
+Here are a couple of steps of how to install liminal in Airflow on Kubernetes:
+* [Find Airflow components](#Find-Airflow-pod-names)
+* [Install liminal in the components](#Install-liminal-in-Airflow-pods)
+* [Setting up liminal](#Setting-up-liminal)
+
+#### Find Airflow pod names:
+
+```sh
+echo -n "Please enter a namespace: "
+read namespace
+components=("web" "scheduler" "worker")
+namespace=${namespace}
+podNames=()
+for component in ${components[@]}
+do
+ podNames+="$(kubectl get pod -n ${namespace} -l "app=airflow,component=${component}" --no-headers -o custom-columns=":metadata.name")\n"
+
+done
+
+echo -e "The following Airflow pods are:\n${podNames}"
+```
+#### Install liminal in Airflow pods:
+
+```sh
+IFS=$'\n'|''; set -f; podNames=( ${podNames} )
+webPodName=${podNames[${#podNames[@]} - 1]}
+
+for podName in `echo -e ${podNames[@]}`
+do
+ echo "Installing liminal in pod: ${podName}"
+ kubectl exec -it -n ${namespace} ${podName} -- bash -c 'pip install --user apache-liminal'
+done
+```
+
+#### Setting up liminal:
+There are a couple of ways to deploy Yamls in Airflow.
Review comment:
Ok I see.
I will create a short video how I did it.
And we need a doc of how to create Airflow on k8s with EFS.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] naturalett commented on a change in pull request #11: liminal installation doc
Posted by GitBox <gi...@apache.org>.
naturalett commented on a change in pull request #11:
URL: https://github.com/apache/incubator-liminal/pull/11#discussion_r594689399
##########
File path: docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md
##########
@@ -0,0 +1,94 @@
+# Install Liminal in Airflow
+* [Prerequisites](#prerequisites)
+ * [Supported Distributions](#supported-distributions)
+* [Get Started](#Get-Started)
+ * [Scripted Installation](#scripted-installation)
+ * [Manual Installation](#manual-installation)
+
+## Prerequisites
+This guide assumes that:
+
+You have a Kubernetes cluster running with Helm (and Tiller if using Helm v2.x) installed.\
Review comment:
It contains bullet points (https://github.com/apache/incubator-liminal/blob/b8cc352e970335c34001ecb11c420830bb9a0af6/docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] naturalett commented on a change in pull request #11: liminal installation doc
Posted by GitBox <gi...@apache.org>.
naturalett commented on a change in pull request #11:
URL: https://github.com/apache/incubator-liminal/pull/11#discussion_r604266201
##########
File path: docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md
##########
@@ -0,0 +1,160 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required bgit y applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Install Liminal in Airflow
+* [Workflow](#workflow)
+* [Prerequisites](#prerequisites)
+ * [Supported Distributions](#supported-distributions)
+* [Get Started](#Get-Started)
+ * [Scripted Installation](#scripted-installation)
+ * [Manual Installation](#manual-installation)
+* [References and other resources](#references-and-other-resources)
+
+## Workflow
+Liminal deploys into the Airflow components and the Yamls home folder is mounted by EFS.
+The process of liminal is emphasized on the following diagram:
+
+
+
+## Prerequisites
+Before you begin, ensure you have met the following requirements:
+
+* You have a Kubernetes cluster running with [Helm][homebrew-helm] (and Tiller if using Helm v2.x) installed
+* You have the kubectl command line [(kubectl CLI)][homebrew-kubectl] installed
+* You have the current [context][cluster-access-kubeconfig] in kubernetes' kubeconfig file
+* You have [Airflow on Kubernetes with AWS EFS][airflowInstallation]
+* You have already created [Liminal Yaml file][liminal-yaml-file] following the liminal [Getting Started Documentation][liminalGetStarted-doc]
+* Make sure that the [example repository][liminal-getting-started-project] is your workspace
+
+### Supported Distributions
+
+|Distribution | Versions |
+|-|-|
+|[Airflow][airflowImage] | apache/airflow:1.10.12-python3.6 |
+|[Airflow Helm Chart][airflowChart] | 7.14.3 |
+
+#### You may need to add the Repo
+
+```sh
+helm repo add airflow-stable https://airflow-helm.github.io/charts
+helm repo update
+```
+
+## Get Started
+
+### Scripted Installation
+
+[Use `install_liminal_in_airflow_on_kubernetes.sh`][liminal-installation-script] to install liminal in Airflow.
+* Move to the [example repository project][liminal-getting-started-project]
+* Copy the [script][liminal-installation-script] to the [example repository project][liminal-getting-started-project] and run it
+
+
+
+### Manual Installation
+
+Here are a couple of steps of how to install liminal in Airflow on Kubernetes:
+* [Find Airflow components](#Find-Airflow-pod-names)
+* [Install liminal in the components](#Install-liminal-in-Airflow-pods)
+* [Setting up liminal](#Setting-up-liminal)
+
+#### Find Airflow pod names:
+
+```sh
+echo -n "Please enter a namespace: "
+read namespace
+components=("web" "scheduler" "worker")
+namespace=${namespace}
+podNames=()
+for component in ${components[@]}
+do
+ podNames+="$(kubectl get pod -n ${namespace} -l "app=airflow,component=${component}" --no-headers -o custom-columns=":metadata.name")\n"
+
+done
+
+echo -e "The following Airflow pods are:\n${podNames}"
+```
+#### Install liminal in Airflow pods:
+
+```sh
+IFS=$'\n'|''; set -f; podNames=( ${podNames} )
+webPodName=${podNames[${#podNames[@]} - 1]}
+
+for podName in `echo -e ${podNames[@]}`
+do
+ echo "Installing liminal in pod: ${podName}"
+ kubectl exec -it -n ${namespace} ${podName} -- bash -c 'pip install --user apache-liminal'
+done
+```
+
+#### Setting up liminal:
+There are a couple of ways to deploy Yamls in Airflow.
Review comment:
Ok I see.
I will create a short video how I did it.
And we need a doc of how to create Airflow on k8s with EFS.
Also should we take care separately to liminal_dags.py? installing liminal doesn't do it. We need to copy it to the DAGs folder.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] assapin commented on a change in pull request #11: liminal installation doc
Posted by GitBox <gi...@apache.org>.
assapin commented on a change in pull request #11:
URL: https://github.com/apache/incubator-liminal/pull/11#discussion_r609713037
##########
File path: docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md
##########
@@ -0,0 +1,123 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required bgit y applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Install Liminal in Airflow
+* [Workflow](#workflow)
+* [Prerequisites](#prerequisites)
+ * [Supported Distributions](#supported-distributions)
+* [Get Started](#Get-Started)
+ * [Package dependency](#package-dependency)
+ * [Mount the EFS](#mount-the-EFS)
+ * [Liminal deployment](#liminal-deployment)
+* [References and other resources](#references-and-other-resources)
+
+## Workflow
+To deploy and run liminal on airflow, we need to complete two tasks:
+1. Ensure that the Liminal python package is installed inside each one of Airflow's pods (schedulers and Web Server)
+2. Ensure that the user's Liminal code (i.e. Yamls) are present in each of the pods' DAGs folder (typically located in /opt/airflow/dags)
+
+The first task is accomplished by iterating over Airflows' k8s pods and running a pip install on each of them.
+
+The second task is accomplished by using a folder on a shared filesystem (EFS) to host the airflow DAGs (and Liminal Yamls). \
+This folder is mounted into Airflow pods' DAGs folder - thus enables each of the pods to pick the files up and run the Liminal DAGs.
+
+The process of liminal is emphasized on the following diagram:
+
+
+
+## Prerequisites
+Before you begin, ensure you have met the following requirements:
+* You have the kubectl command line [(kubectl CLI)][homebrew-kubectl] installed
+* You have the current [context][cluster-access-kubeconfig] in kubernetes' kubeconfig file
+* You have [Airflow on Kubernetes with AWS EFS][airflowInstallation]
+
+### Supported Distributions
+
+|Distribution | Versions |
+|-|-|
+|[Airflow][airflowImage] | apache/airflow:1.10.14-python3.8 |
+|[Airflow Helm Chart][airflowChart] | [7.16.0][airflow-helm-chart-7.16.0] |
+
+#### You may need to add the Repo
+
+```sh
+helm repo add airflow-stable https://airflow-helm.github.io/charts
+helm repo update
+```
+
+## Get Started
+#### Package dependency
+There are multiple ways to install liminal in Airflow and it depends on what will be useful in your environment.\
+1. Use kubectl to execute pip install on the pods.
+2. Add `apache-liminal` as a requirement package to be installed once Airflow starts.
+
+#### Mount the EFS
+There are a couple of ways to deploy Yamls in Airflow our goal is to deploy the actual Yamls (DAGs) into airflow - by placing the Yaml files in each components\` `/opt/airflow/dags` folder.
+
+There are multiple ways to achieve this.
+
+Our recommended approach is to use EFS as a shared folder containing the Liminal Yamls. \
+To this end, you will need to perform the deployment from a (separate) linux box, which can get hold of the Liminal Yamls and deploy them into the EFS - which we shall name "the deployment box".
+
+One time setup:
+
+1. Use the EFS file system that you created for Airflow DAGs folder
+2. Mount the EFS file system onto a deployment box:
+```sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport <EFS_ID>:/ /mnt/efs```
+3. Create a folder on an EFS drive. This will be our target for deployment
+```mkdir -p /mnt/efs/opt/airflow/liminal_home```
+4. ```export LIMINAL_HOME=/mnt/efs/opt/airflow/liminal_home```
+5. Follow the guide <<<>>> for deploying Airflow on K8S, with EFS mount.
Review comment:
Mount the EFS shared folder on Airflow pods. Refer to <<<ENTER LINK HERE>>> for more details on how to achieve this task
##########
File path: docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md
##########
@@ -0,0 +1,123 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required bgit y applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Install Liminal in Airflow
+* [Workflow](#workflow)
+* [Prerequisites](#prerequisites)
+ * [Supported Distributions](#supported-distributions)
+* [Get Started](#Get-Started)
+ * [Package dependency](#package-dependency)
+ * [Mount the EFS](#mount-the-EFS)
+ * [Liminal deployment](#liminal-deployment)
+* [References and other resources](#references-and-other-resources)
+
+## Workflow
+To deploy and run liminal on airflow, we need to complete two tasks:
+1. Ensure that the Liminal python package is installed inside each one of Airflow's pods (schedulers and Web Server)
+2. Ensure that the user's Liminal code (i.e. Yamls) are present in each of the pods' DAGs folder (typically located in /opt/airflow/dags)
+
+The first task is accomplished by iterating over Airflows' k8s pods and running a pip install on each of them.
Review comment:
suggest to replace with "this is a standard airflow management task, which can be achieved in multiple ways depending on your setup (we will cover a few later in this guide).
##########
File path: docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md
##########
@@ -0,0 +1,123 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required bgit y applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Install Liminal in Airflow
+* [Workflow](#workflow)
+* [Prerequisites](#prerequisites)
+ * [Supported Distributions](#supported-distributions)
+* [Get Started](#Get-Started)
+ * [Package dependency](#package-dependency)
+ * [Mount the EFS](#mount-the-EFS)
+ * [Liminal deployment](#liminal-deployment)
+* [References and other resources](#references-and-other-resources)
+
+## Workflow
+To deploy and run liminal on airflow, we need to complete two tasks:
+1. Ensure that the Liminal python package is installed inside each one of Airflow's pods (schedulers and Web Server)
+2. Ensure that the user's Liminal code (i.e. Yamls) are present in each of the pods' DAGs folder (typically located in /opt/airflow/dags)
+
+The first task is accomplished by iterating over Airflows' k8s pods and running a pip install on each of them.
+
+The second task is accomplished by using a folder on a shared filesystem (EFS) to host the airflow DAGs (and Liminal Yamls). \
+This folder is mounted into Airflow pods' DAGs folder - thus enables each of the pods to pick the files up and run the Liminal DAGs.
+
+The process of liminal is emphasized on the following diagram:
+
+
+
+## Prerequisites
+Before you begin, ensure you have met the following requirements:
+* You have the kubectl command line [(kubectl CLI)][homebrew-kubectl] installed
+* You have the current [context][cluster-access-kubeconfig] in kubernetes' kubeconfig file
+* You have [Airflow on Kubernetes with AWS EFS][airflowInstallation]
+
+### Supported Distributions
+
+|Distribution | Versions |
+|-|-|
+|[Airflow][airflowImage] | apache/airflow:1.10.14-python3.8 |
+|[Airflow Helm Chart][airflowChart] | [7.16.0][airflow-helm-chart-7.16.0] |
+
+#### You may need to add the Repo
+
+```sh
+helm repo add airflow-stable https://airflow-helm.github.io/charts
+helm repo update
+```
+
+## Get Started
+#### Package dependency
+There are multiple ways to install liminal in Airflow and it depends on what will be useful in your environment.\
+1. Use kubectl to execute pip install on the pods.
+2. Add `apache-liminal` as a requirement package to be installed once Airflow starts.
+
+#### Mount the EFS
+There are a couple of ways to deploy Yamls in Airflow our goal is to deploy the actual Yamls (DAGs) into airflow - by placing the Yaml files in each components\` `/opt/airflow/dags` folder.
+
+There are multiple ways to achieve this.
+
+Our recommended approach is to use EFS as a shared folder containing the Liminal Yamls. \
+To this end, you will need to perform the deployment from a (separate) linux box, which can get hold of the Liminal Yamls and deploy them into the EFS - which we shall name "the deployment box".
+
+One time setup:
+
+1. Use the EFS file system that you created for Airflow DAGs folder
+2. Mount the EFS file system onto a deployment box:
+```sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport <EFS_ID>:/ /mnt/efs```
+3. Create a folder on an EFS drive. This will be our target for deployment
+```mkdir -p /mnt/efs/opt/airflow/liminal_home```
+4. ```export LIMINAL_HOME=/mnt/efs/opt/airflow/liminal_home```
+5. Follow the guide <<<>>> for deploying Airflow on K8S, with EFS mount.
+Point the pods' PV to the right EFS folder, like so:
+ ```
+ volumeMounts:
+ - name: efs-data
+ mountPath: /opt/airflow/dags
+ ```
+ where efs-data is a PVC pointing to `<EFS_ID>:/opt/airflow/liminal_home`
+
+Once this setup is done (one time), you have:
+1. A deployment box which can deploy Liminal to an EFS folder
+2. Airflow pods which will pick up the Yamls from that folder automatically.
+
+#### Liminal deployment
+Each time you want to deploy the Liminal Yamls from the deployment box:
+```
+liminal deploy --path <<<path to liminal user code>>>
+```
+
+#### Scripted Installation
+Launch EC2 in the same environment of the Airflow and [Use `install_liminal_in_airflow_on_kubernetes.sh`][liminal-installation-script] in order to install liminal.
+
+Script steps to run:
Review comment:
explain that this script shows how to install liminal demo dags from liminal repo, and should be edited by the user to include her own yamls etc.
##########
File path: docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md
##########
@@ -0,0 +1,123 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required bgit y applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Install Liminal in Airflow
+* [Workflow](#workflow)
+* [Prerequisites](#prerequisites)
+ * [Supported Distributions](#supported-distributions)
+* [Get Started](#Get-Started)
+ * [Package dependency](#package-dependency)
+ * [Mount the EFS](#mount-the-EFS)
+ * [Liminal deployment](#liminal-deployment)
+* [References and other resources](#references-and-other-resources)
+
+## Workflow
+To deploy and run liminal on airflow, we need to complete two tasks:
+1. Ensure that the Liminal python package is installed inside each one of Airflow's pods (schedulers and Web Server)
+2. Ensure that the user's Liminal code (i.e. Yamls) are present in each of the pods' DAGs folder (typically located in /opt/airflow/dags)
+
+The first task is accomplished by iterating over Airflows' k8s pods and running a pip install on each of them.
+
+The second task is accomplished by using a folder on a shared filesystem (EFS) to host the airflow DAGs (and Liminal Yamls). \
+This folder is mounted into Airflow pods' DAGs folder - thus enables each of the pods to pick the files up and run the Liminal DAGs.
+
+The process of liminal is emphasized on the following diagram:
Review comment:
emphasized == depicted
##########
File path: docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md
##########
@@ -0,0 +1,123 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required bgit y applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Install Liminal in Airflow
+* [Workflow](#workflow)
+* [Prerequisites](#prerequisites)
+ * [Supported Distributions](#supported-distributions)
+* [Get Started](#Get-Started)
+ * [Package dependency](#package-dependency)
+ * [Mount the EFS](#mount-the-EFS)
+ * [Liminal deployment](#liminal-deployment)
+* [References and other resources](#references-and-other-resources)
+
+## Workflow
+To deploy and run liminal on airflow, we need to complete two tasks:
+1. Ensure that the Liminal python package is installed inside each one of Airflow's pods (schedulers and Web Server)
+2. Ensure that the user's Liminal code (i.e. Yamls) are present in each of the pods' DAGs folder (typically located in /opt/airflow/dags)
+
+The first task is accomplished by iterating over Airflows' k8s pods and running a pip install on each of them.
+
+The second task is accomplished by using a folder on a shared filesystem (EFS) to host the airflow DAGs (and Liminal Yamls). \
+This folder is mounted into Airflow pods' DAGs folder - thus enables each of the pods to pick the files up and run the Liminal DAGs.
+
+The process of liminal is emphasized on the following diagram:
+
+
+
+## Prerequisites
+Before you begin, ensure you have met the following requirements:
+* You have the kubectl command line [(kubectl CLI)][homebrew-kubectl] installed
+* You have the current [context][cluster-access-kubeconfig] in kubernetes' kubeconfig file
+* You have [Airflow on Kubernetes with AWS EFS][airflowInstallation]
+
+### Supported Distributions
+
+|Distribution | Versions |
+|-|-|
+|[Airflow][airflowImage] | apache/airflow:1.10.14-python3.8 |
+|[Airflow Helm Chart][airflowChart] | [7.16.0][airflow-helm-chart-7.16.0] |
Review comment:
This implies we "support the distribution" of this helm chart.
Shall we drop it? and the "you may need to add the repo"?
##########
File path: docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md
##########
@@ -0,0 +1,123 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required bgit y applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Install Liminal in Airflow
+* [Workflow](#workflow)
+* [Prerequisites](#prerequisites)
+ * [Supported Distributions](#supported-distributions)
+* [Get Started](#Get-Started)
+ * [Package dependency](#package-dependency)
+ * [Mount the EFS](#mount-the-EFS)
+ * [Liminal deployment](#liminal-deployment)
+* [References and other resources](#references-and-other-resources)
+
+## Workflow
+To deploy and run liminal on airflow, we need to complete two tasks:
+1. Ensure that the Liminal python package is installed inside each one of Airflow's pods (schedulers and Web Server)
+2. Ensure that the user's Liminal code (i.e. Yamls) are present in each of the pods' DAGs folder (typically located in /opt/airflow/dags)
+
+The first task is accomplished by iterating over Airflows' k8s pods and running a pip install on each of them.
+
+The second task is accomplished by using a folder on a shared filesystem (EFS) to host the airflow DAGs (and Liminal Yamls). \
+This folder is mounted into Airflow pods' DAGs folder - thus enables each of the pods to pick the files up and run the Liminal DAGs.
+
+The process of liminal is emphasized on the following diagram:
+
+
+
+## Prerequisites
+Before you begin, ensure you have met the following requirements:
+* You have the kubectl command line [(kubectl CLI)][homebrew-kubectl] installed
+* You have the current [context][cluster-access-kubeconfig] in kubernetes' kubeconfig file
+* You have [Airflow on Kubernetes with AWS EFS][airflowInstallation]
+
+### Supported Distributions
+
+|Distribution | Versions |
+|-|-|
+|[Airflow][airflowImage] | apache/airflow:1.10.14-python3.8 |
+|[Airflow Helm Chart][airflowChart] | [7.16.0][airflow-helm-chart-7.16.0] |
+
+#### You may need to add the Repo
+
+```sh
+helm repo add airflow-stable https://airflow-helm.github.io/charts
+helm repo update
+```
+
+## Get Started
+#### Package dependency
+There are multiple ways to install liminal in Airflow and it depends on what will be useful in your environment.\
+1. Use kubectl to execute pip install on the pods.
+2. Add `apache-liminal` as a requirement package to be installed once Airflow starts.
Review comment:
3. Build your own Airflow images based on the official docker image from https://github.com/apache/airflow/blob/0d3ee669243a5c5648fa7d707cd3547905772142/Dockerfile,
with the command
docker build --build-arg ADDITIONAL_PYTHON_DEPS='apache-liminal'
##########
File path: docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md
##########
@@ -0,0 +1,123 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required bgit y applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Install Liminal in Airflow
+* [Workflow](#workflow)
+* [Prerequisites](#prerequisites)
+ * [Supported Distributions](#supported-distributions)
+* [Get Started](#Get-Started)
+ * [Package dependency](#package-dependency)
+ * [Mount the EFS](#mount-the-EFS)
+ * [Liminal deployment](#liminal-deployment)
+* [References and other resources](#references-and-other-resources)
+
+## Workflow
+To deploy and run liminal on airflow, we need to complete two tasks:
+1. Ensure that the Liminal python package is installed inside each one of Airflow's pods (schedulers and Web Server)
+2. Ensure that the user's Liminal code (i.e. Yamls) are present in each of the pods' DAGs folder (typically located in /opt/airflow/dags)
+
+The first task is accomplished by iterating over Airflows' k8s pods and running a pip install on each of them.
+
+The second task is accomplished by using a folder on a shared filesystem (EFS) to host the airflow DAGs (and Liminal Yamls). \
+This folder is mounted into Airflow pods' DAGs folder - thus enables each of the pods to pick the files up and run the Liminal DAGs.
+
+The process of liminal is emphasized on the following diagram:
+
+
+
+## Prerequisites
+Before you begin, ensure you have met the following requirements:
+* You have the kubectl command line [(kubectl CLI)][homebrew-kubectl] installed
+* You have the current [context][cluster-access-kubeconfig] in kubernetes' kubeconfig file
+* You have [Airflow on Kubernetes with AWS EFS][airflowInstallation]
+
+### Supported Distributions
+
+|Distribution | Versions |
+|-|-|
+|[Airflow][airflowImage] | apache/airflow:1.10.14-python3.8 |
+|[Airflow Helm Chart][airflowChart] | [7.16.0][airflow-helm-chart-7.16.0] |
+
+#### You may need to add the Repo
+
+```sh
+helm repo add airflow-stable https://airflow-helm.github.io/charts
+helm repo update
+```
+
+## Get Started
+#### Package dependency
+There are multiple ways to install liminal in Airflow and it depends on what will be useful in your environment.\
+1. Use kubectl to execute pip install on the pods.
+2. Add `apache-liminal` as a requirement package to be installed once Airflow starts.
+
+#### Mount the EFS
+There are a couple of ways to deploy Yamls in Airflow our goal is to deploy the actual Yamls (DAGs) into airflow - by placing the Yaml files in each components\` `/opt/airflow/dags` folder.
+
+There are multiple ways to achieve this.
+
+Our recommended approach is to use EFS as a shared folder containing the Liminal Yamls. \
+To this end, you will need to perform the deployment from a (separate) linux box, which can get hold of the Liminal Yamls and deploy them into the EFS - which we shall name "the deployment box".
+
+One time setup:
+
+1. Use the EFS file system that you created for Airflow DAGs folder
+2. Mount the EFS file system onto a deployment box:
+```sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport <EFS_ID>:/ /mnt/efs```
+3. Create a folder on an EFS drive. This will be our target for deployment
+```mkdir -p /mnt/efs/opt/airflow/liminal_home```
+4. ```export LIMINAL_HOME=/mnt/efs/opt/airflow/liminal_home```
+5. Follow the guide <<<>>> for deploying Airflow on K8S, with EFS mount.
+Point the pods' PV to the right EFS folder, like so:
+ ```
+ volumeMounts:
+ - name: efs-data
+ mountPath: /opt/airflow/dags
+ ```
+ where efs-data is a PVC pointing to `<EFS_ID>:/opt/airflow/liminal_home`
+
+Once this setup is done (one time), you have:
+1. A deployment box which can deploy Liminal to an EFS folder
+2. Airflow pods which will pick up the Yamls from that folder automatically.
+
+#### Liminal deployment
+Each time you want to deploy the Liminal Yamls from the deployment box:
Review comment:
AFter the initial setup, we can just use Liminal 'deploy' command to deploy the Yamls into Airflow, like so:
##########
File path: docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md
##########
@@ -0,0 +1,123 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required bgit y applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Install Liminal in Airflow
+* [Workflow](#workflow)
+* [Prerequisites](#prerequisites)
+ * [Supported Distributions](#supported-distributions)
+* [Get Started](#Get-Started)
+ * [Package dependency](#package-dependency)
+ * [Mount the EFS](#mount-the-EFS)
+ * [Liminal deployment](#liminal-deployment)
+* [References and other resources](#references-and-other-resources)
+
+## Workflow
+To deploy and run liminal on airflow, we need to complete two tasks:
+1. Ensure that the Liminal python package is installed inside each one of Airflow's pods (schedulers and Web Server)
+2. Ensure that the user's Liminal code (i.e. Yamls) are present in each of the pods' DAGs folder (typically located in /opt/airflow/dags)
+
+The first task is accomplished by iterating over Airflows' k8s pods and running a pip install on each of them.
+
+The second task is accomplished by using a folder on a shared filesystem (EFS) to host the airflow DAGs (and Liminal Yamls). \
+This folder is mounted into Airflow pods' DAGs folder - thus enables each of the pods to pick the files up and run the Liminal DAGs.
+
+The process of liminal is emphasized on the following diagram:
+
+
+
+## Prerequisites
+Before you begin, ensure you have met the following requirements:
+* You have the kubectl command line [(kubectl CLI)][homebrew-kubectl] installed
+* You have the current [context][cluster-access-kubeconfig] in kubernetes' kubeconfig file
+* You have [Airflow on Kubernetes with AWS EFS][airflowInstallation]
+
+### Supported Distributions
+
+|Distribution | Versions |
+|-|-|
+|[Airflow][airflowImage] | apache/airflow:1.10.14-python3.8 |
+|[Airflow Helm Chart][airflowChart] | [7.16.0][airflow-helm-chart-7.16.0] |
+
+#### You may need to add the Repo
+
+```sh
+helm repo add airflow-stable https://airflow-helm.github.io/charts
+helm repo update
+```
+
+## Get Started
+#### Package dependency
+There are multiple ways to install liminal in Airflow and it depends on what will be useful in your environment.\
+1. Use kubectl to execute pip install on the pods.
+2. Add `apache-liminal` as a requirement package to be installed once Airflow starts.
+
+#### Mount the EFS
+There are a couple of ways to deploy Yamls in Airflow our goal is to deploy the actual Yamls (DAGs) into airflow - by placing the Yaml files in each components\` `/opt/airflow/dags` folder.
+
+There are multiple ways to achieve this.
+
+Our recommended approach is to use EFS as a shared folder containing the Liminal Yamls. \
+To this end, you will need to perform the deployment from a (separate) linux box, which can get hold of the Liminal Yamls and deploy them into the EFS - which we shall name "the deployment box".
Review comment:
We will set up a "deployment box" - a simple EC2 instance with a mount to this folder,
in order to obtain the yamls (e.g. from git) and to copy them into the shared folder.
##########
File path: docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md
##########
@@ -0,0 +1,123 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required bgit y applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Install Liminal in Airflow
+* [Workflow](#workflow)
+* [Prerequisites](#prerequisites)
+ * [Supported Distributions](#supported-distributions)
+* [Get Started](#Get-Started)
+ * [Package dependency](#package-dependency)
+ * [Mount the EFS](#mount-the-EFS)
+ * [Liminal deployment](#liminal-deployment)
+* [References and other resources](#references-and-other-resources)
+
+## Workflow
+To deploy and run liminal on airflow, we need to complete two tasks:
+1. Ensure that the Liminal python package is installed inside each one of Airflow's pods (schedulers and Web Server)
+2. Ensure that the user's Liminal code (i.e. Yamls) are present in each of the pods' DAGs folder (typically located in /opt/airflow/dags)
+
+The first task is accomplished by iterating over Airflows' k8s pods and running a pip install on each of them.
+
+The second task is accomplished by using a folder on a shared filesystem (EFS) to host the airflow DAGs (and Liminal Yamls). \
+This folder is mounted into Airflow pods' DAGs folder - thus enables each of the pods to pick the files up and run the Liminal DAGs.
+
+The process of liminal is emphasized on the following diagram:
+
+
+
+## Prerequisites
+Before you begin, ensure you have met the following requirements:
+* You have the kubectl command line [(kubectl CLI)][homebrew-kubectl] installed
+* You have the current [context][cluster-access-kubeconfig] in kubernetes' kubeconfig file
+* You have [Airflow on Kubernetes with AWS EFS][airflowInstallation]
+
+### Supported Distributions
+
+|Distribution | Versions |
+|-|-|
+|[Airflow][airflowImage] | apache/airflow:1.10.14-python3.8 |
+|[Airflow Helm Chart][airflowChart] | [7.16.0][airflow-helm-chart-7.16.0] |
+
+#### You may need to add the Repo
+
+```sh
+helm repo add airflow-stable https://airflow-helm.github.io/charts
+helm repo update
+```
+
+## Get Started
+#### Package dependency
+There are multiple ways to install liminal in Airflow and it depends on what will be useful in your environment.\
+1. Use kubectl to execute pip install on the pods.
+2. Add `apache-liminal` as a requirement package to be installed once Airflow starts.
+
+#### Mount the EFS
+There are a couple of ways to deploy Yamls in Airflow our goal is to deploy the actual Yamls (DAGs) into airflow - by placing the Yaml files in each components\` `/opt/airflow/dags` folder.
+
+There are multiple ways to achieve this.
Review comment:
re-write the previous + this sentence to be less redundant.
Perhaps:
As we said in the intro, our goal is to deploy the user's Yamls (Dags) into Airflow's dag folder.
The next sentence ("our recommended"...) covers the rest of it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] naturalett commented on a change in pull request #11: liminal installation doc
Posted by GitBox <gi...@apache.org>.
naturalett commented on a change in pull request #11:
URL: https://github.com/apache/incubator-liminal/pull/11#discussion_r604238128
##########
File path: docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md
##########
@@ -0,0 +1,122 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required bgit y applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Install Liminal in Airflow
+* [Prerequisites](#prerequisites)
+ * [Supported Distributions](#supported-distributions)
+* [Get Started](#Get-Started)
+ * [Scripted Installation](#scripted-installation)
+ * [Manual Installation](#manual-installation)
+* [References and other resources](#references-and-other-resources)
+
+## Prerequisites
+Before you begin, ensure you have met the following requirements:
+
+* You have a Kubernetes cluster running with [Helm][homebrew-helm] (and Tiller if using Helm v2.x) installed
+* You have the kubectl command line [(kubectl CLI)][homebrew-kubectl] installed
+* You have the current [context][cluster-access-kubeconfig] in kubernetes' kubeconfig file
+* You have Airflow on Kubernetes with AWS EFS
+* You have already created [custom DAGs][custom-dag] following the liminal [Getting Started Documentation][liminalGetStarted-doc]
+* Make sure that the [example repository][liminal-getting-started-project] is your workspace
+
+### Supported Distributions
+
+|Distribution | Versions |
+|-|-|
+|[Airflow][airflowImage] | apache/airflow:1.10.12-python3.6 |
+|[Airflow Helm Chart][airflowChart] | 7.14.3 |
+
+#### You may need to add the Repo
+
+```sh
+helm repo add airflow-stable https://airflow-helm.github.io/charts
+helm repo update
+```
+
+## Get Started
+
+### Scripted Installation
+
+[Use `install_liminal_in_airflow_on_kubernetes.sh`][liminal-installation-script] to install liminal in Airflow.
+* Move to the [example repository project][liminal-getting-started-project]
+* Copy the [script][liminal-installation-script] to the [example repository project][liminal-getting-started-project] and run it
+
+
+
+### Manual Installation
+
+Here are a couple of steps of how to install liminal in Airflow on Kubernetes:
+* [Find Airflow components](#Find-Airflow-pod-names)
+* [Install liminal in the components](#Install-liminal-in-Airflow-pods)
+* [Setting up liminal](#Setting-up-liminal)
+
+#### Find Airflow pod names:
+
+```sh
+echo -n "Please enter a namespace: "
+read namespace
+components=("web" "scheduler" "worker")
+namespace=${namespace}
+podNames=()
+for component in ${components[@]}
+do
+ podNames+="$(kubectl get pod -n ${namespace} -l "app=airflow,component=${component}" --no-headers -o custom-columns=":metadata.name")\n"
+
+done
+
+echo -e "The following Airflow pods are:\n${podNames}"
+```
+#### Install liminal in Airflow pods:
Review comment:
No.
I will change it to have a requirements txt in the DAGs folder. It will promise that Airflow will install it on every restart.
In the supported distributions I assume that you deploy Airflow Helm Chart 7.16.0. (which install the requirements txt)
There are other ways to install python packages. (init containers and etc.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] naturalett commented on pull request #11: liminal installation doc
Posted by GitBox <gi...@apache.org>.
naturalett commented on pull request #11:
URL: https://github.com/apache/incubator-liminal/pull/11#issuecomment-751231769
@lior-schachter
I made it more descriptive: kubeconfig, ask users for script input
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] assapin commented on a change in pull request #11: liminal installation doc
Posted by GitBox <gi...@apache.org>.
assapin commented on a change in pull request #11:
URL: https://github.com/apache/incubator-liminal/pull/11#discussion_r543324567
##########
File path: docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md
##########
@@ -0,0 +1,94 @@
+# Install Liminal in Airflow
+* [Prerequisites](#prerequisites)
+ * [Supported Distributions](#supported-distributions)
+* [Get Started](#Get-Started)
+ * [Scripted Installation](#scripted-installation)
+ * [Manual Installation](#manual-installation)
+
+## Prerequisites
+This guide assumes that:
+
+You have a Kubernetes cluster running with Helm (and Tiller if using Helm v2.x) installed.\
Review comment:
suggest to use bullet points
##########
File path: docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md
##########
@@ -0,0 +1,94 @@
+# Install Liminal in Airflow
+* [Prerequisites](#prerequisites)
+ * [Supported Distributions](#supported-distributions)
+* [Get Started](#Get-Started)
+ * [Scripted Installation](#scripted-installation)
+ * [Manual Installation](#manual-installation)
+
+## Prerequisites
+This guide assumes that:
+
+You have a Kubernetes cluster running with Helm (and Tiller if using Helm v2.x) installed.\
+You have the kubectl command line (kubectl CLI) installed.\
+You have already created custom DAGs following the liminal [Getting Started Documentation][liminalGetStarted].
Review comment:
Custom DAG -> Liminal Yaml file.
Generally, we should not use DAG when talking about liminal
##########
File path: docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md
##########
@@ -0,0 +1,103 @@
+# Install Liminal in Airflow
+* [Prerequisites](#prerequisites)
+ * [Supported Distributions](#supported-distributions)
+* [Get Started](#Get-Started)
+ * [Scripted Installation](#scripted-installation)
+ * [Manual Installation](#manual-installation)
+* [References and other resources](#references-and-other-resources)
+
+## Prerequisites
Review comment:
perhaps add more wording/diagram about :
1. what we assume the user's setup looks like before liminal
2. how liminal is actually deployed onto this setup (efs, home folders etc.)
3. you can use my diagram: https://app.diagrams.net/#G1gSjTBkutY5gXCwpx8WB5pHp_vP57xm4G
##########
File path: docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md
##########
@@ -0,0 +1,122 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required bgit y applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Install Liminal in Airflow
+* [Prerequisites](#prerequisites)
+ * [Supported Distributions](#supported-distributions)
+* [Get Started](#Get-Started)
+ * [Scripted Installation](#scripted-installation)
+ * [Manual Installation](#manual-installation)
+* [References and other resources](#references-and-other-resources)
+
+## Prerequisites
+Before you begin, ensure you have met the following requirements:
+
+* You have a Kubernetes cluster running with [Helm][homebrew-helm] (and Tiller if using Helm v2.x) installed
+* You have the kubectl command line [(kubectl CLI)][homebrew-kubectl] installed
+* You have the current [context][cluster-access-kubeconfig] in kubernetes' kubeconfig file
+* You have Airflow on Kubernetes with AWS EFS
+* You have already created [custom DAGs][custom-dag] following the liminal [Getting Started Documentation][liminalGetStarted-doc]
+* Make sure that the [example repository][liminal-getting-started-project] is your workspace
+
+### Supported Distributions
+
+|Distribution | Versions |
+|-|-|
+|[Airflow][airflowImage] | apache/airflow:1.10.12-python3.6 |
+|[Airflow Helm Chart][airflowChart] | 7.14.3 |
+
+#### You may need to add the Repo
+
+```sh
+helm repo add airflow-stable https://airflow-helm.github.io/charts
+helm repo update
+```
+
+## Get Started
+
+### Scripted Installation
+
+[Use `install_liminal_in_airflow_on_kubernetes.sh`][liminal-installation-script] to install liminal in Airflow.
+* Move to the [example repository project][liminal-getting-started-project]
+* Copy the [script][liminal-installation-script] to the [example repository project][liminal-getting-started-project] and run it
+
+
+
+### Manual Installation
+
+Here are a couple of steps of how to install liminal in Airflow on Kubernetes:
+* [Find Airflow components](#Find-Airflow-pod-names)
+* [Install liminal in the components](#Install-liminal-in-Airflow-pods)
+* [Setting up liminal](#Setting-up-liminal)
+
+#### Find Airflow pod names:
+
+```sh
+echo -n "Please enter a namespace: "
+read namespace
+components=("web" "scheduler" "worker")
+namespace=${namespace}
+podNames=()
+for component in ${components[@]}
+do
+ podNames+="$(kubectl get pod -n ${namespace} -l "app=airflow,component=${component}" --no-headers -o custom-columns=":metadata.name")\n"
+
+done
+
+echo -e "The following Airflow pods are:\n${podNames}"
+```
+#### Install liminal in Airflow pods:
Review comment:
is this the best practice in airflow on kubernetes for installing custom packages?
##########
File path: docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md
##########
@@ -0,0 +1,122 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required bgit y applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Install Liminal in Airflow
+* [Prerequisites](#prerequisites)
Review comment:
suggest to put the link to airflow and EFS here...
##########
File path: docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md
##########
@@ -0,0 +1,103 @@
+# Install Liminal in Airflow
+* [Prerequisites](#prerequisites)
+ * [Supported Distributions](#supported-distributions)
+* [Get Started](#Get-Started)
+ * [Scripted Installation](#scripted-installation)
+ * [Manual Installation](#manual-installation)
+* [References and other resources](#references-and-other-resources)
+
+## Prerequisites
+Before you begin, ensure you have met the following requirements:
+
+* You have a Kubernetes cluster running with [Helm][homebrew-helm] (and Tiller if using Helm v2.x) installed
+* You have the kubectl command line [(kubectl CLI)][homebrew-kubectl] installed
+* You have the current [context][cluster-access-kubeconfig] in kubernetes' kubeconfig file
+* You have Airflow on Kubernetes with AWS EFS
+* You have already created [custom DAGs][custom-dag] following the liminal [Getting Started Documentation][liminalGetStarted-doc]
+* Make sure that the [example repository][liminal-getting-started-project] is your workspace
+
+### Supported Distributions
+
+|Distribution | Versions |
+|-|-|
+|[Airflow][airflowImage] | apache/airflow:1.10.12-python3.6 |
+|[Airflow Helm Chart][airflowChart] | 7.14.3 |
+
+#### You may need to add the Repo
+
+```sh
+helm repo add airflow-stable https://airflow-helm.github.io/charts
+helm repo update
+```
+
+## Get Started
+
+### Scripted Installation
+
+[Use `install_liminal_in_airflow_on_kubernetes.sh`][liminal-installation-script] to install liminal in Airflow.
+* Move to the [example repository project][liminal-getting-started-project]
+* Copy the [script][liminal-installation-script] to the [example repository project][liminal-getting-started-project] and run it
+
+
+
+### Manual Installation
+
+Here are a couple of steps of how to install liminal in Airflow on Kubernetes:
+* [Find Airflow components](#Find-Airflow-pod-names)
+* [Install liminal in the components](#Install-liminal-in-Airflow-pods)
+* [Setting up liminal](Setting-up-liminal)
+
+#### Find Airflow pod names:
+
+```sh
+echo -n "Please enter a namespace: "
+read namespace
+components=("web" "scheduler" "worker")
+namespace=${namespace}
+podNames=()
+for component in ${components[@]}
+do
+ podNames+="$(kubectl get pod -n ${namespace} -l "app=airflow,component=${component}" --no-headers -o custom-columns=":metadata.name")\n"
+
+done
+
+echo -e "The following Airflow pods are:\n${podNames}"
+```
+#### Install liminal in Airflow pods:
+
+```sh
+IFS=$'\n'|''; set -f; podNames=( ${podNames} )
+webPodName=${podNames[${#podNames[@]} - 1]}
+
+for podName in `echo -e ${podNames[@]}`
+do
+ echo "Installing liminal in pod: ${podName}"
+ kubectl exec -it -n ${namespace} ${podName} -- bash -c 'pip install --user apache-liminal'
+done
+```
+
+#### Setting up liminal:
+
+```sh
+echo "Deploying liminal"
Review comment:
My impression was that we are saying to users:
" we assume your dags sit on a mount in EFS which is mounted to all airflow pods"
in this case:
the machine which deploys the dags also needs to mount the same EFS
and define liminal home as sitting on the EFS
then all the airflow pods will be able to see the DAGs on their file system and pick it up
What I see you did here is to implement deploying the DAGs as kubectl cp
which is of course also possible but then why do we assume EFS?
Also all dags need as far as I can remember to be avail also in the scheduler pods
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] assapin merged pull request #11: liminal installation doc
Posted by GitBox <gi...@apache.org>.
assapin merged pull request #11:
URL: https://github.com/apache/incubator-liminal/pull/11
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] naturalett commented on a change in pull request #11: liminal installation doc
Posted by GitBox <gi...@apache.org>.
naturalett commented on a change in pull request #11:
URL: https://github.com/apache/incubator-liminal/pull/11#discussion_r594712358
##########
File path: docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md
##########
@@ -0,0 +1,122 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required bgit y applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Install Liminal in Airflow
+* [Prerequisites](#prerequisites)
Review comment:
I added a reference link to a blog that describes how to [Setting up Airflow on Kubernetes with AWS EFS](https://github.com/apache/incubator-liminal/blob/b8cc352e970335c34001ecb11c420830bb9a0af6/docs/source/How_to_install_liminal_in_airflow_on_kubernetes.md#references-and-other-resources)
Should I still add it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org