You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@liminal.apache.org by GitBox <gi...@apache.org> on 2021/07/12 22:44:38 UTC
[GitHub] [incubator-liminal] naturalett opened a new pull request #57: iris getting started
naturalett opened a new pull request #57:
URL: https://github.com/apache/incubator-liminal/pull/57
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@liminal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] liorsav commented on a change in pull request #57: iris getting started
Posted by GitBox <gi...@apache.org>.
liorsav commented on a change in pull request #57:
URL: https://github.com/apache/incubator-liminal/pull/57#discussion_r676143721
##########
File path: docs/getting-started/iris_classification.md
##########
@@ -0,0 +1,291 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Getting started / ***Iris Classification***
+
+In this tutorial, we will guide you through setting up Apache Liminal on your local machine and run the simple machine-learning workflow example.
+
+* [Setup your local environment](#Setup-your-local-environment)
+* [Setup liminal](#setup-liminal)
+ * [Liminal build](#Liminal-build)
+ * [Liminal create](#Liminal-create)
+ * [Liminal deploy](#Liminal-deploy)
+ * [Liminal start](#Liminal-start)
+* [Liminal YAML walkthrough](#Liminal-YAML-walkthrough)
+* [Evaluate the Iris Classification model](#Evaluate-the-iris-classification-model)
+* [Debugging Kubernetes Deployments](#Debugging-Kubernetes-Deployments)
+* [Closing up](#Closing-up)
+
+
+#### Prerequisites
+
+* [Python 3 (3.6 and up)](https://www.python.org/downloads)
+* [Python Virtual Environments](https://pypi.org/project/virtualenv)
+* [Docker Desktop](https://www.docker.com/products/docker-desktop)
+
+*Note: Make sure kubernetes cluster is running in docker desktop*
+
+We will define the following steps and services to implement the Iris classification example: \
+Train, Validate & Deploy - Training and validation execution is managed by Liminal Airflow extension. The training task trains a regression model using a public dataset. \
+We then validate the model and deploy it to a model-store in mounted volume. \
+Inference - online inference is done using a Python Flask service running on the local Kubernetes in docker desktop. The service exposes the `/predict` endpoint. It reads the model stored in the mounted drive and uses it to evaluate the request.
+## Setup your local env environment
+
+In the dev folder, clone the example code from liminal:
+
+
+```BASH
+git clone https://github.com/apache/incubator-liminal
+```
+***Note:*** *You just cloned the entire Liminal Project, you actually only need examples folder.*
+
+
+
+Create a python virtual environment to isolate your runs:
+
+```BASH
+cd incubator-liminal/examples/aws-ml-app-demo
+python3 -m venv env
+```
+
+Activate your virtual environment:
+
+```BASH
+source env/bin/activate
+```
+
+Now we are ready to install liminal:
+
+```BASH
+pip install apache-liminal
+```
+
+## Setup liminal
+### Liminal build
+The build will create docker images based on the liminal.yml file in the `images` section.
+```BASH
+liminal build
+```
+
+### Liminal create
+All tasks use a mounted volume as defined in the pipeline YAML. \
+In our case the mounted volume will point to the liminal Iris Classification example.
+The training task trains a regression model using a public dataset. We then validate the model and deploy it to a model-store in the mounted volume.
+
+Create a kubernetes local volume:
+```BASH
+liminal create
+```
+
+### Liminal deploy
+The deploy command deploys a liminal server and deploys any liminal.yml files in your working directory or any of its subdirectories to your liminal home directory.
+```BASH
+liminal deploy --clean
+```
+
+*Note: liminal home directory is located in the path defined in LIMINAL_HOME env variable.
+If the LIMINAL_HOME environemnet variable is not defined, home directory defaults to
+~/liminal_home directory.*
+
+### Liminal start
+The start command spins up 3 containers that load the Apache Airflow stack. Liminal's Airflow extension is responsible to execute the workflows defined in the liminal.yml file as standard Airflow DAGs.
+```BASH
+liminal start
+```
+
+It runs the following three containers:
+* liminal-postgress
+* liminal-webserver
+* liminal-scheduler
+
+Once liminal server has completed starting up, you can navigate to admin UI in your browser:
+[http://localhost:8080](http://localhost:8080)
+
+
+
+
+***Important:** Set off/on toggle to activate your pipeline (DAG), nothing will happen otherwise!*
+
+You can go to graph view to see all the tasks configured in the liminal.yml file:
+[http://localhost:8080/admin/airflow/graph?dag_id=my_datascience_pipeline](
+http://localhost:8080/admin/airflow/graph?dag_id=my_datascience_pipeline
+)
+
+#### Now lets see what actually happened to our task:
+
+
+
+#### Click on “train” and you will get this popup:
+
+
+
+#### Click on “view log” button and you can see the log of the current task run:
+
+
+## Liminal YAML walkthrough
+* [Mounted volumes](#Mounted-volumes)
+* [Pipeline flow](#Pipeline-flow)
+
+### Mounted volumes
+Describe of the mounted volume in your liminal YAML:
+```YAML
+name: MyDataScienceApp
+owner: Bosco Albert Baracus
+volumes:
+ - volume: gettingstartedvol
+ claim_name: gettingstartedvol-pvc
+ local:
+ path: .
+```
+
+### Pipeline flow
+Setting up the pipeline flow with the following tasks:
+```YAML
+pipelines:
+ - pipeline: my_datascience_pipeline
+ ...
+ schedule: 0 * 1 * *
+ tasks:
+ - task: train
+ type: python
+ description: train model
+ image: myorg/mydatascienceapp
+ cmd: python -u training.py train
+ ...
+ - task: validate
+ type: python
+ description: validate model and deploy
+ image: myorg/mydatascienceapp
+ cmd: python -u training.py validate
+ ...
+```
+
+##### Each task will internally mount the volume defined above to an internal representation, described under the task section in the yml:
+
+```YAML
+pipelines:
+ ...
+ tasks:
+ - task: train
+ ...
+ env:
+ MOUNT_PATH: /mnt/gettingstartedvol
+ mounts:
+ - mount: mymount
+ volume: gettingstartedvol
+ path: /mnt/gettingstartedvol
+```
+###### We specify the `MOUNT_PATH` in which we store the trained model.
+
+## Evaluate the iris classification model
+
+Once the iris classification model is complete, you can launch a pod of the pre-built image which contains a flask server:
Review comment:
Once the iris classification model is complete, you can launch a pod of the pre-built image which contains a flask server:
replace with
Once the iris classification model training is completed and model is deployed (to the mounted volume), you can launch a pod of the pre-built image which contains a flask server:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@liminal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] liorsav commented on a change in pull request #57: iris getting started
Posted by GitBox <gi...@apache.org>.
liorsav commented on a change in pull request #57:
URL: https://github.com/apache/incubator-liminal/pull/57#discussion_r673114703
##########
File path: docs/getting-started/iris_classification.md
##########
@@ -0,0 +1,274 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Getting started / ***Iris Classification***
+
+This guide will allow you to set up your first Apache Liminal environment and allow you to create
+some simple ML pipelines. These will be very similar to the ones you are going to build for real
+production scenarios.
+
+## Prerequisites
+
+Python 3 (3.6 and up)
Review comment:
add link as you did for docker desktop
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@liminal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] simlev edited a comment on pull request #57: iris getting started
Posted by GitBox <gi...@apache.org>.
simlev edited a comment on pull request #57:
URL: https://github.com/apache/incubator-liminal/pull/57#issuecomment-879098448
Can we also add a Troubleshooting section?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@liminal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] liorsav commented on pull request #57: iris getting started
Posted by GitBox <gi...@apache.org>.
liorsav commented on pull request #57:
URL: https://github.com/apache/incubator-liminal/pull/57#issuecomment-886395993
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@liminal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] liorsav commented on a change in pull request #57: iris getting started
Posted by GitBox <gi...@apache.org>.
liorsav commented on a change in pull request #57:
URL: https://github.com/apache/incubator-liminal/pull/57#discussion_r673289825
##########
File path: docs/getting-started/iris_classification.md
##########
@@ -0,0 +1,274 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Getting started / ***Iris Classification***
+
+This guide will allow you to set up your first Apache Liminal environment and allow you to create
+some simple ML pipelines. These will be very similar to the ones you are going to build for real
+production scenarios.
+
+## Prerequisites
+
+Python 3 (3.6 and up)
+
+[Docker Desktop](https://www.docker.com/products/docker-desktop)
+
+*Note: Make sure kubernetes cluster is running in docker desktop (or custom kubernetes installation
+on your machine).*
+
+## Deploying the Example
+
+In this tutorial, we will go through setting up Liminal for the first time on your local machine.
+
+### First, let’s build our example project:
+
+In the dev folder, just clone the example code from liminal:
+
+
+```BASH
+git clone https://github.com/apache/incubator-liminal
+```
+***Note:*** *You just cloned the entire Liminal Project, you actually only need examples folder.*
+
+Create a python virtual environment to isolate your runs:
+
+```BASH
+cd incubator-liminal/examples/aws-ml-app-demo
+python3 -m venv env
+```
+
+Activate your virtual environment:
+
+```BASH
+source env/bin/activate
+```
+
+Now we are ready to install liminal:
+
+```BASH
+pip install apache-liminal
+```
+Let's build the images you need for the example:
+```BASH
+liminal build
+```
+##### The build will create docker images based on the liminal.yml file in the `images` section.
+
+
+Create a kubernetes local volume:
+```BASH
+liminal create
+```
+
+
+The deploy command deploys a liminal server and deploys any liminal.yml files in your working
+directory or any of its subdirectories to your liminal home directory.
+```BASH
+liminal deploy --clean
+```
+
+
+*Note: liminal home directory is located in the path defined in LIMINAL_HOME env variable.
+If the LIMINAL_HOME environemnet variable is not defined, home directory defaults to
+~/liminal_home directory.*
+
+Now lets runs liminal:
+```BASH
+liminal start
+```
+The start command spins up the liminal server containers which will run pipelines based on your
+deployed liminal.yml files.
+It runs the following three containers:
+* liminal-postgress
+* liminal-webserver
+* liminal-scheduler
+
+Once liminal server has completed starting up, you can navigate to admin UI in your browser:
+[http://localhost:8080](http://localhost:8080)
+By default liminal server starts Apache Airflow servers and admin UI will be that of Apache Airflow.
Review comment:
remove this line:
By default liminal server starts Apache Airflow servers and admin UI will be that of Apache Airflow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@liminal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] liorsav commented on a change in pull request #57: iris getting started
Posted by GitBox <gi...@apache.org>.
liorsav commented on a change in pull request #57:
URL: https://github.com/apache/incubator-liminal/pull/57#discussion_r673922792
##########
File path: docs/getting-started/iris_classification.md
##########
@@ -0,0 +1,274 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Getting started / ***Iris Classification***
+
+This guide will allow you to set up your first Apache Liminal environment and allow you to create
+some simple ML pipelines. These will be very similar to the ones you are going to build for real
+production scenarios.
+
+## Prerequisites
+
+Python 3 (3.6 and up)
+
+[Docker Desktop](https://www.docker.com/products/docker-desktop)
+
+*Note: Make sure kubernetes cluster is running in docker desktop (or custom kubernetes installation
+on your machine).*
+
+## Deploying the Example
+
+In this tutorial, we will go through setting up Liminal for the first time on your local machine.
+
+### First, let’s build our example project:
+
+In the dev folder, just clone the example code from liminal:
Review comment:
remove 'just'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@liminal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] liorsav commented on a change in pull request #57: iris getting started
Posted by GitBox <gi...@apache.org>.
liorsav commented on a change in pull request #57:
URL: https://github.com/apache/incubator-liminal/pull/57#discussion_r672410485
##########
File path: docs/README.md
##########
@@ -35,10 +35,11 @@ Using simple YAML configuration, create your own schedule data pipelines (a sequ
perform), application servers, and more.
## Getting Started
-A simple getting stated guide for Liminal can be found [here](getting_started.md)
+A simple hello world guide for Liminal can be found [here](getting-started/hello_world.md) \
+A simple Iris classification model guide for Liminal can be found [here](getting-started/iris_classification.md)
Review comment:
A more advanced example which demonstrate a simple data-science workflow can be found [here](getting-started/iris_classification.md
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@liminal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] liorsav commented on a change in pull request #57: iris getting started
Posted by GitBox <gi...@apache.org>.
liorsav commented on a change in pull request #57:
URL: https://github.com/apache/incubator-liminal/pull/57#discussion_r673171798
##########
File path: docs/getting-started/iris_classification.md
##########
@@ -0,0 +1,274 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Getting started / ***Iris Classification***
+
+This guide will allow you to set up your first Apache Liminal environment and allow you to create
+some simple ML pipelines. These will be very similar to the ones you are going to build for real
+production scenarios.
+
+## Prerequisites
+
+Python 3 (3.6 and up)
+
+[Docker Desktop](https://www.docker.com/products/docker-desktop)
+
+*Note: Make sure kubernetes cluster is running in docker desktop (or custom kubernetes installation
+on your machine).*
+
+## Deploying the Example
+
+In this tutorial, we will go through setting up Liminal for the first time on your local machine.
Review comment:
remove this. as you already write the same above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@liminal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] liorsav commented on a change in pull request #57: iris getting started
Posted by GitBox <gi...@apache.org>.
liorsav commented on a change in pull request #57:
URL: https://github.com/apache/incubator-liminal/pull/57#discussion_r673940123
##########
File path: docs/getting-started/iris_classification.md
##########
@@ -0,0 +1,274 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Getting started / ***Iris Classification***
+
+This guide will allow you to set up your first Apache Liminal environment and allow you to create
+some simple ML pipelines. These will be very similar to the ones you are going to build for real
+production scenarios.
+
+## Prerequisites
+
+Python 3 (3.6 and up)
+
+[Docker Desktop](https://www.docker.com/products/docker-desktop)
+
+*Note: Make sure kubernetes cluster is running in docker desktop (or custom kubernetes installation
+on your machine).*
+
+## Deploying the Example
+
+In this tutorial, we will go through setting up Liminal for the first time on your local machine.
+
+### First, let’s build our example project:
+
+In the dev folder, just clone the example code from liminal:
+
+
+```BASH
+git clone https://github.com/apache/incubator-liminal
+```
+***Note:*** *You just cloned the entire Liminal Project, you actually only need examples folder.*
+
+Create a python virtual environment to isolate your runs:
+
+```BASH
+cd incubator-liminal/examples/aws-ml-app-demo
+python3 -m venv env
+```
+
+Activate your virtual environment:
+
+```BASH
+source env/bin/activate
+```
+
+Now we are ready to install liminal:
+
+```BASH
+pip install apache-liminal
+```
+Let's build the images you need for the example:
+```BASH
+liminal build
+```
+##### The build will create docker images based on the liminal.yml file in the `images` section.
+
+
+Create a kubernetes local volume:
+```BASH
+liminal create
+```
+
+
+The deploy command deploys a liminal server and deploys any liminal.yml files in your working
+directory or any of its subdirectories to your liminal home directory.
+```BASH
+liminal deploy --clean
+```
+
+
+*Note: liminal home directory is located in the path defined in LIMINAL_HOME env variable.
+If the LIMINAL_HOME environemnet variable is not defined, home directory defaults to
+~/liminal_home directory.*
+
+Now lets runs liminal:
+```BASH
+liminal start
+```
+The start command spins up the liminal server containers which will run pipelines based on your
+deployed liminal.yml files.
+It runs the following three containers:
+* liminal-postgress
+* liminal-webserver
+* liminal-scheduler
+
+Once liminal server has completed starting up, you can navigate to admin UI in your browser:
+[http://localhost:8080](http://localhost:8080)
+By default liminal server starts Apache Airflow servers and admin UI will be that of Apache Airflow.
+
+
+
+
+***Important:** Set off/on toggle to activate your pipeline (DAG), nothing will happen otherwise!*
+
+You can go to graph view to see all the tasks configured in the liminal.yml file:
+[http://localhost:8080/admin/airflow/graph?dag_id=my_datascience_pipeline](
+http://localhost:8080/admin/airflow/graph?dag_id=my_datascience_pipeline
+)
+
+#### Now lets see what actually happened to our task:
+
+
+
+#### Click on “train” and you will get this popup:
+
+
+
+#### Click on “view log” button and you can see the log of the current task run:
+
+
Review comment:
liminal.yml walkthrough
pipelines
....
mount drive definition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@liminal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] liorsav commented on a change in pull request #57: iris getting started
Posted by GitBox <gi...@apache.org>.
liorsav commented on a change in pull request #57:
URL: https://github.com/apache/incubator-liminal/pull/57#discussion_r676143721
##########
File path: docs/getting-started/iris_classification.md
##########
@@ -0,0 +1,291 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Getting started / ***Iris Classification***
+
+In this tutorial, we will guide you through setting up Apache Liminal on your local machine and run the simple machine-learning workflow example.
+
+* [Setup your local environment](#Setup-your-local-environment)
+* [Setup liminal](#setup-liminal)
+ * [Liminal build](#Liminal-build)
+ * [Liminal create](#Liminal-create)
+ * [Liminal deploy](#Liminal-deploy)
+ * [Liminal start](#Liminal-start)
+* [Liminal YAML walkthrough](#Liminal-YAML-walkthrough)
+* [Evaluate the Iris Classification model](#Evaluate-the-iris-classification-model)
+* [Debugging Kubernetes Deployments](#Debugging-Kubernetes-Deployments)
+* [Closing up](#Closing-up)
+
+
+#### Prerequisites
+
+* [Python 3 (3.6 and up)](https://www.python.org/downloads)
+* [Python Virtual Environments](https://pypi.org/project/virtualenv)
+* [Docker Desktop](https://www.docker.com/products/docker-desktop)
+
+*Note: Make sure kubernetes cluster is running in docker desktop*
+
+We will define the following steps and services to implement the Iris classification example: \
+Train, Validate & Deploy - Training and validation execution is managed by Liminal Airflow extension. The training task trains a regression model using a public dataset. \
+We then validate the model and deploy it to a model-store in mounted volume. \
+Inference - online inference is done using a Python Flask service running on the local Kubernetes in docker desktop. The service exposes the `/predict` endpoint. It reads the model stored in the mounted drive and uses it to evaluate the request.
+## Setup your local env environment
+
+In the dev folder, clone the example code from liminal:
+
+
+```BASH
+git clone https://github.com/apache/incubator-liminal
+```
+***Note:*** *You just cloned the entire Liminal Project, you actually only need examples folder.*
+
+
+
+Create a python virtual environment to isolate your runs:
+
+```BASH
+cd incubator-liminal/examples/aws-ml-app-demo
+python3 -m venv env
+```
+
+Activate your virtual environment:
+
+```BASH
+source env/bin/activate
+```
+
+Now we are ready to install liminal:
+
+```BASH
+pip install apache-liminal
+```
+
+## Setup liminal
+### Liminal build
+The build will create docker images based on the liminal.yml file in the `images` section.
+```BASH
+liminal build
+```
+
+### Liminal create
+All tasks use a mounted volume as defined in the pipeline YAML. \
+In our case the mounted volume will point to the liminal Iris Classification example.
+The training task trains a regression model using a public dataset. We then validate the model and deploy it to a model-store in the mounted volume.
+
+Create a kubernetes local volume:
+```BASH
+liminal create
+```
+
+### Liminal deploy
+The deploy command deploys a liminal server and deploys any liminal.yml files in your working directory or any of its subdirectories to your liminal home directory.
+```BASH
+liminal deploy --clean
+```
+
+*Note: liminal home directory is located in the path defined in LIMINAL_HOME env variable.
+If the LIMINAL_HOME environemnet variable is not defined, home directory defaults to
+~/liminal_home directory.*
+
+### Liminal start
+The start command spins up 3 containers that load the Apache Airflow stack. Liminal's Airflow extension is responsible to execute the workflows defined in the liminal.yml file as standard Airflow DAGs.
+```BASH
+liminal start
+```
+
+It runs the following three containers:
+* liminal-postgress
+* liminal-webserver
+* liminal-scheduler
+
+Once liminal server has completed starting up, you can navigate to admin UI in your browser:
+[http://localhost:8080](http://localhost:8080)
+
+
+
+
+***Important:** Set off/on toggle to activate your pipeline (DAG), nothing will happen otherwise!*
+
+You can go to graph view to see all the tasks configured in the liminal.yml file:
+[http://localhost:8080/admin/airflow/graph?dag_id=my_datascience_pipeline](
+http://localhost:8080/admin/airflow/graph?dag_id=my_datascience_pipeline
+)
+
+#### Now lets see what actually happened to our task:
+
+
+
+#### Click on “train” and you will get this popup:
+
+
+
+#### Click on “view log” button and you can see the log of the current task run:
+
+
+## Liminal YAML walkthrough
+* [Mounted volumes](#Mounted-volumes)
+* [Pipeline flow](#Pipeline-flow)
+
+### Mounted volumes
+Describe of the mounted volume in your liminal YAML:
+```YAML
+name: MyDataScienceApp
+owner: Bosco Albert Baracus
+volumes:
+ - volume: gettingstartedvol
+ claim_name: gettingstartedvol-pvc
+ local:
+ path: .
+```
+
+### Pipeline flow
+Setting up the pipeline flow with the following tasks:
+```YAML
+pipelines:
+ - pipeline: my_datascience_pipeline
+ ...
+ schedule: 0 * 1 * *
+ tasks:
+ - task: train
+ type: python
+ description: train model
+ image: myorg/mydatascienceapp
+ cmd: python -u training.py train
+ ...
+ - task: validate
+ type: python
+ description: validate model and deploy
+ image: myorg/mydatascienceapp
+ cmd: python -u training.py validate
+ ...
+```
+
+##### Each task will internally mount the volume defined above to an internal representation, described under the task section in the yml:
+
+```YAML
+pipelines:
+ ...
+ tasks:
+ - task: train
+ ...
+ env:
+ MOUNT_PATH: /mnt/gettingstartedvol
+ mounts:
+ - mount: mymount
+ volume: gettingstartedvol
+ path: /mnt/gettingstartedvol
+```
+###### We specify the `MOUNT_PATH` in which we store the trained model.
+
+## Evaluate the iris classification model
+
+Once the iris classification model is complete, you can launch a pod of the pre-built image which contains a flask server:
Review comment:
Once the iris classification model is complete, you can launch a pod of the pre-built image which contains a flask server:
replace with
Once the iris classification model training is completed and model is deployed (to the mounted volume), you can launch a pod of the pre-built image which contains a flask server, by applying the following Kubernetes configuration:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@liminal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] liorsav commented on a change in pull request #57: iris getting started
Posted by GitBox <gi...@apache.org>.
liorsav commented on a change in pull request #57:
URL: https://github.com/apache/incubator-liminal/pull/57#discussion_r673921678
##########
File path: docs/getting-started/iris_classification.md
##########
@@ -0,0 +1,274 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Getting started / ***Iris Classification***
+
+This guide will allow you to set up your first Apache Liminal environment and allow you to create
Review comment:
remove this paragraph and replace it with the comment above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@liminal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] liorsav commented on a change in pull request #57: iris getting started
Posted by GitBox <gi...@apache.org>.
liorsav commented on a change in pull request #57:
URL: https://github.com/apache/incubator-liminal/pull/57#discussion_r676118869
##########
File path: docs/getting-started/iris_classification.md
##########
@@ -0,0 +1,291 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Getting started / ***Iris Classification***
+
+In this tutorial, we will guide you through setting up Apache Liminal on your local machine and run the simple machine-learning workflow example.
+
Review comment:
Please move it to the top of page.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@liminal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] liorsav commented on a change in pull request #57: iris getting started
Posted by GitBox <gi...@apache.org>.
liorsav commented on a change in pull request #57:
URL: https://github.com/apache/incubator-liminal/pull/57#discussion_r676118998
##########
File path: docs/getting-started/iris_classification.md
##########
@@ -0,0 +1,291 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Getting started / ***Iris Classification***
+
+In this tutorial, we will guide you through setting up Apache Liminal on your local machine and run the simple machine-learning workflow example.
+
+* [Setup your local environment](#Setup-your-local-environment)
+* [Setup liminal](#setup-liminal)
+ * [Liminal build](#Liminal-build)
+ * [Liminal create](#Liminal-create)
+ * [Liminal deploy](#Liminal-deploy)
+ * [Liminal start](#Liminal-start)
+* [Liminal YAML walkthrough](#Liminal-YAML-walkthrough)
+* [Evaluate the Iris Classification model](#Evaluate-the-iris-classification-model)
+* [Debugging Kubernetes Deployments](#Debugging-Kubernetes-Deployments)
+* [Closing up](#Closing-up)
+
+
+#### Prerequisites
+
+* [Python 3 (3.6 and up)](https://www.python.org/downloads)
+* [Python Virtual Environments](https://pypi.org/project/virtualenv)
+* [Docker Desktop](https://www.docker.com/products/docker-desktop)
+
+*Note: Make sure kubernetes cluster is running in docker desktop*
+
+We will define the following steps and services to implement the Iris classification example: \
+Train, Validate & Deploy - Training and validation execution is managed by Liminal Airflow extension. The training task trains a regression model using a public dataset. \
+We then validate the model and deploy it to a model-store in mounted volume. \
+Inference - online inference is done using a Python Flask service running on the local Kubernetes in docker desktop. The service exposes the `/predict` endpoint. It reads the model stored in the mounted drive and uses it to evaluate the request.
+## Setup your local env environment
+
+In the dev folder, clone the example code from liminal:
+
+
+```BASH
+git clone https://github.com/apache/incubator-liminal
+```
+***Note:*** *You just cloned the entire Liminal Project, you actually only need examples folder.*
+
+
+
+Create a python virtual environment to isolate your runs:
+
+```BASH
+cd incubator-liminal/examples/aws-ml-app-demo
+python3 -m venv env
+```
+
+Activate your virtual environment:
+
+```BASH
+source env/bin/activate
+```
+
+Now we are ready to install liminal:
+
+```BASH
+pip install apache-liminal
+```
+
+## Setup liminal
+### Liminal build
+The build will create docker images based on the liminal.yml file in the `images` section.
+```BASH
+liminal build
+```
+
+### Liminal create
+All tasks use a mounted volume as defined in the pipeline YAML. \
+In our case the mounted volume will point to the liminal Iris Classification example.
+The training task trains a regression model using a public dataset. We then validate the model and deploy it to a model-store in the mounted volume.
+
+Create a kubernetes local volume:
+```BASH
+liminal create
+```
+
+### Liminal deploy
+The deploy command deploys a liminal server and deploys any liminal.yml files in your working directory or any of its subdirectories to your liminal home directory.
+```BASH
+liminal deploy --clean
+```
+
+*Note: liminal home directory is located in the path defined in LIMINAL_HOME env variable.
+If the LIMINAL_HOME environemnet variable is not defined, home directory defaults to
+~/liminal_home directory.*
+
+### Liminal start
+The start command spins up 3 containers that load the Apache Airflow stack. Liminal's Airflow extension is responsible to execute the workflows defined in the liminal.yml file as standard Airflow DAGs.
+```BASH
+liminal start
+```
+
+It runs the following three containers:
+* liminal-postgress
+* liminal-webserver
+* liminal-scheduler
+
+Once liminal server has completed starting up, you can navigate to admin UI in your browser:
+[http://localhost:8080](http://localhost:8080)
+
+
+
+
+***Important:** Set off/on toggle to activate your pipeline (DAG), nothing will happen otherwise!*
+
+You can go to graph view to see all the tasks configured in the liminal.yml file:
+[http://localhost:8080/admin/airflow/graph?dag_id=my_datascience_pipeline](
+http://localhost:8080/admin/airflow/graph?dag_id=my_datascience_pipeline
+)
+
+#### Now lets see what actually happened to our task:
+
+
+
+#### Click on “train” and you will get this popup:
+
+
+
+#### Click on “view log” button and you can see the log of the current task run:
+
+
+## Liminal YAML walkthrough
+* [Mounted volumes](#Mounted-volumes)
+* [Pipeline flow](#Pipeline-flow)
+
+### Mounted volumes
+Describe of the mounted volume in your liminal YAML:
+```YAML
+name: MyDataScienceApp
+owner: Bosco Albert Baracus
+volumes:
+ - volume: gettingstartedvol
+ claim_name: gettingstartedvol-pvc
+ local:
+ path: .
+```
+
+### Pipeline flow
+Setting up the pipeline flow with the following tasks:
+```YAML
+pipelines:
+ - pipeline: my_datascience_pipeline
+ ...
+ schedule: 0 * 1 * *
+ tasks:
+ - task: train
+ type: python
+ description: train model
+ image: myorg/mydatascienceapp
+ cmd: python -u training.py train
+ ...
+ - task: validate
+ type: python
+ description: validate model and deploy
+ image: myorg/mydatascienceapp
+ cmd: python -u training.py validate
+ ...
+```
+
+##### Each task will internally mount the volume defined above to an internal representation, described under the task section in the yml:
+
+```YAML
+pipelines:
+ ...
+ tasks:
+ - task: train
+ ...
+ env:
+ MOUNT_PATH: /mnt/gettingstartedvol
+ mounts:
+ - mount: mymount
+ volume: gettingstartedvol
+ path: /mnt/gettingstartedvol
+```
+###### We specify the `MOUNT_PATH` in which we store the trained model.
+
+## Evaluate the iris classification model
+
+Once the iris classification model is complete, you can launch a pod of the pre-built image which contains a flask server:
Review comment:
Not clear. How to run this part. lets talk.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@liminal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] simlev commented on pull request #57: iris getting started
Posted by GitBox <gi...@apache.org>.
simlev commented on pull request #57:
URL: https://github.com/apache/incubator-liminal/pull/57#issuecomment-879098448
Can we also add a Troubleshooting section.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@liminal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] liorsav commented on a change in pull request #57: iris getting started
Posted by GitBox <gi...@apache.org>.
liorsav commented on a change in pull request #57:
URL: https://github.com/apache/incubator-liminal/pull/57#discussion_r673115712
##########
File path: docs/getting-started/iris_classification.md
##########
@@ -0,0 +1,274 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Getting started / ***Iris Classification***
+
+This guide will allow you to set up your first Apache Liminal environment and allow you to create
+some simple ML pipelines. These will be very similar to the ones you are going to build for real
+production scenarios.
+
+## Prerequisites
+
Review comment:
please add virtualenv. you are using it below.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@liminal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] liorsav commented on a change in pull request #57: iris getting started
Posted by GitBox <gi...@apache.org>.
liorsav commented on a change in pull request #57:
URL: https://github.com/apache/incubator-liminal/pull/57#discussion_r672410485
##########
File path: docs/README.md
##########
@@ -35,10 +35,11 @@ Using simple YAML configuration, create your own schedule data pipelines (a sequ
perform), application servers, and more.
## Getting Started
-A simple getting stated guide for Liminal can be found [here](getting_started.md)
+A simple hello world guide for Liminal can be found [here](getting-started/hello_world.md) \
+A simple Iris classification model guide for Liminal can be found [here](getting-started/iris_classification.md)
Review comment:
A more advanced example which demonstrate a simple data-science workflow can be found [here](getting-started/iris_classification.md
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@liminal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] liorsav commented on a change in pull request #57: iris getting started
Posted by GitBox <gi...@apache.org>.
liorsav commented on a change in pull request #57:
URL: https://github.com/apache/incubator-liminal/pull/57#discussion_r673281133
##########
File path: docs/getting-started/iris_classification.md
##########
@@ -0,0 +1,274 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Getting started / ***Iris Classification***
+
+This guide will allow you to set up your first Apache Liminal environment and allow you to create
+some simple ML pipelines. These will be very similar to the ones you are going to build for real
+production scenarios.
+
+## Prerequisites
+
+Python 3 (3.6 and up)
+
+[Docker Desktop](https://www.docker.com/products/docker-desktop)
+
+*Note: Make sure kubernetes cluster is running in docker desktop (or custom kubernetes installation
+on your machine).*
+
+## Deploying the Example
+
+In this tutorial, we will go through setting up Liminal for the first time on your local machine.
+
+### First, let’s build our example project:
+
+In the dev folder, just clone the example code from liminal:
+
+
+```BASH
+git clone https://github.com/apache/incubator-liminal
+```
+***Note:*** *You just cloned the entire Liminal Project, you actually only need examples folder.*
+
+Create a python virtual environment to isolate your runs:
+
+```BASH
+cd incubator-liminal/examples/aws-ml-app-demo
+python3 -m venv env
+```
+
+Activate your virtual environment:
+
+```BASH
+source env/bin/activate
+```
+
+Now we are ready to install liminal:
+
+```BASH
+pip install apache-liminal
+```
+Let's build the images you need for the example:
+```BASH
+liminal build
+```
+##### The build will create docker images based on the liminal.yml file in the `images` section.
+
+
+Create a kubernetes local volume:
+```BASH
+liminal create
+```
+
+
+The deploy command deploys a liminal server and deploys any liminal.yml files in your working
+directory or any of its subdirectories to your liminal home directory.
+```BASH
+liminal deploy --clean
+```
+
+
+*Note: liminal home directory is located in the path defined in LIMINAL_HOME env variable.
+If the LIMINAL_HOME environemnet variable is not defined, home directory defaults to
+~/liminal_home directory.*
+
+Now lets runs liminal:
+```BASH
+liminal start
+```
+The start command spins up the liminal server containers which will run pipelines based on your
+deployed liminal.yml files.
Review comment:
The start command spins up 3 containers which will run the workflows defined in the
liminal.yml file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@liminal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] liorsav commented on a change in pull request #57: iris getting started
Posted by GitBox <gi...@apache.org>.
liorsav commented on a change in pull request #57:
URL: https://github.com/apache/incubator-liminal/pull/57#discussion_r673938780
##########
File path: docs/getting-started/iris_classification.md
##########
@@ -0,0 +1,274 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Getting started / ***Iris Classification***
+
+This guide will allow you to set up your first Apache Liminal environment and allow you to create
+some simple ML pipelines. These will be very similar to the ones you are going to build for real
+production scenarios.
+
+## Prerequisites
+
+Python 3 (3.6 and up)
+
+[Docker Desktop](https://www.docker.com/products/docker-desktop)
+
+*Note: Make sure kubernetes cluster is running in docker desktop (or custom kubernetes installation
+on your machine).*
+
+## Deploying the Example
+
+In this tutorial, we will go through setting up Liminal for the first time on your local machine.
+
+### First, let’s build our example project:
+
+In the dev folder, just clone the example code from liminal:
+
+
+```BASH
+git clone https://github.com/apache/incubator-liminal
+```
+***Note:*** *You just cloned the entire Liminal Project, you actually only need examples folder.*
+
+Create a python virtual environment to isolate your runs:
+
+```BASH
+cd incubator-liminal/examples/aws-ml-app-demo
+python3 -m venv env
+```
+
+Activate your virtual environment:
+
+```BASH
+source env/bin/activate
+```
+
+Now we are ready to install liminal:
+
+```BASH
+pip install apache-liminal
+```
+Let's build the images you need for the example:
+```BASH
+liminal build
+```
+##### The build will create docker images based on the liminal.yml file in the `images` section.
+
+
+Create a kubernetes local volume:
+```BASH
+liminal create
Review comment:
add the mount description - without the yml.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@liminal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] liorsav commented on a change in pull request #57: iris getting started
Posted by GitBox <gi...@apache.org>.
liorsav commented on a change in pull request #57:
URL: https://github.com/apache/incubator-liminal/pull/57#discussion_r676118058
##########
File path: docs/getting-started/iris_classification.md
##########
@@ -0,0 +1,291 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Getting started / ***Iris Classification***
+
+In this tutorial, we will guide you through setting up Apache Liminal on your local machine and run the simple machine-learning workflow example.
Review comment:
It should be after the ToC.
REPLACE
In this tutorial, we will guide you through setting up Apache Liminal on your local machine and run the simple machine-learning workflow example.
WITH
In this tutorial, we will guide you through setting up Apache Liminal on your local machine and run a simple machine-learning workflow, based on the classic Iris dataset classification example; more details in this [link] (https://scikit-learn.org/stable/auto_examples/datasets/plot_iris_dataset.html).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@liminal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] liorsav commented on a change in pull request #57: iris getting started
Posted by GitBox <gi...@apache.org>.
liorsav commented on a change in pull request #57:
URL: https://github.com/apache/incubator-liminal/pull/57#discussion_r673936699
##########
File path: docs/getting-started/iris_classification.md
##########
@@ -0,0 +1,274 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Getting started / ***Iris Classification***
+
+This guide will allow you to set up your first Apache Liminal environment and allow you to create
+some simple ML pipelines. These will be very similar to the ones you are going to build for real
+production scenarios.
+
+## Prerequisites
+
+Python 3 (3.6 and up)
+
+[Docker Desktop](https://www.docker.com/products/docker-desktop)
+
+*Note: Make sure kubernetes cluster is running in docker desktop (or custom kubernetes installation
+on your machine).*
+
Review comment:
Please add this section:
Example flow:
We will define the following steps and services to implement the Iris classification example:
Train, Validate & Deploy - Training and validation execution is managed by Liminal Airflow extension. The training task trains a regression model using a public dataset. We then validate the model and deploy it to a model-store in mounted volume.
Inference - online inference is done using a Python Flask service running on the local Kubernetes in docker desktop. The service exposes the /predict endpoint. It reads the model stored in the mounted drive and uses it to evaluate the request.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@liminal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] liorsav commented on a change in pull request #57: iris getting started
Posted by GitBox <gi...@apache.org>.
liorsav commented on a change in pull request #57:
URL: https://github.com/apache/incubator-liminal/pull/57#discussion_r673171798
##########
File path: docs/getting-started/iris_classification.md
##########
@@ -0,0 +1,274 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Getting started / ***Iris Classification***
+
+This guide will allow you to set up your first Apache Liminal environment and allow you to create
+some simple ML pipelines. These will be very similar to the ones you are going to build for real
+production scenarios.
+
+## Prerequisites
+
+Python 3 (3.6 and up)
+
+[Docker Desktop](https://www.docker.com/products/docker-desktop)
+
+*Note: Make sure kubernetes cluster is running in docker desktop (or custom kubernetes installation
+on your machine).*
+
+## Deploying the Example
+
+In this tutorial, we will go through setting up Liminal for the first time on your local machine.
Review comment:
remove this. as you already write the same about.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@liminal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] liorsav commented on a change in pull request #57: iris getting started
Posted by GitBox <gi...@apache.org>.
liorsav commented on a change in pull request #57:
URL: https://github.com/apache/incubator-liminal/pull/57#discussion_r673176441
##########
File path: docs/getting-started/iris_classification.md
##########
@@ -0,0 +1,274 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Getting started / ***Iris Classification***
Review comment:
replace :
This guide will allow you to set up your first Apache Liminal environment and allow you to create
some simple ML pipelines. These will be very similar to the ones you are going to build for real
production scenarios.
with:
In this tutorial, we will guide you through setting up Apache Liminal on your local machine and run the a simple machine-learning workflow example.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@liminal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] liorsav commented on a change in pull request #57: iris getting started
Posted by GitBox <gi...@apache.org>.
liorsav commented on a change in pull request #57:
URL: https://github.com/apache/incubator-liminal/pull/57#discussion_r673176441
##########
File path: docs/getting-started/iris_classification.md
##########
@@ -0,0 +1,274 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Getting started / ***Iris Classification***
Review comment:
In this tutorial, we will go through setting up Apache Liminal on your local machine and run the first machine-learning workflow example.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@liminal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] liorsav commented on a change in pull request #57: iris getting started
Posted by GitBox <gi...@apache.org>.
liorsav commented on a change in pull request #57:
URL: https://github.com/apache/incubator-liminal/pull/57#discussion_r673171798
##########
File path: docs/getting-started/iris_classification.md
##########
@@ -0,0 +1,274 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Getting started / ***Iris Classification***
+
+This guide will allow you to set up your first Apache Liminal environment and allow you to create
+some simple ML pipelines. These will be very similar to the ones you are going to build for real
+production scenarios.
+
+## Prerequisites
+
+Python 3 (3.6 and up)
+
+[Docker Desktop](https://www.docker.com/products/docker-desktop)
+
+*Note: Make sure kubernetes cluster is running in docker desktop (or custom kubernetes installation
+on your machine).*
+
+## Deploying the Example
+
+In this tutorial, we will go through setting up Liminal for the first time on your local machine.
Review comment:
remove this. as you already write the same above.
In this tutorial, we will go through setting up Liminal for the first time on your local machine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@liminal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] liorsav commented on a change in pull request #57: iris getting started
Posted by GitBox <gi...@apache.org>.
liorsav commented on a change in pull request #57:
URL: https://github.com/apache/incubator-liminal/pull/57#discussion_r673289825
##########
File path: docs/getting-started/iris_classification.md
##########
@@ -0,0 +1,274 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Getting started / ***Iris Classification***
+
+This guide will allow you to set up your first Apache Liminal environment and allow you to create
+some simple ML pipelines. These will be very similar to the ones you are going to build for real
+production scenarios.
+
+## Prerequisites
+
+Python 3 (3.6 and up)
+
+[Docker Desktop](https://www.docker.com/products/docker-desktop)
+
+*Note: Make sure kubernetes cluster is running in docker desktop (or custom kubernetes installation
+on your machine).*
+
+## Deploying the Example
+
+In this tutorial, we will go through setting up Liminal for the first time on your local machine.
+
+### First, let’s build our example project:
+
+In the dev folder, just clone the example code from liminal:
+
+
+```BASH
+git clone https://github.com/apache/incubator-liminal
+```
+***Note:*** *You just cloned the entire Liminal Project, you actually only need examples folder.*
+
+Create a python virtual environment to isolate your runs:
+
+```BASH
+cd incubator-liminal/examples/aws-ml-app-demo
+python3 -m venv env
+```
+
+Activate your virtual environment:
+
+```BASH
+source env/bin/activate
+```
+
+Now we are ready to install liminal:
+
+```BASH
+pip install apache-liminal
+```
+Let's build the images you need for the example:
+```BASH
+liminal build
+```
+##### The build will create docker images based on the liminal.yml file in the `images` section.
+
+
+Create a kubernetes local volume:
+```BASH
+liminal create
+```
+
+
+The deploy command deploys a liminal server and deploys any liminal.yml files in your working
+directory or any of its subdirectories to your liminal home directory.
+```BASH
+liminal deploy --clean
+```
+
+
+*Note: liminal home directory is located in the path defined in LIMINAL_HOME env variable.
+If the LIMINAL_HOME environemnet variable is not defined, home directory defaults to
+~/liminal_home directory.*
+
+Now lets runs liminal:
+```BASH
+liminal start
+```
+The start command spins up the liminal server containers which will run pipelines based on your
+deployed liminal.yml files.
+It runs the following three containers:
+* liminal-postgress
+* liminal-webserver
+* liminal-scheduler
+
+Once liminal server has completed starting up, you can navigate to admin UI in your browser:
+[http://localhost:8080](http://localhost:8080)
+By default liminal server starts Apache Airflow servers and admin UI will be that of Apache Airflow.
Review comment:
remove this line
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@liminal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] liorsav commented on a change in pull request #57: iris getting started
Posted by GitBox <gi...@apache.org>.
liorsav commented on a change in pull request #57:
URL: https://github.com/apache/incubator-liminal/pull/57#discussion_r673281133
##########
File path: docs/getting-started/iris_classification.md
##########
@@ -0,0 +1,274 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Getting started / ***Iris Classification***
+
+This guide will allow you to set up your first Apache Liminal environment and allow you to create
+some simple ML pipelines. These will be very similar to the ones you are going to build for real
+production scenarios.
+
+## Prerequisites
+
+Python 3 (3.6 and up)
+
+[Docker Desktop](https://www.docker.com/products/docker-desktop)
+
+*Note: Make sure kubernetes cluster is running in docker desktop (or custom kubernetes installation
+on your machine).*
+
+## Deploying the Example
+
+In this tutorial, we will go through setting up Liminal for the first time on your local machine.
+
+### First, let’s build our example project:
+
+In the dev folder, just clone the example code from liminal:
+
+
+```BASH
+git clone https://github.com/apache/incubator-liminal
+```
+***Note:*** *You just cloned the entire Liminal Project, you actually only need examples folder.*
+
+Create a python virtual environment to isolate your runs:
+
+```BASH
+cd incubator-liminal/examples/aws-ml-app-demo
+python3 -m venv env
+```
+
+Activate your virtual environment:
+
+```BASH
+source env/bin/activate
+```
+
+Now we are ready to install liminal:
+
+```BASH
+pip install apache-liminal
+```
+Let's build the images you need for the example:
+```BASH
+liminal build
+```
+##### The build will create docker images based on the liminal.yml file in the `images` section.
+
+
+Create a kubernetes local volume:
+```BASH
+liminal create
+```
+
+
+The deploy command deploys a liminal server and deploys any liminal.yml files in your working
+directory or any of its subdirectories to your liminal home directory.
+```BASH
+liminal deploy --clean
+```
+
+
+*Note: liminal home directory is located in the path defined in LIMINAL_HOME env variable.
+If the LIMINAL_HOME environemnet variable is not defined, home directory defaults to
+~/liminal_home directory.*
+
+Now lets runs liminal:
+```BASH
+liminal start
+```
+The start command spins up the liminal server containers which will run pipelines based on your
+deployed liminal.yml files.
Review comment:
replace:
The start command spins up the liminal server containers which will run pipelines based on your
deployed liminal.yml files.
with:
The start command spins up 3 containers that load the Apache Airflow stack. Liminal's Airflow extension is responsible to execute the workflows defined in the liminal.yml file as standard Airflow DAGs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@liminal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] liorsav commented on a change in pull request #57: iris getting started
Posted by GitBox <gi...@apache.org>.
liorsav commented on a change in pull request #57:
URL: https://github.com/apache/incubator-liminal/pull/57#discussion_r672410485
##########
File path: docs/README.md
##########
@@ -35,10 +35,11 @@ Using simple YAML configuration, create your own schedule data pipelines (a sequ
perform), application servers, and more.
## Getting Started
-A simple getting stated guide for Liminal can be found [here](getting_started.md)
+A simple hello world guide for Liminal can be found [here](getting-started/hello_world.md) \
+A simple Iris classification model guide for Liminal can be found [here](getting-started/iris_classification.md)
Review comment:
A more advanced example which demonstrate a simple data-science workflow can be found [here](getting-started/iris_classification.md
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@liminal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] naturalett commented on pull request #57: iris getting started
Posted by GitBox <gi...@apache.org>.
naturalett commented on pull request #57:
URL: https://github.com/apache/incubator-liminal/pull/57#issuecomment-879921615
@simlev Yes. I will add.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@liminal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] liorsav merged pull request #57: iris getting started
Posted by GitBox <gi...@apache.org>.
liorsav merged pull request #57:
URL: https://github.com/apache/incubator-liminal/pull/57
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@liminal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] liorsav commented on a change in pull request #57: iris getting started
Posted by GitBox <gi...@apache.org>.
liorsav commented on a change in pull request #57:
URL: https://github.com/apache/incubator-liminal/pull/57#discussion_r673117055
##########
File path: docs/getting-started/iris_classification.md
##########
@@ -0,0 +1,274 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Getting started / ***Iris Classification***
+
+This guide will allow you to set up your first Apache Liminal environment and allow you to create
+some simple ML pipelines. These will be very similar to the ones you are going to build for real
+production scenarios.
+
+## Prerequisites
+
+Python 3 (3.6 and up)
+
+[Docker Desktop](https://www.docker.com/products/docker-desktop)
+
+*Note: Make sure kubernetes cluster is running in docker desktop (or custom kubernetes installation
Review comment:
please delete this - (or custom kubernetes installation on your machine)
its confusing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@liminal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] liorsav commented on a change in pull request #57: iris getting started
Posted by GitBox <gi...@apache.org>.
liorsav commented on a change in pull request #57:
URL: https://github.com/apache/incubator-liminal/pull/57#discussion_r673176441
##########
File path: docs/getting-started/iris_classification.md
##########
@@ -0,0 +1,274 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Getting started / ***Iris Classification***
Review comment:
In this tutorial, we will guide you through setting up Apache Liminal on your local machine and run the a simple machine-learning workflow example.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@liminal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] liorsav commented on a change in pull request #57: iris getting started
Posted by GitBox <gi...@apache.org>.
liorsav commented on a change in pull request #57:
URL: https://github.com/apache/incubator-liminal/pull/57#discussion_r673281133
##########
File path: docs/getting-started/iris_classification.md
##########
@@ -0,0 +1,274 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Getting started / ***Iris Classification***
+
+This guide will allow you to set up your first Apache Liminal environment and allow you to create
+some simple ML pipelines. These will be very similar to the ones you are going to build for real
+production scenarios.
+
+## Prerequisites
+
+Python 3 (3.6 and up)
+
+[Docker Desktop](https://www.docker.com/products/docker-desktop)
+
+*Note: Make sure kubernetes cluster is running in docker desktop (or custom kubernetes installation
+on your machine).*
+
+## Deploying the Example
+
+In this tutorial, we will go through setting up Liminal for the first time on your local machine.
+
+### First, let’s build our example project:
+
+In the dev folder, just clone the example code from liminal:
+
+
+```BASH
+git clone https://github.com/apache/incubator-liminal
+```
+***Note:*** *You just cloned the entire Liminal Project, you actually only need examples folder.*
+
+Create a python virtual environment to isolate your runs:
+
+```BASH
+cd incubator-liminal/examples/aws-ml-app-demo
+python3 -m venv env
+```
+
+Activate your virtual environment:
+
+```BASH
+source env/bin/activate
+```
+
+Now we are ready to install liminal:
+
+```BASH
+pip install apache-liminal
+```
+Let's build the images you need for the example:
+```BASH
+liminal build
+```
+##### The build will create docker images based on the liminal.yml file in the `images` section.
+
+
+Create a kubernetes local volume:
+```BASH
+liminal create
+```
+
+
+The deploy command deploys a liminal server and deploys any liminal.yml files in your working
+directory or any of its subdirectories to your liminal home directory.
+```BASH
+liminal deploy --clean
+```
+
+
+*Note: liminal home directory is located in the path defined in LIMINAL_HOME env variable.
+If the LIMINAL_HOME environemnet variable is not defined, home directory defaults to
+~/liminal_home directory.*
+
+Now lets runs liminal:
+```BASH
+liminal start
+```
+The start command spins up the liminal server containers which will run pipelines based on your
+deployed liminal.yml files.
Review comment:
The start command spins up 3 containers that load the Apache Airflow stack. Liminal's Airflow extension is responsible to execute the workflows defined in the liminal.yml file as standard Airflow DAGs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@liminal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] naturalett commented on pull request #57: iris getting started
Posted by GitBox <gi...@apache.org>.
naturalett commented on pull request #57:
URL: https://github.com/apache/incubator-liminal/pull/57#issuecomment-878658246
@aviemzur @lior-schachter Please review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@liminal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org