You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@liminal.apache.org by GitBox <gi...@apache.org> on 2020/11/15 14:38:45 UTC
[GitHub] [incubator-liminal] amihayz opened a new pull request #8: Examples and Docs
amihayz opened a new pull request #8:
URL: https://github.com/apache/incubator-liminal/pull/8
All examples moved to (just one, for now):
/examples
All docs are in /docs folder
docs are created using sphinx and are ready for integration into:
readthedocs.org - need a webhook for this to work
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] amihayz commented on a change in pull request #8: Examples and Docs
Posted by GitBox <gi...@apache.org>.
amihayz commented on a change in pull request #8:
URL: https://github.com/apache/incubator-liminal/pull/8#discussion_r525224679
##########
File path: docs/source/getting_started_with_liminal.md
##########
@@ -0,0 +1,153 @@
+
+
+# Getting Started with Apache Liminal
+
+Liminal is a MachineLearning orchestration platform, or in other words: Liminal enables data scientists and data engineers to define their entire ML system and flow using configuration. From feature engineering to production monitoring - and the framework takes care of the infra behind the scenes and seamlessly integrates with the infrastructure behind the scenes.
+
+
+
+## Quick Start
+
+So, you’ve heard about liminal and decided you want to give it a try.
Review comment:
removed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] aviemzur commented on a change in pull request #8: Examples and Docs
Posted by GitBox <gi...@apache.org>.
aviemzur commented on a change in pull request #8:
URL: https://github.com/apache/incubator-liminal/pull/8#discussion_r525124363
##########
File path: docs/source/getting_started_with_liminal.md
##########
@@ -0,0 +1,153 @@
+
+
+# Getting Started with Apache Liminal
+
+Liminal is a MachineLearning orchestration platform, or in other words: Liminal enables data scientists and data engineers to define their entire ML system and flow using configuration. From feature engineering to production monitoring - and the framework takes care of the infra behind the scenes and seamlessly integrates with the infrastructure behind the scenes.
+
+
+
+## Quick Start
+
+So, you’ve heard about liminal and decided you want to give it a try.
+
+This guide will allow you to set up your first apache Liminal environment and allow you to create some simple ML pipelines. These will be very similar to the ones you are going to build for real production scenarios. This is actually the magic behind Liminal.
+
+
+
+## Prerequisites
+
+Python 3 (3.6 and up)
+
+[Docker Desktop](https://www.docker.com/products/docker-desktop)
+
+*Note: make sure your kubernetes cluster is running*
+
+### Apache Liminal Hello World
+
+In this tutorial, we will go through setting up Liminal for the first time on your local dev machine.
+
+I’m running this on my macBook, so this will cover mac related installation aspects and kinks (if there are any).
+
+First, let’s build our examples project:
+
+In the dev folder, just clone the example code from liminal:
+
+
+```
+git clone https://github.com/apache/incubator-liminal
+```
+***Note:*** *You just cloned the entire Liminal Project, you actually need just the examples folder.*
+
+Create a python virtualenv to isolate your runs, like this:
+
+
+```
+cd incubator-liminal/examples/liminal-getting-started
+python3 -m venv env
+```
+
+
+And activate your virtual environment:
+
+
+```
+source ../env/bin/activate
+```
+
+Now we are ready to install liminal:
+
+
+```
+pip uninstall apache-liminal
Review comment:
remove uninstall command from here
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] aviemzur commented on a change in pull request #8: Examples and Docs
Posted by GitBox <gi...@apache.org>.
aviemzur commented on a change in pull request #8:
URL: https://github.com/apache/incubator-liminal/pull/8#discussion_r525125959
##########
File path: docs/source/getting_started_with_liminal.md
##########
@@ -0,0 +1,153 @@
+
+
+# Getting Started with Apache Liminal
+
+Liminal is a MachineLearning orchestration platform, or in other words: Liminal enables data scientists and data engineers to define their entire ML system and flow using configuration. From feature engineering to production monitoring - and the framework takes care of the infra behind the scenes and seamlessly integrates with the infrastructure behind the scenes.
+
+
+
+## Quick Start
+
+So, you’ve heard about liminal and decided you want to give it a try.
+
+This guide will allow you to set up your first apache Liminal environment and allow you to create some simple ML pipelines. These will be very similar to the ones you are going to build for real production scenarios. This is actually the magic behind Liminal.
+
+
+
+## Prerequisites
+
+Python 3 (3.6 and up)
+
+[Docker Desktop](https://www.docker.com/products/docker-desktop)
+
+*Note: make sure your kubernetes cluster is running*
+
+### Apache Liminal Hello World
+
+In this tutorial, we will go through setting up Liminal for the first time on your local dev machine.
+
+I’m running this on my macBook, so this will cover mac related installation aspects and kinks (if there are any).
+
+First, let’s build our examples project:
+
+In the dev folder, just clone the example code from liminal:
+
+
+```
+git clone https://github.com/apache/incubator-liminal
+```
+***Note:*** *You just cloned the entire Liminal Project, you actually need just the examples folder.*
+
+Create a python virtualenv to isolate your runs, like this:
+
+
+```
+cd incubator-liminal/examples/liminal-getting-started
+python3 -m venv env
+```
+
+
+And activate your virtual environment:
+
+
+```
+source ../env/bin/activate
+```
+
+Now we are ready to install liminal:
+
+
+```
+pip uninstall apache-liminal
+pip install apache-liminal
+rm -rf ~/liminal_home
+```
+Let's build the images you need for the example:
+```
+liminal build
+```
+The build will create docker images based on the Liminal.yml file, eash task and service will have its docker image created.
+
+
+```
+liminal deploy --clean # the --clean is for removing old containers from the previous install
+```
+The deploy will create missing containers from the images created above as well as basic Airflow containers needed to run the piplines, and deploy them to your local kubernetes cluster
Review comment:
Uses docker (docker-compose) not kubernetes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] amihayz commented on a change in pull request #8: Examples and Docs
Posted by GitBox <gi...@apache.org>.
amihayz commented on a change in pull request #8:
URL: https://github.com/apache/incubator-liminal/pull/8#discussion_r525241472
##########
File path: docs/source/getting_started_with_liminal.md
##########
@@ -0,0 +1,153 @@
+
+
+# Getting Started with Apache Liminal
+
+Liminal is a MachineLearning orchestration platform, or in other words: Liminal enables data scientists and data engineers to define their entire ML system and flow using configuration. From feature engineering to production monitoring - and the framework takes care of the infra behind the scenes and seamlessly integrates with the infrastructure behind the scenes.
+
+
+
+## Quick Start
+
+So, you’ve heard about liminal and decided you want to give it a try.
+
+This guide will allow you to set up your first apache Liminal environment and allow you to create some simple ML pipelines. These will be very similar to the ones you are going to build for real production scenarios. This is actually the magic behind Liminal.
+
+
+
+## Prerequisites
+
+Python 3 (3.6 and up)
+
+[Docker Desktop](https://www.docker.com/products/docker-desktop)
+
+*Note: make sure your kubernetes cluster is running*
+
+### Apache Liminal Hello World
+
+In this tutorial, we will go through setting up Liminal for the first time on your local dev machine.
+
+I’m running this on my macBook, so this will cover mac related installation aspects and kinks (if there are any).
+
+First, let’s build our examples project:
+
+In the dev folder, just clone the example code from liminal:
+
+
+```
+git clone https://github.com/apache/incubator-liminal
+```
+***Note:*** *You just cloned the entire Liminal Project, you actually need just the examples folder.*
+
+Create a python virtualenv to isolate your runs, like this:
+
+
+```
+cd incubator-liminal/examples/liminal-getting-started
+python3 -m venv env
+```
+
+
+And activate your virtual environment:
+
+
+```
+source ../env/bin/activate
+```
+
+Now we are ready to install liminal:
+
+
+```
+pip uninstall apache-liminal
+pip install apache-liminal
+rm -rf ~/liminal_home
+```
+Let's build the images you need for the example:
+```
+liminal build
+```
+The build will create docker images based on the Liminal.yml file, eash task and service will have its docker image created.
+
+
+```
+liminal deploy --clean # the --clean is for removing old containers from the previous install
+```
+The deploy will create missing containers from the images created above as well as basic Airflow containers needed to run the piplines, and deploy them to your local kubernetes cluster
+
+*Note: Liminal creates a bunch of assets, some are going to be located under: the directory in your LIMINAL_HOME env variable. But default LIMINAL_HOME is set to ~/liminal_home directory.*
+
+Now lets runs it:
+```
+liminal start
+```
+Liminal now spins up the AirFlow containers it will run the dag created based on the Liminal.yml file.
+It includes these three containers:
+* liminal-postgress
+* liminal-webserver
+* liminal-scheduler
+
+Once it finished loading the,
+Go to the airflow admin in the browser:
+
+
+```
+http://localhost:8080/admin
+```
+You can just click: [http://localhost:8080/admin](http://localhost:8080/admin)
+
+
+
+
+***Important:** Click on the “On” button to activate the dag, nothing will happen otherwise!*
+
+You can go to tree view to see all the tasks configured in the liminal.yml file: \
+[http://localhost:8080/admin/airflow/tree?dag_id=example_pipeline](http://localhost:8080/admin/airflow/tree?dag_id=example_pipeline)
+
+Now lets see what actually happened to our task:
+
+
+
+Click on “hello_world_example” and you will get this popup: \
+
+ \
+Click on “view log” button and you can see the log of the current task run: \
+
+
+
+
+And That's it!
+
+### Here are the entire list of commands, if you want to start from scratch:
+
+```
+git clone https://github.com/amihayz/liminal-examples
+cd liminal-examples
+python3 -m venv env
+source env/bin/activate
+pip uninstall apache-liminal
+pip install apache-liminal
+rm -rf ~/liminal_home
+Liminal build
+liminal deploy --clean
+liminal start
+```
+
+### Closing up
+
+Killing (cmd-c) the cmd you are running in would partially stop the containers in Docker desktop.
Review comment:
control-c
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] aviemzur commented on a change in pull request #8: Examples and Docs
Posted by GitBox <gi...@apache.org>.
aviemzur commented on a change in pull request #8:
URL: https://github.com/apache/incubator-liminal/pull/8#discussion_r525129811
##########
File path: docs/source/getting_started_with_liminal.md
##########
@@ -0,0 +1,153 @@
+
+
+# Getting Started with Apache Liminal
+
+Liminal is a MachineLearning orchestration platform, or in other words: Liminal enables data scientists and data engineers to define their entire ML system and flow using configuration. From feature engineering to production monitoring - and the framework takes care of the infra behind the scenes and seamlessly integrates with the infrastructure behind the scenes.
+
+
+
+## Quick Start
+
+So, you’ve heard about liminal and decided you want to give it a try.
+
+This guide will allow you to set up your first apache Liminal environment and allow you to create some simple ML pipelines. These will be very similar to the ones you are going to build for real production scenarios. This is actually the magic behind Liminal.
+
+
+
+## Prerequisites
+
+Python 3 (3.6 and up)
+
+[Docker Desktop](https://www.docker.com/products/docker-desktop)
+
+*Note: make sure your kubernetes cluster is running*
+
+### Apache Liminal Hello World
+
+In this tutorial, we will go through setting up Liminal for the first time on your local dev machine.
+
+I’m running this on my macBook, so this will cover mac related installation aspects and kinks (if there are any).
+
+First, let’s build our examples project:
+
+In the dev folder, just clone the example code from liminal:
+
+
+```
+git clone https://github.com/apache/incubator-liminal
+```
+***Note:*** *You just cloned the entire Liminal Project, you actually need just the examples folder.*
+
+Create a python virtualenv to isolate your runs, like this:
+
+
+```
+cd incubator-liminal/examples/liminal-getting-started
+python3 -m venv env
+```
+
+
+And activate your virtual environment:
+
+
+```
+source ../env/bin/activate
+```
+
+Now we are ready to install liminal:
+
+
+```
+pip uninstall apache-liminal
+pip install apache-liminal
+rm -rf ~/liminal_home
+```
+Let's build the images you need for the example:
+```
+liminal build
+```
+The build will create docker images based on the Liminal.yml file, eash task and service will have its docker image created.
+
+
+```
+liminal deploy --clean # the --clean is for removing old containers from the previous install
+```
+The deploy will create missing containers from the images created above as well as basic Airflow containers needed to run the piplines, and deploy them to your local kubernetes cluster
+
+*Note: Liminal creates a bunch of assets, some are going to be located under: the directory in your LIMINAL_HOME env variable. But default LIMINAL_HOME is set to ~/liminal_home directory.*
+
+Now lets runs it:
+```
+liminal start
+```
+Liminal now spins up the AirFlow containers it will run the dag created based on the Liminal.yml file.
+It includes these three containers:
+* liminal-postgress
+* liminal-webserver
+* liminal-scheduler
+
+Once it finished loading the,
+Go to the airflow admin in the browser:
+
+
+```
+http://localhost:8080/admin
+```
+You can just click: [http://localhost:8080/admin](http://localhost:8080/admin)
+
+
+
+
+***Important:** Click on the “On” button to activate the dag, nothing will happen otherwise!*
+
+You can go to tree view to see all the tasks configured in the liminal.yml file: \
+[http://localhost:8080/admin/airflow/tree?dag_id=example_pipeline](http://localhost:8080/admin/airflow/tree?dag_id=example_pipeline)
+
+Now lets see what actually happened to our task:
+
+
+
+Click on “hello_world_example” and you will get this popup: \
+
+ \
+Click on “view log” button and you can see the log of the current task run: \
+
+
+
+
+And That's it!
+
+### Here are the entire list of commands, if you want to start from scratch:
+
+```
+git clone https://github.com/amihayz/liminal-examples
+cd liminal-examples
+python3 -m venv env
+source env/bin/activate
+pip uninstall apache-liminal
+pip install apache-liminal
+rm -rf ~/liminal_home
+Liminal build
+liminal deploy --clean
+liminal start
+```
+
+### Closing up
+
+Killing (cmd-c) the cmd you are running in would partially stop the containers in Docker desktop.
Review comment:
`cmd-c` -> `^c` / `control+c` ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] aviemzur commented on a change in pull request #8: Examples and Docs
Posted by GitBox <gi...@apache.org>.
aviemzur commented on a change in pull request #8:
URL: https://github.com/apache/incubator-liminal/pull/8#discussion_r525114028
##########
File path: docs/source/getting_started_with_liminal.md
##########
@@ -0,0 +1,153 @@
+
+
+# Getting Started with Apache Liminal
+
+Liminal is a MachineLearning orchestration platform, or in other words: Liminal enables data scientists and data engineers to define their entire ML system and flow using configuration. From feature engineering to production monitoring - and the framework takes care of the infra behind the scenes and seamlessly integrates with the infrastructure behind the scenes.
+
+
+
+## Quick Start
+
+So, you’ve heard about liminal and decided you want to give it a try.
Review comment:
Less casual?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] aviemzur commented on a change in pull request #8: Examples and Docs
Posted by GitBox <gi...@apache.org>.
aviemzur commented on a change in pull request #8:
URL: https://github.com/apache/incubator-liminal/pull/8#discussion_r525130924
##########
File path: docs/source/getting_started_with_liminal.md
##########
@@ -0,0 +1,153 @@
+
+
+# Getting Started with Apache Liminal
+
+Liminal is a MachineLearning orchestration platform, or in other words: Liminal enables data scientists and data engineers to define their entire ML system and flow using configuration. From feature engineering to production monitoring - and the framework takes care of the infra behind the scenes and seamlessly integrates with the infrastructure behind the scenes.
+
+
+
+## Quick Start
+
+So, you’ve heard about liminal and decided you want to give it a try.
+
+This guide will allow you to set up your first apache Liminal environment and allow you to create some simple ML pipelines. These will be very similar to the ones you are going to build for real production scenarios. This is actually the magic behind Liminal.
+
+
+
+## Prerequisites
+
+Python 3 (3.6 and up)
+
+[Docker Desktop](https://www.docker.com/products/docker-desktop)
+
+*Note: make sure your kubernetes cluster is running*
+
+### Apache Liminal Hello World
+
+In this tutorial, we will go through setting up Liminal for the first time on your local dev machine.
+
+I’m running this on my macBook, so this will cover mac related installation aspects and kinks (if there are any).
+
+First, let’s build our examples project:
+
+In the dev folder, just clone the example code from liminal:
+
+
+```
+git clone https://github.com/apache/incubator-liminal
+```
+***Note:*** *You just cloned the entire Liminal Project, you actually need just the examples folder.*
+
+Create a python virtualenv to isolate your runs, like this:
+
+
+```
+cd incubator-liminal/examples/liminal-getting-started
+python3 -m venv env
+```
+
+
+And activate your virtual environment:
+
+
+```
+source ../env/bin/activate
+```
+
+Now we are ready to install liminal:
+
+
+```
+pip uninstall apache-liminal
+pip install apache-liminal
+rm -rf ~/liminal_home
+```
+Let's build the images you need for the example:
+```
+liminal build
+```
+The build will create docker images based on the Liminal.yml file, eash task and service will have its docker image created.
+
+
+```
+liminal deploy --clean # the --clean is for removing old containers from the previous install
+```
+The deploy will create missing containers from the images created above as well as basic Airflow containers needed to run the piplines, and deploy them to your local kubernetes cluster
+
+*Note: Liminal creates a bunch of assets, some are going to be located under: the directory in your LIMINAL_HOME env variable. But default LIMINAL_HOME is set to ~/liminal_home directory.*
+
+Now lets runs it:
+```
+liminal start
+```
+Liminal now spins up the AirFlow containers it will run the dag created based on the Liminal.yml file.
+It includes these three containers:
+* liminal-postgress
+* liminal-webserver
+* liminal-scheduler
+
+Once it finished loading the,
+Go to the airflow admin in the browser:
+
+
+```
+http://localhost:8080/admin
+```
+You can just click: [http://localhost:8080/admin](http://localhost:8080/admin)
+
+
+
+
+***Important:** Click on the “On” button to activate the dag, nothing will happen otherwise!*
+
+You can go to tree view to see all the tasks configured in the liminal.yml file: \
+[http://localhost:8080/admin/airflow/tree?dag_id=example_pipeline](http://localhost:8080/admin/airflow/tree?dag_id=example_pipeline)
+
+Now lets see what actually happened to our task:
+
+
+
+Click on “hello_world_example” and you will get this popup: \
+
+ \
+Click on “view log” button and you can see the log of the current task run: \
+
+
+
+
+And That's it!
+
+### Here are the entire list of commands, if you want to start from scratch:
+
+```
+git clone https://github.com/amihayz/liminal-examples
+cd liminal-examples
+python3 -m venv env
+source env/bin/activate
+pip uninstall apache-liminal
+pip install apache-liminal
+rm -rf ~/liminal_home
+Liminal build
+liminal deploy --clean
+liminal start
+```
+
+### Closing up
+
+Killing (cmd-c) the cmd you are running in would partially stop the containers in Docker desktop.
+But, for making sure the dockers have closed:
+
+
+```
+docker container stop liminal-postgress
+docker container stop liminal-webserver
+docker container stop liminal-scheduler
+```
+
+
+And don't forget to stop the python virtualenv, just write:
Review comment:
`stop` -> `deactivate`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] amihayz commented on a change in pull request #8: Examples and Docs
Posted by GitBox <gi...@apache.org>.
amihayz commented on a change in pull request #8:
URL: https://github.com/apache/incubator-liminal/pull/8#discussion_r525911151
##########
File path: docs/source/getting_started_with_liminal.md
##########
@@ -0,0 +1,153 @@
+
+
+# Getting Started with Apache Liminal
+
+Liminal is a MachineLearning orchestration platform, or in other words: Liminal enables data scientists and data engineers to define their entire ML system and flow using configuration. From feature engineering to production monitoring - and the framework takes care of the infra behind the scenes and seamlessly integrates with the infrastructure behind the scenes.
+
+
+
+## Quick Start
+
+So, you’ve heard about liminal and decided you want to give it a try.
+
+This guide will allow you to set up your first apache Liminal environment and allow you to create some simple ML pipelines. These will be very similar to the ones you are going to build for real production scenarios. This is actually the magic behind Liminal.
+
+
+
+## Prerequisites
+
+Python 3 (3.6 and up)
+
+[Docker Desktop](https://www.docker.com/products/docker-desktop)
+
+*Note: make sure your kubernetes cluster is running*
+
+### Apache Liminal Hello World
+
+In this tutorial, we will go through setting up Liminal for the first time on your local dev machine.
+
+I’m running this on my macBook, so this will cover mac related installation aspects and kinks (if there are any).
+
+First, let’s build our examples project:
+
+In the dev folder, just clone the example code from liminal:
+
+
+```
+git clone https://github.com/apache/incubator-liminal
+```
+***Note:*** *You just cloned the entire Liminal Project, you actually need just the examples folder.*
+
+Create a python virtualenv to isolate your runs, like this:
+
+
+```
+cd incubator-liminal/examples/liminal-getting-started
+python3 -m venv env
+```
+
+
+And activate your virtual environment:
+
+
+```
+source ../env/bin/activate
+```
+
+Now we are ready to install liminal:
+
+
+```
+pip uninstall apache-liminal
+pip install apache-liminal
+rm -rf ~/liminal_home
+```
+Let's build the images you need for the example:
+```
+liminal build
+```
+The build will create docker images based on the Liminal.yml file, eash task and service will have its docker image created.
+
+
+```
+liminal deploy --clean # the --clean is for removing old containers from the previous install
+```
+The deploy will create missing containers from the images created above as well as basic Airflow containers needed to run the piplines, and deploy them to your local kubernetes cluster
+
+*Note: Liminal creates a bunch of assets, some are going to be located under: the directory in your LIMINAL_HOME env variable. But default LIMINAL_HOME is set to ~/liminal_home directory.*
+
+Now lets runs it:
+```
+liminal start
+```
+Liminal now spins up the AirFlow containers it will run the dag created based on the Liminal.yml file.
+It includes these three containers:
+* liminal-postgress
+* liminal-webserver
+* liminal-scheduler
+
+Once it finished loading the,
+Go to the airflow admin in the browser:
+
+
+```
+http://localhost:8080/admin
+```
+You can just click: [http://localhost:8080/admin](http://localhost:8080/admin)
+
+
+
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] amihayz commented on a change in pull request #8: Examples and Docs
Posted by GitBox <gi...@apache.org>.
amihayz commented on a change in pull request #8:
URL: https://github.com/apache/incubator-liminal/pull/8#discussion_r525910843
##########
File path: docs/source/getting_started_with_liminal.md
##########
@@ -0,0 +1,153 @@
+
+
+# Getting Started with Apache Liminal
+
+Liminal is a MachineLearning orchestration platform, or in other words: Liminal enables data scientists and data engineers to define their entire ML system and flow using configuration. From feature engineering to production monitoring - and the framework takes care of the infra behind the scenes and seamlessly integrates with the infrastructure behind the scenes.
+
+
+
+## Quick Start
+
+So, you’ve heard about liminal and decided you want to give it a try.
+
+This guide will allow you to set up your first apache Liminal environment and allow you to create some simple ML pipelines. These will be very similar to the ones you are going to build for real production scenarios. This is actually the magic behind Liminal.
+
+
+
+## Prerequisites
+
+Python 3 (3.6 and up)
+
+[Docker Desktop](https://www.docker.com/products/docker-desktop)
+
+*Note: make sure your kubernetes cluster is running*
Review comment:
fixed lingo
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] amihayz commented on a change in pull request #8: Examples and Docs
Posted by GitBox <gi...@apache.org>.
amihayz commented on a change in pull request #8:
URL: https://github.com/apache/incubator-liminal/pull/8#discussion_r525240591
##########
File path: docs/source/getting_started_with_liminal.md
##########
@@ -0,0 +1,153 @@
+
+
+# Getting Started with Apache Liminal
+
+Liminal is a MachineLearning orchestration platform, or in other words: Liminal enables data scientists and data engineers to define their entire ML system and flow using configuration. From feature engineering to production monitoring - and the framework takes care of the infra behind the scenes and seamlessly integrates with the infrastructure behind the scenes.
+
+
+
+## Quick Start
+
+So, you’ve heard about liminal and decided you want to give it a try.
+
+This guide will allow you to set up your first apache Liminal environment and allow you to create some simple ML pipelines. These will be very similar to the ones you are going to build for real production scenarios. This is actually the magic behind Liminal.
+
+
+
+## Prerequisites
+
+Python 3 (3.6 and up)
+
+[Docker Desktop](https://www.docker.com/products/docker-desktop)
+
+*Note: make sure your kubernetes cluster is running*
+
+### Apache Liminal Hello World
+
+In this tutorial, we will go through setting up Liminal for the first time on your local dev machine.
+
+I’m running this on my macBook, so this will cover mac related installation aspects and kinks (if there are any).
+
+First, let’s build our examples project:
+
+In the dev folder, just clone the example code from liminal:
+
+
+```
+git clone https://github.com/apache/incubator-liminal
+```
+***Note:*** *You just cloned the entire Liminal Project, you actually need just the examples folder.*
+
+Create a python virtualenv to isolate your runs, like this:
+
+
+```
+cd incubator-liminal/examples/liminal-getting-started
+python3 -m venv env
+```
+
+
+And activate your virtual environment:
+
+
+```
+source ../env/bin/activate
+```
+
+Now we are ready to install liminal:
+
+
+```
+pip uninstall apache-liminal
+pip install apache-liminal
+rm -rf ~/liminal_home
+```
+Let's build the images you need for the example:
+```
+liminal build
+```
+The build will create docker images based on the Liminal.yml file, eash task and service will have its docker image created.
+
+
+```
+liminal deploy --clean # the --clean is for removing old containers from the previous install
+```
+The deploy will create missing containers from the images created above as well as basic Airflow containers needed to run the piplines, and deploy them to your local kubernetes cluster
+
+*Note: Liminal creates a bunch of assets, some are going to be located under: the directory in your LIMINAL_HOME env variable. But default LIMINAL_HOME is set to ~/liminal_home directory.*
+
+Now lets runs it:
+```
+liminal start
+```
+Liminal now spins up the AirFlow containers it will run the dag created based on the Liminal.yml file.
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] aviemzur merged pull request #8: Examples and Docs
Posted by GitBox <gi...@apache.org>.
aviemzur merged pull request #8:
URL: https://github.com/apache/incubator-liminal/pull/8
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] aviemzur commented on a change in pull request #8: Examples and Docs
Posted by GitBox <gi...@apache.org>.
aviemzur commented on a change in pull request #8:
URL: https://github.com/apache/incubator-liminal/pull/8#discussion_r525116850
##########
File path: docs/source/getting_started_with_liminal.md
##########
@@ -0,0 +1,153 @@
+
+
+# Getting Started with Apache Liminal
+
+Liminal is a MachineLearning orchestration platform, or in other words: Liminal enables data scientists and data engineers to define their entire ML system and flow using configuration. From feature engineering to production monitoring - and the framework takes care of the infra behind the scenes and seamlessly integrates with the infrastructure behind the scenes.
+
+
+
+## Quick Start
+
+So, you’ve heard about liminal and decided you want to give it a try.
+
+This guide will allow you to set up your first apache Liminal environment and allow you to create some simple ML pipelines. These will be very similar to the ones you are going to build for real production scenarios. This is actually the magic behind Liminal.
+
+
+
+## Prerequisites
+
+Python 3 (3.6 and up)
+
+[Docker Desktop](https://www.docker.com/products/docker-desktop)
+
+*Note: make sure your kubernetes cluster is running*
Review comment:
I think there shouldn't be a space between docker desktop and this note.
Suggestion:
```
Make sure kubernetes cluster is running in docker desktop (or custom kubernetes installation on your machine).
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] aviemzur commented on a change in pull request #8: Examples and Docs
Posted by GitBox <gi...@apache.org>.
aviemzur commented on a change in pull request #8:
URL: https://github.com/apache/incubator-liminal/pull/8#discussion_r525111292
##########
File path: docs/source/conf.py
##########
@@ -0,0 +1,175 @@
+# -*- coding: utf-8 -*-
+#
+# Configuration file for the Sphinx documentation builder.
+#
+# This file does only contain a selection of the most common options. For a
+# full list see the documentation:
+# http://www.sphinx-doc.org/en/master/config
+
+# -- Path setup --------------------------------------------------------------
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+#
+# import os
+# import sys
+# sys.path.insert(0, os.path.abspath('.'))
+
+
+# -- Project information -----------------------------------------------------
+
+project = u'Apchae Liminal'
+copyright = u'2020, Amihay Zer-Kavod'
Review comment:
Apache
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] aviemzur commented on a change in pull request #8: Examples and Docs
Posted by GitBox <gi...@apache.org>.
aviemzur commented on a change in pull request #8:
URL: https://github.com/apache/incubator-liminal/pull/8#discussion_r525126164
##########
File path: docs/source/getting_started_with_liminal.md
##########
@@ -0,0 +1,153 @@
+
+
+# Getting Started with Apache Liminal
+
+Liminal is a MachineLearning orchestration platform, or in other words: Liminal enables data scientists and data engineers to define their entire ML system and flow using configuration. From feature engineering to production monitoring - and the framework takes care of the infra behind the scenes and seamlessly integrates with the infrastructure behind the scenes.
+
+
+
+## Quick Start
+
+So, you’ve heard about liminal and decided you want to give it a try.
+
+This guide will allow you to set up your first apache Liminal environment and allow you to create some simple ML pipelines. These will be very similar to the ones you are going to build for real production scenarios. This is actually the magic behind Liminal.
+
+
+
+## Prerequisites
+
+Python 3 (3.6 and up)
+
+[Docker Desktop](https://www.docker.com/products/docker-desktop)
+
+*Note: make sure your kubernetes cluster is running*
+
+### Apache Liminal Hello World
+
+In this tutorial, we will go through setting up Liminal for the first time on your local dev machine.
+
+I’m running this on my macBook, so this will cover mac related installation aspects and kinks (if there are any).
+
+First, let’s build our examples project:
+
+In the dev folder, just clone the example code from liminal:
+
+
+```
+git clone https://github.com/apache/incubator-liminal
+```
+***Note:*** *You just cloned the entire Liminal Project, you actually need just the examples folder.*
+
+Create a python virtualenv to isolate your runs, like this:
+
+
+```
+cd incubator-liminal/examples/liminal-getting-started
+python3 -m venv env
+```
+
+
+And activate your virtual environment:
+
+
+```
+source ../env/bin/activate
+```
+
+Now we are ready to install liminal:
+
+
+```
+pip uninstall apache-liminal
+pip install apache-liminal
+rm -rf ~/liminal_home
+```
+Let's build the images you need for the example:
+```
+liminal build
+```
+The build will create docker images based on the Liminal.yml file, eash task and service will have its docker image created.
+
+
+```
+liminal deploy --clean # the --clean is for removing old containers from the previous install
+```
+The deploy will create missing containers from the images created above as well as basic Airflow containers needed to run the piplines, and deploy them to your local kubernetes cluster
+
+*Note: Liminal creates a bunch of assets, some are going to be located under: the directory in your LIMINAL_HOME env variable. But default LIMINAL_HOME is set to ~/liminal_home directory.*
+
+Now lets runs it:
+```
+liminal start
+```
+Liminal now spins up the AirFlow containers it will run the dag created based on the Liminal.yml file.
Review comment:
`AirFlow` -> `airflow`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] aviemzur commented on a change in pull request #8: Examples and Docs
Posted by GitBox <gi...@apache.org>.
aviemzur commented on a change in pull request #8:
URL: https://github.com/apache/incubator-liminal/pull/8#discussion_r525132645
##########
File path: docs/source/liminal_architecture.md
##########
@@ -0,0 +1,43 @@
+# Apache Liminal High Level Architecture
+Liminal is an end-to-end platform for data engineers & scientists, allowing them to build, train and deploy machine learning models in a robust and agile way. The platform provides the abstractions and declarative capabilities for data extraction & feature engineering followed by model training and serving. Apache Liminal's goal is to operationalise the machine learning process, allowing data scientists to quickly transition from a successful experiment to an automated pipeline of model training, validation, deployment and inference in production, freeing them from engineering and non-functional tasks, and allowing them to focus on machine learning code and artifacts.
+
+## Motivation
+The challenges involved in operationalizing machine learning models are one of the main reasons why many machine learning projects never make it to production.
+The process involves automating and orchestrating multiple steps which run on heterogeneous infrastructure - different compute environments, data processing platforms, ML frameworks, notebooks, containers and monitoring tools.
+There are no mature standards for this workflow, and most organizations do not have the experience to build it in-house. In the best case, dev-ds-devops teams form in order to accomplish this task together; in many cases, it's the data scientists who try to deal with this themselves without the knowledge or the inclination to become infrastructure experts.
+As a result, many projects never make it through the cycle. Those who do suffer from a very long lead time from a successful experiment to an operational, refreshable, deployed and monitored model in production.
+The goal of Apache Liminal is to simplify the creation and management of machine learning pipelines by data engineers & scientists. The platform provides declarative building blocks which define the workflow, orchestrate the underlying infrastructure, take care of non functional concerns, enabling focus in business logic / algorithm code.
+Some Commercial E2E solutions have started to emerge in the last few years, however, they are limited to specific parts of the workflow, such as Databricks MLFlow. Other solutions are tied to specific environments (e.g. SageMaker on AWS).
+
+## High Level Architecture
+The platform is aimed to provide data engineers & scientists with a solution for end to end flows from model training to real time inference in production. It’s architecture enables and promotes adoption of specific components in existing (non-Liminal) frameworks, as well as seamless integration with other open source projects. Liminal was created to enable scalability in ML efforts and after a thorough review of available solutions and frameworks, which did not meet our main KPIs:
+Provide an opinionated but customizable end-to-end workflow
+Abstract away the complexity of underlying infrastructure
+Support major open source tools and cloud-native infrastructure to carry out many of the steps
+Allow teams to leverage their existing investments or bring in their tools of choice into the workflow
+We have found that other tech companies in the Israeli Hi-Tech ecosystem also have an interest in such a platform, hence decided to share our work with the community.
+The following diagram depicts these main components and where Apache Liminal comes in:
+
+
+
+A classical data scientist workflow includes some base phases:
+_Train, Deploy and Consume._
+
+**The Train phase includes the following tasks:**
+
+1. Fetch - get the data needed to build a model - usually using SQL
+1. Clean - make sure the data is useful for building the model
+1. Prepare - split data and encode features from the data according to model needs
+1. Train - Build the model and tune it
+1. Evaluate - make sure the model is correct - run it on a test set, etc…
+1. Validate - make sure the model is up to the standards you need
+
+**The Deploy phase includes these tasks:**
+1. Deploy - make it available for usage in production
+1. Inference - Batch or Real-time - use the model to evaluate data by your offline or online by your applications
+1. Consume - The actual use of the models created by applications and ETLs, usually through APIs to the batch or real-time inference that usually rely on Model and Feature stores.
+
+Liminal provides its users a declarative composition capabilities to materialize these steps in a robust way, while exploiting existing frameworks and tools. e.g. Data science frameworks such as scikit-learn, Tensor flow, Keras and such, for running core data science algorithms; as numerous core mechanisms as data stores, processing engines, parallelism, schedulers, code deployment as well as batch and real-time inference.
+Liminal allows the creation and wiring of these kinds of functional and non functional tasks while making the underlying infrastructure used by these tasks very easy to use and even abstracted away entirely. While handling the non-functional aspects as monitoring (in a standard fashion) deployment, scheduling, resource management and execution.
+
+
Review comment:
`raibow` -> `liminal` ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] aviemzur commented on a change in pull request #8: Examples and Docs
Posted by GitBox <gi...@apache.org>.
aviemzur commented on a change in pull request #8:
URL: https://github.com/apache/incubator-liminal/pull/8#discussion_r525130309
##########
File path: docs/source/getting_started_with_liminal.md
##########
@@ -0,0 +1,153 @@
+
+
+# Getting Started with Apache Liminal
+
+Liminal is a MachineLearning orchestration platform, or in other words: Liminal enables data scientists and data engineers to define their entire ML system and flow using configuration. From feature engineering to production monitoring - and the framework takes care of the infra behind the scenes and seamlessly integrates with the infrastructure behind the scenes.
+
+
+
+## Quick Start
+
+So, you’ve heard about liminal and decided you want to give it a try.
+
+This guide will allow you to set up your first apache Liminal environment and allow you to create some simple ML pipelines. These will be very similar to the ones you are going to build for real production scenarios. This is actually the magic behind Liminal.
+
+
+
+## Prerequisites
+
+Python 3 (3.6 and up)
+
+[Docker Desktop](https://www.docker.com/products/docker-desktop)
+
+*Note: make sure your kubernetes cluster is running*
+
+### Apache Liminal Hello World
+
+In this tutorial, we will go through setting up Liminal for the first time on your local dev machine.
+
+I’m running this on my macBook, so this will cover mac related installation aspects and kinks (if there are any).
+
+First, let’s build our examples project:
+
+In the dev folder, just clone the example code from liminal:
+
+
+```
+git clone https://github.com/apache/incubator-liminal
+```
+***Note:*** *You just cloned the entire Liminal Project, you actually need just the examples folder.*
+
+Create a python virtualenv to isolate your runs, like this:
+
+
+```
+cd incubator-liminal/examples/liminal-getting-started
+python3 -m venv env
+```
+
+
+And activate your virtual environment:
+
+
+```
+source ../env/bin/activate
+```
+
+Now we are ready to install liminal:
+
+
+```
+pip uninstall apache-liminal
+pip install apache-liminal
+rm -rf ~/liminal_home
+```
+Let's build the images you need for the example:
+```
+liminal build
+```
+The build will create docker images based on the Liminal.yml file, eash task and service will have its docker image created.
+
+
+```
+liminal deploy --clean # the --clean is for removing old containers from the previous install
+```
+The deploy will create missing containers from the images created above as well as basic Airflow containers needed to run the piplines, and deploy them to your local kubernetes cluster
+
+*Note: Liminal creates a bunch of assets, some are going to be located under: the directory in your LIMINAL_HOME env variable. But default LIMINAL_HOME is set to ~/liminal_home directory.*
+
+Now lets runs it:
+```
+liminal start
+```
+Liminal now spins up the AirFlow containers it will run the dag created based on the Liminal.yml file.
+It includes these three containers:
+* liminal-postgress
+* liminal-webserver
+* liminal-scheduler
+
+Once it finished loading the,
+Go to the airflow admin in the browser:
+
+
+```
+http://localhost:8080/admin
+```
+You can just click: [http://localhost:8080/admin](http://localhost:8080/admin)
+
+
+
+
+***Important:** Click on the “On” button to activate the dag, nothing will happen otherwise!*
+
+You can go to tree view to see all the tasks configured in the liminal.yml file: \
+[http://localhost:8080/admin/airflow/tree?dag_id=example_pipeline](http://localhost:8080/admin/airflow/tree?dag_id=example_pipeline)
+
+Now lets see what actually happened to our task:
+
+
+
+Click on “hello_world_example” and you will get this popup: \
+
+ \
+Click on “view log” button and you can see the log of the current task run: \
+
+
+
+
+And That's it!
+
+### Here are the entire list of commands, if you want to start from scratch:
+
+```
+git clone https://github.com/amihayz/liminal-examples
+cd liminal-examples
+python3 -m venv env
+source env/bin/activate
+pip uninstall apache-liminal
+pip install apache-liminal
+rm -rf ~/liminal_home
+Liminal build
+liminal deploy --clean
+liminal start
+```
+
+### Closing up
+
+Killing (cmd-c) the cmd you are running in would partially stop the containers in Docker desktop.
+But, for making sure the dockers have closed:
+
+
+```
+docker container stop liminal-postgress
Review comment:
Inline all three in one line with spaces between the container names
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] amihayz commented on a change in pull request #8: Examples and Docs
Posted by GitBox <gi...@apache.org>.
amihayz commented on a change in pull request #8:
URL: https://github.com/apache/incubator-liminal/pull/8#discussion_r525239845
##########
File path: docs/source/getting_started_with_liminal.md
##########
@@ -0,0 +1,153 @@
+
+
+# Getting Started with Apache Liminal
+
+Liminal is a MachineLearning orchestration platform, or in other words: Liminal enables data scientists and data engineers to define their entire ML system and flow using configuration. From feature engineering to production monitoring - and the framework takes care of the infra behind the scenes and seamlessly integrates with the infrastructure behind the scenes.
+
+
+
+## Quick Start
+
+So, you’ve heard about liminal and decided you want to give it a try.
+
+This guide will allow you to set up your first apache Liminal environment and allow you to create some simple ML pipelines. These will be very similar to the ones you are going to build for real production scenarios. This is actually the magic behind Liminal.
+
+
+
+## Prerequisites
+
+Python 3 (3.6 and up)
+
+[Docker Desktop](https://www.docker.com/products/docker-desktop)
+
+*Note: make sure your kubernetes cluster is running*
+
+### Apache Liminal Hello World
+
+In this tutorial, we will go through setting up Liminal for the first time on your local dev machine.
+
+I’m running this on my macBook, so this will cover mac related installation aspects and kinks (if there are any).
+
+First, let’s build our examples project:
+
+In the dev folder, just clone the example code from liminal:
+
+
+```
+git clone https://github.com/apache/incubator-liminal
+```
+***Note:*** *You just cloned the entire Liminal Project, you actually need just the examples folder.*
+
+Create a python virtualenv to isolate your runs, like this:
+
+
+```
+cd incubator-liminal/examples/liminal-getting-started
+python3 -m venv env
+```
+
+
+And activate your virtual environment:
+
+
+```
+source ../env/bin/activate
+```
+
+Now we are ready to install liminal:
+
+
+```
+pip uninstall apache-liminal
+pip install apache-liminal
+rm -rf ~/liminal_home
+```
+Let's build the images you need for the example:
+```
+liminal build
+```
+The build will create docker images based on the Liminal.yml file, eash task and service will have its docker image created.
+
+
+```
+liminal deploy --clean # the --clean is for removing old containers from the previous install
+```
+The deploy will create missing containers from the images created above as well as basic Airflow containers needed to run the piplines, and deploy them to your local kubernetes cluster
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] aviemzur commented on a change in pull request #8: Examples and Docs
Posted by GitBox <gi...@apache.org>.
aviemzur commented on a change in pull request #8:
URL: https://github.com/apache/incubator-liminal/pull/8#discussion_r525128882
##########
File path: docs/source/getting_started_with_liminal.md
##########
@@ -0,0 +1,153 @@
+
+
+# Getting Started with Apache Liminal
+
+Liminal is a MachineLearning orchestration platform, or in other words: Liminal enables data scientists and data engineers to define their entire ML system and flow using configuration. From feature engineering to production monitoring - and the framework takes care of the infra behind the scenes and seamlessly integrates with the infrastructure behind the scenes.
+
+
+
+## Quick Start
+
+So, you’ve heard about liminal and decided you want to give it a try.
+
+This guide will allow you to set up your first apache Liminal environment and allow you to create some simple ML pipelines. These will be very similar to the ones you are going to build for real production scenarios. This is actually the magic behind Liminal.
+
+
+
+## Prerequisites
+
+Python 3 (3.6 and up)
+
+[Docker Desktop](https://www.docker.com/products/docker-desktop)
+
+*Note: make sure your kubernetes cluster is running*
+
+### Apache Liminal Hello World
+
+In this tutorial, we will go through setting up Liminal for the first time on your local dev machine.
+
+I’m running this on my macBook, so this will cover mac related installation aspects and kinks (if there are any).
+
+First, let’s build our examples project:
+
+In the dev folder, just clone the example code from liminal:
+
+
+```
+git clone https://github.com/apache/incubator-liminal
+```
+***Note:*** *You just cloned the entire Liminal Project, you actually need just the examples folder.*
+
+Create a python virtualenv to isolate your runs, like this:
+
+
+```
+cd incubator-liminal/examples/liminal-getting-started
+python3 -m venv env
+```
+
+
+And activate your virtual environment:
+
+
+```
+source ../env/bin/activate
+```
+
+Now we are ready to install liminal:
+
+
+```
+pip uninstall apache-liminal
+pip install apache-liminal
+rm -rf ~/liminal_home
+```
+Let's build the images you need for the example:
+```
+liminal build
+```
+The build will create docker images based on the Liminal.yml file, eash task and service will have its docker image created.
+
+
+```
+liminal deploy --clean # the --clean is for removing old containers from the previous install
+```
+The deploy will create missing containers from the images created above as well as basic Airflow containers needed to run the piplines, and deploy them to your local kubernetes cluster
+
+*Note: Liminal creates a bunch of assets, some are going to be located under: the directory in your LIMINAL_HOME env variable. But default LIMINAL_HOME is set to ~/liminal_home directory.*
+
+Now lets runs it:
+```
+liminal start
+```
+Liminal now spins up the AirFlow containers it will run the dag created based on the Liminal.yml file.
+It includes these three containers:
+* liminal-postgress
+* liminal-webserver
+* liminal-scheduler
+
+Once it finished loading the,
+Go to the airflow admin in the browser:
+
+
+```
+http://localhost:8080/admin
+```
+You can just click: [http://localhost:8080/admin](http://localhost:8080/admin)
+
+
+
Review comment:
`drawing` appears in the final resulting page
Also in other pictures below
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] aviemzur commented on a change in pull request #8: Examples and Docs
Posted by GitBox <gi...@apache.org>.
aviemzur commented on a change in pull request #8:
URL: https://github.com/apache/incubator-liminal/pull/8#discussion_r525134689
##########
File path: examples/liminal-getting-started/requirements.txt
##########
@@ -0,0 +1,2 @@
+
+pillow
Review comment:
Remove pillow from all our examples and add something that has a compatible license
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] aviemzur commented on a change in pull request #8: Examples and Docs
Posted by GitBox <gi...@apache.org>.
aviemzur commented on a change in pull request #8:
URL: https://github.com/apache/incubator-liminal/pull/8#discussion_r525126164
##########
File path: docs/source/getting_started_with_liminal.md
##########
@@ -0,0 +1,153 @@
+
+
+# Getting Started with Apache Liminal
+
+Liminal is a MachineLearning orchestration platform, or in other words: Liminal enables data scientists and data engineers to define their entire ML system and flow using configuration. From feature engineering to production monitoring - and the framework takes care of the infra behind the scenes and seamlessly integrates with the infrastructure behind the scenes.
+
+
+
+## Quick Start
+
+So, you’ve heard about liminal and decided you want to give it a try.
+
+This guide will allow you to set up your first apache Liminal environment and allow you to create some simple ML pipelines. These will be very similar to the ones you are going to build for real production scenarios. This is actually the magic behind Liminal.
+
+
+
+## Prerequisites
+
+Python 3 (3.6 and up)
+
+[Docker Desktop](https://www.docker.com/products/docker-desktop)
+
+*Note: make sure your kubernetes cluster is running*
+
+### Apache Liminal Hello World
+
+In this tutorial, we will go through setting up Liminal for the first time on your local dev machine.
+
+I’m running this on my macBook, so this will cover mac related installation aspects and kinks (if there are any).
+
+First, let’s build our examples project:
+
+In the dev folder, just clone the example code from liminal:
+
+
+```
+git clone https://github.com/apache/incubator-liminal
+```
+***Note:*** *You just cloned the entire Liminal Project, you actually need just the examples folder.*
+
+Create a python virtualenv to isolate your runs, like this:
+
+
+```
+cd incubator-liminal/examples/liminal-getting-started
+python3 -m venv env
+```
+
+
+And activate your virtual environment:
+
+
+```
+source ../env/bin/activate
+```
+
+Now we are ready to install liminal:
+
+
+```
+pip uninstall apache-liminal
+pip install apache-liminal
+rm -rf ~/liminal_home
+```
+Let's build the images you need for the example:
+```
+liminal build
+```
+The build will create docker images based on the Liminal.yml file, eash task and service will have its docker image created.
+
+
+```
+liminal deploy --clean # the --clean is for removing old containers from the previous install
+```
+The deploy will create missing containers from the images created above as well as basic Airflow containers needed to run the piplines, and deploy them to your local kubernetes cluster
+
+*Note: Liminal creates a bunch of assets, some are going to be located under: the directory in your LIMINAL_HOME env variable. But default LIMINAL_HOME is set to ~/liminal_home directory.*
+
+Now lets runs it:
+```
+liminal start
+```
+Liminal now spins up the AirFlow containers it will run the dag created based on the Liminal.yml file.
Review comment:
`AirFlow` -> `Apache Airflow`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-liminal] aviemzur commented on a change in pull request #8: Examples and Docs
Posted by GitBox <gi...@apache.org>.
aviemzur commented on a change in pull request #8:
URL: https://github.com/apache/incubator-liminal/pull/8#discussion_r525113656
##########
File path: docs/source/getting_started_with_liminal.md
##########
@@ -0,0 +1,153 @@
+
+
+# Getting Started with Apache Liminal
+
+Liminal is a MachineLearning orchestration platform, or in other words: Liminal enables data scientists and data engineers to define their entire ML system and flow using configuration. From feature engineering to production monitoring - and the framework takes care of the infra behind the scenes and seamlessly integrates with the infrastructure behind the scenes.
Review comment:
```suggestion
Liminal is a data systems orchestration platform. Liminal enables data scientists and data engineers to define their data systems and flow using configuration.
From feature engineering to production monitoring - and the framework takes care of the infra behind the scenes and seamlessly integrates with the infrastructure behind the scenes.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org