You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@liminal.apache.org by li...@apache.org on 2021/03/30 14:23:26 UTC
[incubator-liminal] branch master updated: docs (#33)
This is an automated email from the ASF dual-hosted git repository.
lior pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-liminal.git
The following commit(s) were added to refs/heads/master by this push:
new 6253f8b docs (#33)
6253f8b is described below
commit 6253f8b2c9dc244af032979ec6d462dc3e07e170
Author: Aviem Zur <av...@gmail.com>
AuthorDate: Tue Mar 30 17:23:16 2021 +0300
docs (#33)
looks good
---
.asf.yaml | 2 +-
.bumpversion.cfg | 18 ++
...o_contrib_to_liminal_docs.md => CONTRIBUTING.md | 0
README.md | 16 +-
docs/Makefile | 2 +-
docs/README.md | 44 ++++
.../liminal_architecture.md => architecture.md} | 4 +-
docs/{source => }/conf.py | 30 ++-
docs/getting_started.md | 169 +++++++++++++
docs/{source => }/index.rst | 19 +-
docs/liminal/README.md | 45 ++++
docs/liminal/advanced.liminal.yml.md | 280 +++++++++++++++++++++
docs/liminal/executors/README.md | 57 +++++
docs/liminal/executors/index.rst | 73 ++++++
.../executors/kubernetes.md} | 29 ++-
docs/liminal/images/README.md | 61 +++++
docs/liminal/images/index.rst | 75 ++++++
.../images/python.md} | 21 +-
docs/liminal/images/python_server.md | 55 ++++
docs/{source => liminal}/index.rst | 31 ++-
docs/liminal/liminal.yml.md | 220 ++++++++++++++++
docs/liminal/metrics_backends/README.md | 46 ++++
.../metrics_backends/aws_cloudwatch.md} | 30 ++-
.../{source => liminal/metrics_backends}/index.rst | 45 ++--
docs/liminal/monitoring.md | 49 ++++
docs/liminal/pipelines.md | 77 ++++++
.../liminal_yaml_guide.md => liminal/services.md} | 24 +-
docs/liminal/tasks/README.md | 50 ++++
docs/{source => liminal/tasks}/index.rst | 54 ++--
docs/liminal/tasks/python.md | 56 +++++
docs/make.bat | 2 +-
docs/{source => }/nstatic/airflow_main.png | Bin
docs/{source => }/nstatic/airflow_task_log.png | Bin
docs/{source => }/nstatic/airflow_view_dag.png | Bin
docs/{source => }/nstatic/airflow_view_log.png | Bin
docs/{source => }/nstatic/liminal_arch_001.png | Bin
docs/{source => }/nstatic/liminal_arch_002.png | Bin
docs/{source => }/nstatic/liminal_logo.png | Bin
docs/source/How_to_build_a_local_liminal.md | 70 ------
docs/source/getting_started_with_liminal.md | 182 --------------
40 files changed, 1588 insertions(+), 348 deletions(-)
diff --git a/.asf.yaml b/.asf.yaml
index 70e2126..72982c6 100644
--- a/.asf.yaml
+++ b/.asf.yaml
@@ -9,7 +9,7 @@
#
# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required bgit y applicable law or agreed to in writing,
+# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
diff --git a/.bumpversion.cfg b/.bumpversion.cfg
index d5ae75b..91e2db5 100644
--- a/.bumpversion.cfg
+++ b/.bumpversion.cfg
@@ -1,3 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
[bumpversion]
current_version = 0.0.1rc6
parse = (?P<major>\d+)\.(?P<minor>\d+)\.(?P<patch>\.?\w*)(\-(?P<release>[a-z]+)(?P<build>\d+))?
diff --git a/docs/source/howto_contrib_to_liminal_docs.md b/CONTRIBUTING.md
similarity index 100%
rename from docs/source/howto_contrib_to_liminal_docs.md
rename to CONTRIBUTING.md
diff --git a/README.md b/README.md
index 8d9e61e..9328474 100644
--- a/README.md
+++ b/README.md
@@ -34,8 +34,14 @@ non-functional tasks, and allowing them to focus on machine learning code and ar
Using simple YAML configuration, create your own schedule data pipelines (a sequence of tasks to
perform), application servers, and more.
-## Getting Started Documentation
-A simple getting stated guide for Liminal can be found [here](docs/source/getting_started_with_liminal.md)
+## Getting Started
+A simple getting stated guide for Liminal can be found [here](docs/getting_started.md)
+
+## Apache Liminal Documentation
+Full documentation of Apache Liminal can be found [here](docs/liminal)
+
+## High Level Architecture
+High level architecture documentation can be found [here](docs/architecture.md)
## Example YAML config file
```yaml
@@ -193,5 +199,9 @@ You should see the steps in your pipeline getting executed in "real time" by cli
-### Running Tests (for contributors)
+# Contributing
+
+More information on contributing can be found [here](CONTRIBUTING.md)
+
+## Running Tests (for contributors)
When doing local development and running Liminal unit-tests, make sure to set LIMINAL_STAND_ALONE_MODE=True
diff --git a/docs/Makefile b/docs/Makefile
index bc97de8..26a8d56 100644
--- a/docs/Makefile
+++ b/docs/Makefile
@@ -21,7 +21,7 @@
# You can set these variables from the command line.
SPHINXOPTS =
SPHINXBUILD = sphinx-build
-SOURCEDIR = source
+SOURCEDIR = .
BUILDDIR = build
# Put it first so that "make" without argument is like "make help".
diff --git a/docs/README.md b/docs/README.md
new file mode 100644
index 0000000..588af11
--- /dev/null
+++ b/docs/README.md
@@ -0,0 +1,44 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Apache Liminal
+
+Apache Liminal is an end-to-end platform for data engineers & scientists, allowing them to build,
+train and deploy machine learning models in a robust and agile way.
+
+The platform provides the abstractions and declarative capabilities for
+data extraction & feature engineering followed by model training and serving.
+Liminal's goal is to operationalize the machine learning process, allowing data scientists to
+quickly transition from a successful experiment to an automated pipeline of model training,
+validation, deployment and inference in production, freeing them from engineering and
+non-functional tasks, and allowing them to focus on machine learning code and artifacts.
+
+## Basics
+
+Using simple YAML configuration, create your own schedule data pipelines (a sequence of tasks to
+perform), application servers, and more.
+
+## Getting Started
+A simple getting stated guide for Liminal can be found [here](getting_started.md)
+
+## Apache Liminal Documentation
+Full documentation of Apache Liminal can be found [here](liminal)
+
+## High Level Architecture
+High level architecture documentation can be found [here](architecture.md)
diff --git a/docs/source/liminal_architecture.md b/docs/architecture.md
similarity index 98%
rename from docs/source/liminal_architecture.md
rename to docs/architecture.md
index 5522b23..d1567c2 100644
--- a/docs/source/liminal_architecture.md
+++ b/docs/architecture.md
@@ -17,7 +17,7 @@ specific language governing permissions and limitations
under the License.
-->
-# Apache Liminal High Level Architecture
+# High Level Architecture
Liminal is an end-to-end platform for data engineers & scientists, allowing them to build, train and deploy machine learning models in a robust and agile way. The platform provides the abstractions and declarative capabilities for data extraction & feature engineering followed by model training and serving. Apache Liminal's goal is to operationalise the machine learning process, allowing data scientists to quickly transition from a successful experiment to an automated pipeline of model [...]
## Motivation
@@ -62,4 +62,4 @@ _Train, Deploy and Consume._
Liminal provides its users a declarative composition capabilities to materialize these steps in a robust way, while exploiting existing frameworks and tools. e.g. Data science frameworks such as scikit-learn, Tensor flow, Keras and such, for running core data science algorithms; as numerous core mechanisms as data stores, processing engines, parallelism, schedulers, code deployment as well as batch and real-time inference.
Liminal allows the creation and wiring of these kinds of functional and non functional tasks while making the underlying infrastructure used by these tasks very easy to use and even abstracted away entirely. While handling the non-functional aspects as monitoring (in a standard fashion) deployment, scheduling, resource management and execution.
-
\ No newline at end of file
+
diff --git a/docs/source/conf.py b/docs/conf.py
similarity index 93%
rename from docs/source/conf.py
rename to docs/conf.py
index 436d0c8..5eb6888 100644
--- a/docs/source/conf.py
+++ b/docs/conf.py
@@ -29,24 +29,24 @@
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
-
+#
import os
import sys
-sys.path.insert(0, os.path.abspath('.'))
-sys.path.insert(0, os.path.abspath('../'))
+
+sys.path.insert(0, os.path.abspath('source'))
+sys.path.insert(0, os.path.abspath('/'))
# -- Project information -----------------------------------------------------
project = u'Apchae Liminal'
copyright = u'Apache Liminal'
-author = u'Amihay Zer-Kavod'
+author = u'Apache Software Foundation'
# The short X.Y version
version = u''
# The full version, including alpha/beta/rc tags
release = u'0.0.1'
-
# -- General configuration ---------------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
@@ -84,12 +84,11 @@ language = None
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
-exclude_patterns = []
+exclude_patterns = ['README.md', '**/README.md']
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = None
-
# -- Options for HTML output -------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
@@ -124,7 +123,6 @@ html_static_path = ['nstatic']
# Output file base name for HTML help builder.
htmlhelp_basename = 'ApchaeLiminaldoc'
-
# -- Options for LaTeX output ------------------------------------------------
latex_elements = {
@@ -150,10 +148,9 @@ latex_elements = {
# author, documentclass [howto, manual, or own class]).
latex_documents = [
(master_doc, 'ApchaeLiminal.tex', u'Apchae Liminal Documentation',
- u'Amihay Zer-Kavod', 'manual'),
+ u'Apache Software Foundation', 'manual'),
]
-
# -- Options for manual page output ------------------------------------------
# One entry per manual page. List of tuples
@@ -163,7 +160,6 @@ man_pages = [
[author], 1)
]
-
# -- Options for Texinfo output ----------------------------------------------
# Grouping the document tree into Texinfo files. List of tuples
@@ -175,7 +171,6 @@ texinfo_documents = [
'Miscellaneous'),
]
-
# -- Options for Epub output -------------------------------------------------
# Bibliographic Dublin Core info.
@@ -192,3 +187,14 @@ epub_title = project
# A list of files that should not be packed into the epub file.
epub_exclude_files = ['search.html']
+
+
+def on_missing_reference(app, env, node, contnode):
+ if node['reftype'] == 'any':
+ return contnode
+ else:
+ return None
+
+
+def setup(app):
+ app.connect('missing-reference', on_missing_reference)
diff --git a/docs/getting_started.md b/docs/getting_started.md
new file mode 100644
index 0000000..5e5a991
--- /dev/null
+++ b/docs/getting_started.md
@@ -0,0 +1,169 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Getting started
+
+This guide will allow you to set up your first Apache Liminal environment and allow you to create
+some simple ML pipelines. These will be very similar to the ones you are going to build for real
+production scenarios.
+
+## Prerequisites
+
+Python 3 (3.6 and up)
+
+[Docker Desktop](https://www.docker.com/products/docker-desktop)
+
+*Note: Make sure kubernetes cluster is running in docker desktop (or custom kubernetes installation
+on your machine).*
+
+## Hello World
+
+In this tutorial, we will go through setting up Liminal for the first time on your local machine.
+
+First, let’s build our examples project:
+
+In the dev folder, just clone the example code from liminal:
+
+
+```
+git clone https://github.com/apache/incubator-liminal
+```
+***Note:*** *You just cloned the entire Liminal Project, you actually only need examples folder.*
+
+Create a python virtual environment to isolate your runs:
+
+```
+cd incubator-liminal/examples/liminal-getting-started
+python3 -m venv env
+```
+
+Activate your virtual environment:
+
+```
+source env/bin/activate
+```
+
+Now we are ready to install liminal:
+
+```
+pip install apache-liminal
+```
+Let's build the images you need for the example:
+```
+liminal build
+```
+The build will create docker images based on the liminal.yml file in the `images` section.
+
+```
+liminal deploy --clean
+```
+The deploy command deploys a liminal server and deploys any liminal.yml files in your working
+directory or any of its subdirectories to your liminal home directory.
+
+*Note: liminal home directory is located in the path defined in LIMINAL_HOME env variable.
+If the LIMINAL_HOME environemnet variable is not defined, home directory defaults to
+~/liminal_home directory.*
+
+Now lets runs liminal:
+```
+liminal start
+```
+The start command spins up the liminal server containers which will run pipelines based on your
+deployed liminal.yml files.
+It runs the following three containers:
+* liminal-postgress
+* liminal-webserver
+* liminal-scheduler
+
+Once liminal server has completed starting up, you can navigate to admin UI in your browser:
+[http://localhost:8080](http://localhost:8080)
+By default liminal server starts Apache Airflow servers and admin UI will be that of Apache Airflow.
+
+
+
+
+***Important:** Set off/on toggle to activate your pipeline (DAG), nothing will happen otherwise!*
+
+You can go to tree view to see all the tasks configured in the liminal.yml file:
+[http://localhost:8080/admin/airflow/tree?dag_id=example_pipeline](
+http://localhost:8080/admin/airflow/tree?dag_id=example_pipeline
+)
+
+Now lets see what actually happened to our task:
+
+
+
+Click on “hello_world_example” and you will get this popup: \
+
+ \
+Click on “view log” button and you can see the log of the current task run: \
+
+
+
+
+## Mounted volumes
+All tasks use a mounted volume as defined in the pipeline YAML:
+```YAML
+name: GettingStartedPipeline
+volumes:
+ - volume: gettingstartedvol
+ local:
+ path: ./
+```
+In our case the mounted volume will point to the liminal hello world example.
+The hello world task will read the **hello_world.json** file from the mounted volume and will write
+the **hello_world_output.json** to it.
+
+*Note:* Each task will internally mount the volume defined above to an internal representation,
+described under the task section in the yml:
+
+```YAML
+ task:
+ ...
+ mounts:
+ - mount: taskmount
+ volume: gettingstartedvol
+ path: /mnt/vol1
+```
+
+## Here are the entire list of commands, if you want to start from scratch:
+
+```
+git clone https://github.com/apache/incubator-liminal
+cd examples
+python3 -m venv env
+source env/bin/activate
+pip uninstall apache-liminal
+pip install apache-liminal
+Liminal build
+liminal deploy --clean
+liminal start
+```
+
+## Closing up
+
+To make sure liminal containers are stopped use:
+```
+liminal stop
+```
+
+To deactivate the python virtual env use:
+```
+deactivate
+```
diff --git a/docs/source/index.rst b/docs/index.rst
similarity index 71%
copy from docs/source/index.rst
copy to docs/index.rst
index 1f4f3a3..f1decfd 100644
--- a/docs/source/index.rst
+++ b/docs/index.rst
@@ -24,17 +24,20 @@
under the License.
..
-Welcome to Apchae Liminal's documentation!
-==========================================
+Apache Liminal
+==============
+
+Apache Liminal is a data systems orchestration platform. Liminal enables data scientists and data
+engineers to define their data systems and flow using configuration. From feature engineering to
+production monitoring - and the framework takes care of the infra behind the scenes and seamlessly
+integrates with the infrastructure behind the scenes.
.. toctree::
- :maxdepth: 2
- :caption: Contents:
+ :maxdepth: 1
- getting_started_with_liminal
- liminal_architecture
- liminal_yaml_guide
- howto_contrib_to_liminal_docs
+ getting_started
+ liminal/index
+ architecture
Indices and tables
==================
diff --git a/docs/liminal/README.md b/docs/liminal/README.md
new file mode 100644
index 0000000..3a5a55d
--- /dev/null
+++ b/docs/liminal/README.md
@@ -0,0 +1,45 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Apache Liminal Documentation
+
+## Overview
+
+Apache Liminal is an end-to-end platform for data engineers & scientists, allowing them to build,
+train and deploy machine learning models in a robust and agile way.
+
+The platform provides the abstractions and declarative capabilities for
+data extraction & feature engineering followed by model training and serving.
+Liminal's goal is to operationalize the machine learning process, allowing data scientists to
+quickly transition from a successful experiment to an automated pipeline of model training,
+validation, deployment and inference in production, freeing them from engineering and
+non-functional tasks, and allowing them to focus on machine learning code and artifacts.
+
+## Chapters
+
+1. [liminal.yml](liminal.yml.md)
+2. [Images](images)
+3. [Pipelines](pipelines.md)
+4. [Tasks](tasks)
+5. [Services](services.md)
+6. [Monitoring](monitoring.md)
+7. [Metrics Backends](metrics_backends)
+8. [Alerts Backends](alerts_backends)
+9. [Advanced liminal.yml](advanced.liminal.yml.md)
+10. [Executors](executors)
diff --git a/docs/liminal/advanced.liminal.yml.md b/docs/liminal/advanced.liminal.yml.md
new file mode 100644
index 0000000..77442dc
--- /dev/null
+++ b/docs/liminal/advanced.liminal.yml.md
@@ -0,0 +1,280 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Advanced liminal.yml
+
+In this section you will learn about advanced features of liminal.yml
+
+## Variables
+
+Much like in programming languages, you can define variables for re-use across your
+liminal.yml file.
+
+```yaml
+variables:
+ myvar1: myvalue1
+ myvar2: myvalue2
+```
+
+In the `variables` section of your liminal.yml you can define your variables as key value pairs
+
+## Placeholders
+
+You can use placeholders of format `{{myplaceholder}}` in most any string value in your liminal.yml
+Make sure that any string that includes placeholders is surrounded by single or double quotes.
+
+For example:
+```yaml
+variables:
+ image_name: myorg/myrepo:myapp
+images:
+ - image: '{{image_name}}'
+ type: python
+ source: .
+pipelines:
+ - pipeline: my_pipeline
+ tasks:
+ - task: my_python_task
+ type: python
+ image: "{{image_name}}"
+ cmd: python -u my_module.py
+```
+
+### Placeholders rendering
+
+The process of rendering placeholders checks for values in several places, by order.
+
+#### Pipeline placeholder rendering
+
+When running pipelines placeholders will be rendered by replacing values from sources in the
+following order:
+
+1. Current [DAG run conf](https://airflow.apache.org/docs/apache-airflow/stable/dag-run.html)
+(Airflow).
+2. liminal.yml [variables](#variables) section.
+3. [Airflow variables](https://airflow.apache.org/docs/apache-airflow/stable/howto/variable.html)
+(Airflow).
+4. [Airflow macros](https://airflow.apache.org/docs/apache-airflow/stable/macros-ref.html) (Airflow)
+. For example: `"{{yesterday_ds}}"`. For more information see:
+5. Airflow [Jinja Templating](https://airflow.apache.org/docs/apache-airflow/stable/concepts.html#jinja-templating)
+(Airflow).
+
+#### Build placeholder rendering
+
+When using `liminal build` placeholders will be rendered by replacing values from sources in the
+following order:
+
+1. Environment variables.
+2. liminal.yml [variables](#variables) section.
+
+## Task variables
+
+In addition to the variables section you can also set specific variables for specific `task`s using
+the `variables` attribute of that task. For example:
+
+```yaml
+ - task: my_task
+ type: python
+ cmd: python -u my_module.py
+ variables:
+ myvar1: myvalue1
+ myvar2: myvalue2
+```
+
+You can also pass a reference to a variable map (dictionary) set in your variables section:
+
+```yaml
+variables:
+ my_variable_map:
+ myvar1: myvalue1
+ myvar2: myvalue2
+ my_variable_map2:
+ myvar1: myvalue_a
+ myvar2: myvalue_x
+pipelines:
+ - pipeline: my_pipeline
+ tasks:
+ - task: my_task1
+ type: python
+ cmd: python -u my_module.py
+ variables: my_variable_map1
+ - task: my_task2
+ type: python
+ cmd: python -u my_module.py
+ variables: my_variable_map2
+```
+
+## Executors
+
+For fully detailed information on executors see: [executors](executors).
+
+Each `task` in your pipelines has a reference to an executor from the `executors` section of your
+liminal.yml file. If not specified, the appropriate default executor for that task type is used.
+
+```yaml
+executors:
+ - executor: my_kubernetes_executor
+ type: kubernetes
+ resources:
+ request_memory: 128Mi
+pipelines:
+ pipeline: my_pipeline
+ tasks:
+ - task: my_python_task
+ type: python
+ image: myorg/myrepo:mypythonapp
+ executor: my_kubernetes_executor
+ cmd: python -u my_module.py
+```
+
+In the example above we define an `executor` of type `kubernetes` with custom resources
+configuration.
+
+`executors` is a section in the root of your liminal.yml file and is a list of `executor`s defined
+by the following attributes:
+
+### executor attributes
+
+`executor`: name of your executor.
+
+`type`: type of the executor. The current available image types are: `kubernetes` and `emr`.
+
+Different executor types support their own additional configuration.
+
+## Task Defaults
+
+`task_defaults` is a section in the root of your liminal.yml file in which default attributes can be
+set for each `task` type.
+
+```yaml
+task_defaults:
+ python:
+ executor: my_kubernetes_executor
+ env_vars:
+ env: {{env}}
+ foo: bar
+```
+
+In the example above we set default attributes for any `python` task in any pipeline in the liminal
+system defined by our liminal.yml file. Each `python` task that does not set these attributes will
+default to the setting in `task_defaults`.
+
+If the same attribute is defined in both `task_defaults` and in the `task`, definitions from the
+`task` take precedence. If a map (dictionary) of values (for example `env_vars`) is defined in both
+`task_defaults` the two maps will be merged, with definitions in the `task` taking precedence in
+case key is defined in both maps.
+
+## Pipeline Defaults
+
+`pipeline_defaults` is a section in the root of your liminal.yml file in which default attributes
+can be set for each `pipeline` in the liminal system defined in your liminal.yml file.
+
+```yaml
+pipeline_defaults:
+ start_date: 2021-01-01
+ timeout_minutes: 30
+```
+
+In the example above we set default attributes for any `pipeline` defined in our liminal.yml file.
+Each `pipeline` that does not set these attributes will default to the setting in
+`pipeline_defaults`.
+
+If the same attribute is defined in both `pipeline_defaults` and in the `pipeline`, definitions
+from the `pipeline` take precedence.
+
+If `tasks` section is defined in `pipeline_defaults` each pipeline defined in our liminal.yml file
+will have the tasks defined in `pipeline_defaults`.
+
+A special `task` type `pipeline` may be used in `pipeline_defaults` `tasks` section. This task
+type is interpreted as "tasks defined in pipeline go here". This allows flexibility of defining
+common tasks to be run before and after the tasks of each pipeline defined in our liminal.yml
+file. For example:
+
+```yaml
+pipeline_defaults:
+ tasks:
+ - task: my_common_setup_task
+ type: python
+ image: myorg/myrepo:mypythonapp
+ cmd: python -u my_setup_module.py
+ - task: pipeline_tasks
+ type: pipeline
+ - task: my_common_teardown_task1
+ type: python
+ image: myorg/myrepo:mypythonapp
+ cmd: python -u my_teardown_module1.py
+ - task: my_common_teardown_task2
+ type: python
+ image: myorg/myrepo:mypythonapp
+ cmd: python -u my_teardown_module2.py
+```
+
+In the example above we set a list of `tasks` in `pipeline_defaults` which leads to each pipeline
+defined in our liminal.yml file will have `my_common_setup_task` run before its tasks and
+`my_common_teardown_task1` and `my_common_teardown_task2` after its tasks.
+
+## Inheritence
+
+Each liminal.yml defines a subliminal layer of a liminal system. A subliminal layer can inherit
+attributes of a superliminal layer by specifying `super: name_of_super` in the root of the yml.
+
+```yaml
+name: my_system
+super: my_super
+```
+
+A super is found by this name if a liminal.yml file in your environment exists with that name. A
+superliminal layer liminal.yml file needs to define its `type` as `super`:
+
+```yaml
+name: my_super
+type: super
+```
+
+If not specified, the `type` of a liminal.yml file defaults to `sub` (subliminal).
+
+A superliminal can also inherit from another superliminal by defining its own `super` attribute.
+
+```yaml
+name: my_super
+type: super
+super: my_other_super
+```
+
+A liminal system can have 1 subliminal layer but many superliminal layers using inheritence.
+This allows us to chain common behaviors of our systems into several superliminal layers.
+
+If no `super` is defined for a liminal.yml file then it defaults to having the
+[hyperliminal](https://github.com/Natural-Intelligence/liminal/blob/master/liminal/core/config/defaults/hyperliminal/liminal.yml)
+be its super.
+
+### superliminal attribtues
+
+A superliminal layer can define the following attributes:
+
+```
+variables
+executors
+monitoring
+task_defaults
+pipeline_defaults
+```
+
+If the same attribute section is defined in both a superliminal and a lower layer of the system
+they will be merged, with key collisions favoring the lower level layer.
diff --git a/docs/liminal/executors/README.md b/docs/liminal/executors/README.md
new file mode 100644
index 0000000..efe8614
--- /dev/null
+++ b/docs/liminal/executors/README.md
@@ -0,0 +1,57 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Executors
+
+Each `task` in your pipelines has a reference to an executor from the `executors` section of your
+liminal.yml file. If not specified, the appropriate default executor for that task type is used.
+
+```yaml
+executors:
+ - executor: my_kubernetes_executor
+ type: kubernetes
+ resources:
+ request_memory: 128Mi
+pipelines:
+ pipeline: my_pipeline
+ tasks:
+ - task: my_python_task
+ type: python
+ image: myorg/myrepo:mypythonapp
+ executor: my_kubernetes_executor
+ cmd: python -u my_module.py
+```
+
+In the example above we define an `executor` of type `kubernetes` with custom resources
+configuration.
+
+`executors` is a section in the root of your liminal.yml file and is a list of `executor`s defined
+by the following attributes:
+
+## executor attributes
+
+`executor`: name of your executor.
+
+`type`: type of the executor. The current available image types are: `kubernetes` and `emr`.
+
+Different executor types support their own additional configuration.
+
+## task types
+
+1. [kubernetes](kubernetes.md)
diff --git a/docs/liminal/executors/index.rst b/docs/liminal/executors/index.rst
new file mode 100644
index 0000000..ea98cc5
--- /dev/null
+++ b/docs/liminal/executors/index.rst
@@ -0,0 +1,73 @@
+.. Apchae Liminal documentation master file, created by
+ sphinx-quickstart on Sun Nov 15 08:45:27 2020.
+ You can adapt this file completely to your liking, but it should at least
+ contain the root `toctree` directive.
+
+..
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+..
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+..
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+..
+
+Executors
+=========
+
+Each ``task`` in your pipelines has a reference to an executor from the ``executors`` section of
+your liminal.yml file. If not specified, the appropriate default executor for that task type is
+used.
+
+.. code-block:: yaml
+
+ executors:
+ - executor: my_kubernetes_executor
+ type: kubernetes
+ resources:
+ request_memory: 128Mi
+ pipelines:
+ pipeline: my_pipeline
+ tasks:
+ - task: my_python_task
+ type: python
+ image: myorg/myrepo:mypythonapp
+ executor: my_kubernetes_executor
+ cmd: python -u my_module.py
+
+..
+
+In the example above we define an ``executor`` of type ``kubernetes`` with custom resources
+configuration.
+
+``executors`` is a section in the root of your liminal.yml file and is a list of ``executor`` s
+defined by the following attributes:
+
+executor attributes
+'''''''''''''''''''
+
+``executor``: name of your executor.
+
+``type``: type of the executor. The current available image types are: ``kubernetes`` and ``emr``.
+
+Different executor types support their own additional configuration.
+
+executor types
+''''''''''''''
+
+.. toctree::
+ :maxdepth: 1
+
+ kubernetes
diff --git a/docs/source/liminal_yaml_guide.md b/docs/liminal/executors/kubernetes.md
similarity index 57%
copy from docs/source/liminal_yaml_guide.md
copy to docs/liminal/executors/kubernetes.md
index 3017755..17332af 100644
--- a/docs/source/liminal_yaml_guide.md
+++ b/docs/liminal/executors/kubernetes.md
@@ -17,10 +17,27 @@ specific language governing permissions and limitations
under the License.
-->
-# Liminal yaml file explained
+# kubernetes executor
-## Main Structure of liminal.yml
-TDB
-## main properties
-### task
-### service
\ No newline at end of file
+The `kubernetes` executor allows you to run tasks on kubernetes.
+
+```yaml
+ - executor: my_kubernetes_executor
+ type: kubernetes
+```
+
+## attributes
+
+If running pipelines on Airflow, any attribute of
+[KubernetesPodOperator](https://airflow.apache.org/docs/apache-airflow/1.10.12/_api/airflow/contrib/operators/kubernetes_pod_operator/index.html)
+can be set as an attribute of the executor.
+
+Note that some of these attributes are set in the `task`:
+```
+image
+cmd
+mounts
+name
+env_vars
+```
+For example, see: [python task](../tasks/python.md)
diff --git a/docs/liminal/images/README.md b/docs/liminal/images/README.md
new file mode 100644
index 0000000..bee8a22
--- /dev/null
+++ b/docs/liminal/images/README.md
@@ -0,0 +1,61 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Images
+
+In the `images` section you can configure how to pack your code into docker images. This can be
+achieved in liminal easily, without any Docker knowledge needed by setting just a few attributes:
+
+```yaml
+images:
+ - image: myorg/myrepo:mypythonapp
+ type: python
+ source: .
+ - image: myorg/myrepo:myserver
+ type: python_server
+ source: path/to/my/server/code
+ endpoints:
+ - endpoint: /myendpoint
+ module: my_module
+```
+
+`images` is a section in the root lof your liminal.yml file and is a list of `image`s, defined
+by the following attributes:
+
+## image attributes
+
+`image`: name of your image, usually will be in the form of `user/repository:tag` common in docker
+image repositories such as Docker Hub, but for local development this can be any string.
+
+`type`: type of the image. Usually this will match your coding language, or your coding langauge
+followed by an `_` and a specific style of application (such as a server).
+
+`source`: location of source files to include in the image, this can be a relative path within your
+project, or even `.` which means the entire project should be packaged in the image.
+
+`build_cmd`: certain languages require to be built or compiled, here you can provide your build
+command to be executed in order to build your code.
+
+Other image types might require additional configuration, for example, servers require `endpoints`
+to be configured.
+
+## image types
+
+1. [python](python.md)
+2. [python server](python_server.md)
diff --git a/docs/liminal/images/index.rst b/docs/liminal/images/index.rst
new file mode 100644
index 0000000..84daa59
--- /dev/null
+++ b/docs/liminal/images/index.rst
@@ -0,0 +1,75 @@
+.. Apchae Liminal documentation master file, created by
+ sphinx-quickstart on Sun Nov 15 08:45:27 2020.
+ You can adapt this file completely to your liking, but it should at least
+ contain the root `toctree` directive.
+
+..
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+..
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+..
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+..
+
+Images
+======
+
+In the ``images`` section you can configure how to pack your code into docker images. This can be
+achieved in liminal easily, without any Docker knowledge needed by setting just a few attributes:
+
+.. code-block:: yaml
+
+ images:
+ - image: myorg/myrepo:mypythonapp
+ type: python
+ source: .
+ - image: myorg/myrepo:myserver
+ type: python_server
+ source: path/to/my/server/code
+ endpoints:
+ - endpoint: /myendpoint
+ module: my_module
+..
+
+``images`` is a section in the root lof your liminal.yml file and is a list of
+``image`` s, defined by the following attributes:
+
+image attributes
+''''''''''''''''
+
+``image``: name of your image, usually will be in the form of ``user/repository:tag`` common in
+docker image repositories such as Docker Hub, but for local development this can be any string.
+
+``type``: type of the image. Usually this will match your coding language, or your coding langauge
+followed by an ``_`` and a specific style of application (such as a server).
+
+``source``: location of source files to include in the image, this can be a relative path within
+your project, or even ``.`` which means the entire project should be packaged in the image.
+
+``build_cmd``: certain languages require to be built or compiled, here you can provide your build
+command to be executed in order to build your code.
+
+Other image types might require additional configuration, for example, servers require ``endpoints``
+to be configured.
+
+image types
+'''''''''''
+
+.. toctree::
+ :maxdepth: 1
+
+ python
+ python_server
diff --git a/docs/source/liminal_yaml_guide.md b/docs/liminal/images/python.md
similarity index 64%
copy from docs/source/liminal_yaml_guide.md
copy to docs/liminal/images/python.md
index 3017755..ab75739 100644
--- a/docs/source/liminal_yaml_guide.md
+++ b/docs/liminal/images/python.md
@@ -17,10 +17,19 @@ specific language governing permissions and limitations
under the License.
-->
-# Liminal yaml file explained
+# python image
-## Main Structure of liminal.yml
-TDB
-## main properties
-### task
-### service
\ No newline at end of file
+The `python` image builder packages your python code as a docker image.
+
+```yaml
+ - image: myorg/myrepo:mypythonapp
+ type: python
+ source: relative/path/to/source
+```
+
+## attributes
+
+`image`: name of your image.
+
+`source`: location of source files to include in the image, this can be a relative path within your
+project, or even `.` which means the entire project should be packaged in the image.
diff --git a/docs/liminal/images/python_server.md b/docs/liminal/images/python_server.md
new file mode 100644
index 0000000..09adecc
--- /dev/null
+++ b/docs/liminal/images/python_server.md
@@ -0,0 +1,55 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# python_server image
+
+The `python_server` image builder packages your python code as a docker image which starts a web
+server to serve your code via HTTP calls.
+
+```yaml
+ - image: myorg/myrepo:mypythonserver
+ type: python_server
+ source: relative/path/to/source
+ endpoints:
+ - endpoint: /myendpoint
+ module: my_module
+ function: my_function
+ - endpoint: /myotherendpoint
+ module: my_module2
+ function: my_function2
+```
+
+## attributes
+
+`image`: name of your image.
+
+`source`: location of source files to include in the image, this can be a relative path within your
+project, or even `.` which means the entire project should be packaged in the image.
+
+`endpoint`: list of `endpoint`s to expose in the server.
+
+## endpoint attributes
+
+`endpoint`: path to your endpoint in the server.
+
+`module`: module containing your user code.
+
+`function`: name of the function within the module that should be called when handling this
+endpoint. the function should expect a dictionary as an input (or None if no input) and should
+return a string to return in the response to the HTTP call.
diff --git a/docs/source/index.rst b/docs/liminal/index.rst
similarity index 76%
copy from docs/source/index.rst
copy to docs/liminal/index.rst
index 1f4f3a3..2b152d6 100644
--- a/docs/source/index.rst
+++ b/docs/liminal/index.rst
@@ -24,21 +24,20 @@
under the License.
..
-Welcome to Apchae Liminal's documentation!
-==========================================
+Apache Liminal Documentation
+============================
-.. toctree::
- :maxdepth: 2
- :caption: Contents:
-
- getting_started_with_liminal
- liminal_architecture
- liminal_yaml_guide
- howto_contrib_to_liminal_docs
+Chapters:
-Indices and tables
-==================
-
-* :ref:`genindex`
-* :ref:`modindex`
-* :ref:`search`
+.. toctree::
+ :maxdepth: 1
+
+ liminal.yml
+ images/index.rst
+ pipelines
+ tasks/index.rst
+ services
+ monitoring
+ metrics_backends/index.rst
+ advanced.liminal.yml
+ executors/index.rst
diff --git a/docs/liminal/liminal.yml.md b/docs/liminal/liminal.yml.md
new file mode 100644
index 0000000..91b3f4a
--- /dev/null
+++ b/docs/liminal/liminal.yml.md
@@ -0,0 +1,220 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+## liminal.yml
+
+Definition of your own liminal system is done via yml configuration file named liminal.yml in
+your project. This yml is the one place in which you define all characteristics and behavior of your
+liminal system.
+In this section we'll go over the anatomy of the liminal.yml file.
+
+### root attributes
+
+At the root level of your limimal.yml you can define the most basic root characteristics of your
+liminal system such as name and owner:
+
+```yaml
+name: MyLiminalSystem
+owner: Bosco Albert Baracus
+```
+
+### images
+
+For fully detailed information on pipelines see: [images](images).
+
+In the `images` section you can configure how to pack your code into docker images. This can be
+achieved in liminal easily, without any Docker knowledge needed by setting just a few attributes:
+
+```yaml
+images:
+ - image: myorg/myrepo:mypythonapp
+ type: python
+ source: .
+ - image: myorg/myrepo:myserver
+ type: python_server
+ source: path/to/my/server/code
+ endpoints:
+ - endpoint: /myendpoint
+ module: my_module
+ function: my_function
+```
+
+`images` is a section in the root lof your liminal.yml file and is a list of `image`s, defined
+by the following attributes:
+
+#### image attributes
+
+`image`: name of your image, usually will be in the form of `user/repository:tag` common in docker
+image repositories such as Docker Hub, but for local development this can be any string.
+
+`type`: type of the image. Usually this will match your coding language, or your coding langauge
+followed by an `_` and a specific style of application (such as a server).
+The current available image types are: `python`.
+
+`source`: location of source files to include in the image, this can be a relative path within your
+project, or even `.` which means the entire project should be packaged in the image.
+
+`build_cmd`: certain languages require to be built or compiled, here you can provide your build
+command to be executed in order to build your code.
+
+Other image types might require additional configuration, for example, servers require `endpoints`
+to be configured.
+
+### pipelines
+
+For fully detailed information on pipelines see: [pipelines](pipelines.md).
+
+In the `pipelines` section you can configure scheduled/manually run pipelines of tasks to be
+executed sequentially:
+
+```yaml
+pipelines:
+ - pipeline: my_data_pipeline
+ timeout_minutes: 30
+ start_date: 2020-03-01
+ schedule: 0 10 * * *
+ tasks:
+ - task: my_sql_task
+ type: sql
+ query: "SELECT * FROM
+ {{my_database_name}}.{{my_table_name}}
+ WHERE event_date_prt >=
+ '{{yesterday_ds}}'"
+ AND cms_platform = 'xsite'
+ output_table: my_db.my_out_table
+ output_path: s3://my_bky/{{env}}/mydir
+ - task: my_python_task
+ type: python
+ image: myorg/myrepo:pythonapp
+ cmd: python my_python_app.py
+ env_vars:
+ env: {{env}}
+ fizz: buzz
+ - task: my_python_task
+ type: python
+ image: myorg/myrepo:mypythonapp
+ cmd: python -u my_module.py
+ env_vars:
+ env: {{env}}
+ fizz: buzz
+```
+
+`pipelines` is a section in the root lof your liminal.yml file and is a list of `pipeline`s defined
+by the following attributes:
+
+#### pipeline attributes
+
+`pipeline`: name of your pipeline (must be unique per liminal server).
+
+`timeout_minutes`: maximum allowed pipeline run time in minutes, if run exceeds this time, pipeline
+and all running tasks will fail.
+
+`start_date`: start date for the pipeline.
+
+`schedule`: to be configured if the pipeline should run on a schedule. Format is
+[cron expression](https://en.wikipedia.org/wiki/Cron#CRON_expression).
+
+`tasks`: list of `task`s, defined by the following attributes:
+
+##### task attributes
+
+For fully detailed information on tasks see: [tasks](tasks).
+
+`task`: name of your task (must be made of alphanumeric, dash and/or underscore characters only).
+
+`type`: type of the task. Examples of available task types are: `python`.
+and more..
+
+Different task types require their own additional configuration. For example, `python` task requires
+`image` to be configured.
+
+### services
+
+For fully detailed information on services see: [services](services.md).
+
+In the `services` section you can configure constantly running applications such as
+servers.
+
+```yaml
+services:
+ - service: my_server
+ image: myorg/myrepo:myserver
+```
+
+`services` is a section in the root lof your liminal.yml file and is a list of `service`s, defined
+by the following attributes:
+
+#### service attributes
+
+`service`: name of your service.
+
+`image`: the service's docker image.
+
+### monitoring
+
+For fully detailed information on monitoring see: [monitoring](monitoring.md).
+
+In the `monitoring` section you can configure monitoring for your pipelines and services.
+
+```yaml
+monitoring:
+ metrics_backends:
+ - metrics_backend: cloudwatch_metrics
+ type: aws_cloudwatch
+ namespace: DataPipeline
+ AWS_REGION_NAME: us-east-1
+ alerts_backends:
+ - alerts_backend: cloudwatch_alerts
+ type: aws_cloudwatch
+ metrics_backend: cloudwatch_metrics
+ ok_actions: ['arn:aws:sns:...']
+ alarm_actions: ['arn:aws:sns:...']
+```
+
+`monitoring` is a section in the root lof your liminal.yml file and is a list of `metrics_backend`s and
+a list of `alerts_backend`s.
+
+`metrics_backend`s are where metrics for your pipelines are automatically sent.
+
+`alerts_backends`s automatically register alerts based on `metrics_backend` they are paired with.
+
+#### metrics_backend attributes
+
+`metrics_backend`: name of your metrics backend
+
+`type`: type of the metrics backend. The current available metrics backends are: `aws_cloudwatch`.
+
+Different metrics backend types require their own additional configuration. For example,
+`aws_cloudwatch` metrics backend requires `namespace` to be configured.
+
+#### alerts_backend attributes
+
+`alerts_backend`: name of your alerts backend
+
+`type`: type of the alerts backend. The current available alerts backends are: `aws_cloudwatch`.
+
+`metrics_backend`: name of the metrics backends to register alerts for.
+
+Different alerts backend types require their own additional configuration. For example,
+`aws_cloudwatch` alerts backend requires `alarm_actions` to be configured.
+
+### advanced topics
+
+For documentation of more advanced features of liminal.yml see:
+[advanced liminal.yml](advanced.liminal.yml.md)
diff --git a/docs/liminal/metrics_backends/README.md b/docs/liminal/metrics_backends/README.md
new file mode 100644
index 0000000..fc6f3ba
--- /dev/null
+++ b/docs/liminal/metrics_backends/README.md
@@ -0,0 +1,46 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Metrics Backends
+
+`metrics_backend` is the definition of a metrics backend to which all automatically
+generated metrics from your pipeline are sent to and is part of the `metrics_backends` list in the
+`monitoring` section of your liminal.yml
+
+```yaml
+ - metrics_backend: cloudwatch_metrics
+ type: aws_cloudwatch
+ namespace: DataPipeline
+ AWS_REGION_NAME: us-east-1
+```
+
+A `metrics_backend` is defined by the following attribtues:
+
+## metrics_backend attributes
+
+`metrics_backend`: name of your metrics backend
+
+`type`: type of the metrics backend. The current available metrics backends are: `aws_cloudwatch`.
+
+Different metrics backend types require their own additional configuration. For example,
+`aws_cloudwatch` metrics backend requires `namespace` to be configured.
+
+## metrics_backend types
+
+1. [aws cloudwatch](aws_cloudwatch.md)
diff --git a/docs/source/liminal_yaml_guide.md b/docs/liminal/metrics_backends/aws_cloudwatch.md
similarity index 52%
copy from docs/source/liminal_yaml_guide.md
copy to docs/liminal/metrics_backends/aws_cloudwatch.md
index 3017755..7c23590 100644
--- a/docs/source/liminal_yaml_guide.md
+++ b/docs/liminal/metrics_backends/aws_cloudwatch.md
@@ -17,10 +17,28 @@ specific language governing permissions and limitations
under the License.
-->
-# Liminal yaml file explained
+# aws_cloudwatch metrics backend
-## Main Structure of liminal.yml
-TDB
-## main properties
-### task
-### service
\ No newline at end of file
+The `aws_cloudwatch` metrics backend allows you to send all automatically generated metrics from
+your pipelines to AWS CloudWatch.
+
+```yaml
+ - metrics_backend: cloudwatch_metrics
+ type: aws_cloudwatch
+ namespace: DataPipeline
+ AWS_REGION_NAME: us-east-1
+```
+
+## attributes
+
+`metrics_backend`: name of your metrics backend.
+
+`type`: type of the metrics backend. The current available metrics backends are: `aws_cloudwatch`.
+
+`namespace`: target namespace on AWS CloudWatch.
+
+`AWS_REGION_NAME`: target AWS region to report metrics to.
+
+`AWS_ACCESS_KEY_ID`: AWS access key id (optional).
+
+`AWS_SECRET_ACCESS_KEY`: AWS secret access key (optional).
diff --git a/docs/source/index.rst b/docs/liminal/metrics_backends/index.rst
similarity index 52%
copy from docs/source/index.rst
copy to docs/liminal/metrics_backends/index.rst
index 1f4f3a3..c32772f 100644
--- a/docs/source/index.rst
+++ b/docs/liminal/metrics_backends/index.rst
@@ -24,21 +24,38 @@
under the License.
..
-Welcome to Apchae Liminal's documentation!
-==========================================
+Metrics Backends
+================
-.. toctree::
- :maxdepth: 2
- :caption: Contents:
+``metrics_backend`` is the definition of a metrics backend to which all automatically
+generated metrics from your pipeline are sent to and is part of the ``metrics_backends`` list in the
+``monitoring`` section of your liminal.yml
+
+.. code-block:: yaml
+
+ - metrics_backend: cloudwatch_metrics
+ type: aws_cloudwatch
+ namespace: DataPipeline
+ AWS_REGION_NAME: us-east-1
+
+..
+
+A ``metrics_backend`` is defined by the following attributes:
- getting_started_with_liminal
- liminal_architecture
- liminal_yaml_guide
- howto_contrib_to_liminal_docs
+metrics_backend attributes
+''''''''''''''''''''''''''
-Indices and tables
-==================
+``metrics_backend``: name of your metrics backend
+
+``type``: type of the metrics backend. The current available metrics backends are: ``aws_cloudwatch``.
+
+Different metrics backend types require their own additional configuration. For example,
+``aws_cloudwatch`` metrics backend requires ``namespace`` to be configured.
+
+metrics_backend types
+'''''''''''''''''''''
+
+.. toctree::
+ :maxdepth: 1
-* :ref:`genindex`
-* :ref:`modindex`
-* :ref:`search`
+ aws_cloudwatch
diff --git a/docs/liminal/monitoring.md b/docs/liminal/monitoring.md
new file mode 100644
index 0000000..fa73705
--- /dev/null
+++ b/docs/liminal/monitoring.md
@@ -0,0 +1,49 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Monitoring
+
+In the `monitoring` section you can configure monitoring for your pipelines and services.
+
+```yaml
+monitoring:
+ metrics_backends:
+ - metrics_backend: cloudwatch_metrics
+ type: aws_cloudwatch
+ namespace: DataPipeline
+ AWS_REGION_NAME: us-east-1
+```
+
+`monitoring` is a section in the root lof your liminal.yml file and is a list of `metrics_backend`s and
+a list of `alerts_backend`s.
+
+`metrics_backend`s are where metrics for your pipelines are automatically sent.
+
+`alerts_backends`s automatically register alerts based on `metrics_backend` they are paired with.
+
+## metrics_backend attributes
+
+For fully detailed information on metrics backends see: [metrics backends](metrics_backends).
+
+`metrics_backend`: name of your metrics backend
+
+`type`: type of the metrics backend. The current available metrics backends are: `aws_cloudwatch`.
+
+Different metrics backend types require their own additional configuration. For example,
+`aws_cloudwatch` metrics backend requires `namespace` to be configured.
diff --git a/docs/liminal/pipelines.md b/docs/liminal/pipelines.md
new file mode 100644
index 0000000..070a8ed
--- /dev/null
+++ b/docs/liminal/pipelines.md
@@ -0,0 +1,77 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Pipelines
+
+In the `pipelines` section you can configure scheduled/manually run pipelines of tasks to be
+executed sequentially:
+
+```yaml
+pipelines:
+ - pipeline: my_data_pipeline
+ timeout_minutes: 30
+ start_date: 2020-03-01
+ schedule: 0 10 * * *
+ tasks:
+ - task: my_sql_task
+ type: sql
+ query: "SELECT * FROM
+ {{my_database_name}}.{{my_table_name}}
+ WHERE event_date_prt >=
+ '{{yesterday_ds}}'"
+ AND cms_platform = 'xsite'
+ output_table: my_db.my_out_table
+ output_path: s3://my_bky/{{env}}/mydir
+ - task: my_python_task
+ type: python
+ image: myorg/myrepo:pythonapp
+ cmd: python my_python_app.py
+ env_vars:
+ env: {{env}}
+ fizz: buzz
+```
+
+`pipelines` is a section in the root lof your liminal.yml file and is a list of `pipeline`s defined
+by the following attributes:
+
+## pipeline attributes
+
+`pipeline`: name of your pipeline (must be unique per liminal server).
+
+`timeout_minutes`: maximum allowed pipeline run time in minutes, if run exceeds this time, pipeline
+and all running tasks will fail.
+
+`start_date`: start date for the pipeline.
+
+`schedule`: to be configured if the pipeline should run on a schedule. Format is
+[cron expression](https://en.wikipedia.org/wiki/Cron#CRON_expression).
+
+`tasks`: list of `task`s, defined by the following attributes:
+
+## task attributes
+
+For fully detailed information on tasks see: [tasks](tasks).
+
+`task`: name of your task (must be made of alphanumeric, dash and/or underscore characters only).
+
+`type`: type of the task. Examples of available task types are: `python`.
+and more..
+
+Different task types require their own additional configuration. For example, `python` task requires
+`image` to be configured.
diff --git a/docs/source/liminal_yaml_guide.md b/docs/liminal/services.md
similarity index 65%
rename from docs/source/liminal_yaml_guide.md
rename to docs/liminal/services.md
index 3017755..9e5ca11 100644
--- a/docs/source/liminal_yaml_guide.md
+++ b/docs/liminal/services.md
@@ -17,10 +17,22 @@ specific language governing permissions and limitations
under the License.
-->
-# Liminal yaml file explained
+# Services
-## Main Structure of liminal.yml
-TDB
-## main properties
-### task
-### service
\ No newline at end of file
+In the `services` section you can configure constantly running applications such as
+servers.
+
+```yaml
+services:
+ - service: my_server
+ image: myorg/myrepo:myserver
+```
+
+`services` is a section in the root lof your liminal.yml file and is a list of `service`s, defined
+by the following attributes:
+
+## service attributes
+
+`service`: name of your service.
+
+`image`: the service's docker image.
diff --git a/docs/liminal/tasks/README.md b/docs/liminal/tasks/README.md
new file mode 100644
index 0000000..1f64fd8
--- /dev/null
+++ b/docs/liminal/tasks/README.md
@@ -0,0 +1,50 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Tasks
+
+`task` is the definition of a specific step in your pipeline, and is part of the `tasks` list
+in your pipeline definition.
+
+For fully detailed information on pipelines see: [pipelines](../pipelines.md).
+
+```yaml
+ - task: my_python_task
+ image: myorg/myrepo:mypythonapp
+ cmd: python -u my_module.py
+ env_vars:
+ env: {{env}}
+ fizz: buzz
+```
+
+A `task` is defined by the following attributes:
+
+## task attributes
+
+`task`: name of your task (must be made of alphanumeric, dash and/or underscore characters only).
+
+`type`: type of the task. Examples of available task types are: `python`
+and more..
+
+Different task types require their own additional configuration. For example, `python` task requires
+`image` to be configured.
+
+## task types
+
+1. [python](python.md)
diff --git a/docs/source/index.rst b/docs/liminal/tasks/index.rst
similarity index 50%
rename from docs/source/index.rst
rename to docs/liminal/tasks/index.rst
index 1f4f3a3..f85c3df 100644
--- a/docs/source/index.rst
+++ b/docs/liminal/tasks/index.rst
@@ -24,21 +24,47 @@
under the License.
..
-Welcome to Apchae Liminal's documentation!
-==========================================
+Tasks
+=====
-.. toctree::
- :maxdepth: 2
- :caption: Contents:
+``task`` is the definition of a specific step in your pipeline, and is part of the ``tasks`` list
+in your pipeline definition.
+
+For fully detailed information on pipelines see: `pipelines`_.
+
+.. _pipelines: ../pipelines.html
+
+.. code-block:: yaml
+
+ - task: my_python_task
+ type: python
+ image: myorg/myrepo:mypythonapp
+ cmd: python -u my_module.py
+ env_vars:
+ env: {{env}}
+ fizz: buzz
+
+..
- getting_started_with_liminal
- liminal_architecture
- liminal_yaml_guide
- howto_contrib_to_liminal_docs
+A ``task`` is defined by the following attributes:
-Indices and tables
-==================
+In the ``images`` section you can configure how to pack your code into docker images. This can be
+achieved in liminal easily, without any Docker knowledge needed by setting just a few attributes:
+
+task attributes
+''''''''''''''''
+
+``task``: name of your task (must be made of alphanumeric, dash and/or underscore characters only).
+
+``type``: type of the task.
+
+Different task types require their own additional configuration. For example, ``python`` task
+requires ``image`` to be configured.
+
+task types
+''''''''''
+
+.. toctree::
+ :maxdepth: 1
-* :ref:`genindex`
-* :ref:`modindex`
-* :ref:`search`
+ python
diff --git a/docs/liminal/tasks/python.md b/docs/liminal/tasks/python.md
new file mode 100644
index 0000000..34a439f
--- /dev/null
+++ b/docs/liminal/tasks/python.md
@@ -0,0 +1,56 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# python task
+
+The `python` task allows you to run python code packaged as docker images.
+
+```yaml
+ - task: my_python_task
+ type: python
+ image: myorg/myrepo:mypythonapp
+ cmd: python my_python_app.py
+ env_vars:
+ env: '{{env}}'
+ fizz: buzz
+ mounts:
+ - mount: mymount
+ volume: myvol1
+ path: /mnt/vol1
+```
+
+## attributes
+
+`task`: name of your task (must be made of alphanumeric, dash and/or underscore characters only).
+
+`image`: name of image to run.
+
+`cmd`: command to run when running the image.
+
+`env_vars`: environment variables to set when running the image.
+
+`mounts`: list of `mount`s defined by the following attributes:
+
+### mount attributes
+
+`mount`: name of the mount.
+
+`volume`: volume to mount. volumes are defined in the `volumes` section of liminal.yml
+
+`path`: path in which to mount the volume. this is the path accessible to user code.
diff --git a/docs/make.bat b/docs/make.bat
index d5854e9..d6470d6 100644
--- a/docs/make.bat
+++ b/docs/make.bat
@@ -25,7 +25,7 @@ REM Command file for Sphinx documentation
if "%SPHINXBUILD%" == "" (
set SPHINXBUILD=sphinx-build
)
-set SOURCEDIR=source
+set SOURCEDIR=.
set BUILDDIR=build
if "%1" == "" goto help
diff --git a/docs/source/nstatic/airflow_main.png b/docs/nstatic/airflow_main.png
similarity index 100%
rename from docs/source/nstatic/airflow_main.png
rename to docs/nstatic/airflow_main.png
diff --git a/docs/source/nstatic/airflow_task_log.png b/docs/nstatic/airflow_task_log.png
similarity index 100%
rename from docs/source/nstatic/airflow_task_log.png
rename to docs/nstatic/airflow_task_log.png
diff --git a/docs/source/nstatic/airflow_view_dag.png b/docs/nstatic/airflow_view_dag.png
similarity index 100%
rename from docs/source/nstatic/airflow_view_dag.png
rename to docs/nstatic/airflow_view_dag.png
diff --git a/docs/source/nstatic/airflow_view_log.png b/docs/nstatic/airflow_view_log.png
similarity index 100%
rename from docs/source/nstatic/airflow_view_log.png
rename to docs/nstatic/airflow_view_log.png
diff --git a/docs/source/nstatic/liminal_arch_001.png b/docs/nstatic/liminal_arch_001.png
similarity index 100%
rename from docs/source/nstatic/liminal_arch_001.png
rename to docs/nstatic/liminal_arch_001.png
diff --git a/docs/source/nstatic/liminal_arch_002.png b/docs/nstatic/liminal_arch_002.png
similarity index 100%
rename from docs/source/nstatic/liminal_arch_002.png
rename to docs/nstatic/liminal_arch_002.png
diff --git a/docs/source/nstatic/liminal_logo.png b/docs/nstatic/liminal_logo.png
similarity index 100%
rename from docs/source/nstatic/liminal_logo.png
rename to docs/nstatic/liminal_logo.png
diff --git a/docs/source/How_to_build_a_local_liminal.md b/docs/source/How_to_build_a_local_liminal.md
deleted file mode 100644
index 8b5dada..0000000
--- a/docs/source/How_to_build_a_local_liminal.md
+++ /dev/null
@@ -1,70 +0,0 @@
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-# Building a local installation of Liminal
-
-Start a venv in the liminal example folder
-```
-python3 -m venv env
-```
-
-And activate your virtual environment:
-```
-source env/bin/activate
-```
-clean up old versions
-```
-pip uninstall apache-liminal
-```
-make sure you have wheel install (should be OK)
-```
-pip install wheel
-```
-Set a version for the build:
-```
-export LIMINAL_BUILD_VERSION=0.0.1.MYVER
-```
-Build the liminal version in the base of the cloned liminal folder, go to where you cloned it:
-```
-cd <liminal_path>
-git clone https://github.com/apache/incubator-liminal
-python3 setup.py sdist bdist_wheel
-```
-
-make sure the distribution created is copied to the ./scripts folder
-```
-cp ./dist/apache_liminal-0.0.1.MYVER-py3-none-any.whl ./scripts
-```
-
-local install in target directory
-```
-cd example_folder
-pip install <liminal_path>/scripts/apache_liminal-0.0.1.MYVER-py3-none-any.whl
-liminal build
-liminal deploy --clean
-liminal start
-```
-
-How to remove a version and rebuild:
-```
-cd example_folder
-pip uninstall apache-liminal
-cd <liminal_path>
-rm -r ./dist
-```
diff --git a/docs/source/getting_started_with_liminal.md b/docs/source/getting_started_with_liminal.md
deleted file mode 100644
index 1d9a300..0000000
--- a/docs/source/getting_started_with_liminal.md
+++ /dev/null
@@ -1,182 +0,0 @@
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-# Getting Started with Apache Liminal
-
-Liminal is a data systems orchestration platform. Liminal enables data scientists and data engineers to define their data systems and flow using configuration.
-From feature engineering to production monitoring - and the framework takes care of the infra behind the scenes and seamlessly integrates with the infrastructure behind the scenes.
-
-
-## Quick Start
-
-This guide will allow you to set up your first apache Liminal environment and allow you to create some simple ML pipelines. These will be very similar to the ones you are going to build for real production scenarios. This is actually the magic behind Liminal.
-
-## Prerequisites
-
-Python 3 (3.6 and up)
-
-[Docker Desktop](https://www.docker.com/products/docker-desktop)
-
-*Note: Make sure kubernetes cluster is running in docker desktop (or custom kubernetes installation on your machine).*
-
-### Apache Liminal Hello World
-
-In this tutorial, we will go through setting up Liminal for the first time on your local dev machine.
-
-I’m running this on my macBook, so this will cover mac related installation aspects and kinks (if there are any).
-
-First, let’s build our examples project:
-
-In the dev folder, just clone the example code from liminal:
-
-
-```
-git clone https://github.com/apache/incubator-liminal
-```
-***Note:*** *You just cloned the entire Liminal Project, you actually need just the examples folder.*
-
-Create a python virtualenv to isolate your runs, like this:
-
-```
-cd incubator-liminal/examples/liminal-getting-started
-python3 -m venv env
-```
-
-And activate your virtual environment:
-
-```
-source env/bin/activate
-```
-
-Now we are ready to install liminal:
-
-```
-pip install apache-liminal
-rm -rf ~/liminal_home
-```
-Let's build the images you need for the example:
-```
-liminal build
-```
-The build will create docker images based on the Liminal.yml file, each task and service will have its docker image created.
-
-```
-liminal deploy --clean
-```
-The deploy will create missing containers from the images created above as well as basic Apache AirFlow containers needed to run the pipelines, and deploy them to your local docker (docker-compose)
-
-*Note: Liminal creates a bunch of assets, some are going to be located under: the directory in your LIMINAL_HOME env variable. But default LIMINAL_HOME is set to ~/liminal_home directory.*
-
-*Note: the --clean flag is for removing old containers from the previous installs, it will not remove old images.*
-
-Now lets runs it:
-```
-liminal start
-```
-Liminal now spins up the Apache AirFlow containers it will run the dag created based on the Liminal.yml file.
-It includes these three containers:
-* liminal-postgress
-* liminal-webserver
-* liminal-scheduler
-
-Once it finished loading the,
-Go to the Apache AirFlow admin in the browser:
-
-
-```
-http://localhost:8080/admin
-```
-You can just click: [http://localhost:8080/admin](http://localhost:8080/admin)
-
-
-
-
-***Important:** Click on the “On” button to activate the dag, nothing will happen otherwise!*
-
-You can go to tree view to see all the tasks configured in the liminal.yml file: \
-[http://localhost:8080/admin/Apache AirFlow/tree?dag_id=example_pipeline](http://localhost:8080/admin/Apache AirFlow/tree?dag_id=example_pipeline)
-
-Now lets see what actually happened to our task:
-
-
-
-Click on “hello_world_example” and you will get this popup: \
-
- \
-Click on “view log” button and you can see the log of the current task run: \
-
-
-
-
-### mounted volumes
-All Tasks use a mounted volume as defined in the pipeline YAML:
-```YAML
-name: GettingStartedPipeline
-volumes:
- - volume: gettingstartedvol
- local:
- path: ./
-```
-In our case the mounted volume will point to the liminal hello world example.
-The hello world task will read the **hello_world.json** file from the mounted volume and will write the **hello_world_output.json** to it.
-
-*Note:* Each task will internally mount the volume defined above to an internal representation, described under the task section in the yml:
-
-```YAML
- task:
- ...
- mounts:
- - mount: taskmount
- volume: gettingstartedvol
- path: /mnt/vol1
-```
-
-### Here are the entire list of commands, if you want to start from scratch:
-
-```
-git clone https://github.com/apache/incubator-liminal
-cd examples
-python3 -m venv env
-source env/bin/activate
-pip uninstall apache-liminal
-pip install apache-liminal
-rm -rf ~/liminal_home
-Liminal build
-liminal deploy --clean
-liminal start
-```
-
-### Closing up
-
-Making sure the airflow containers are closed, just click:
-
-```
-liminal stop
-```
-
-Or you can do it manually, stop the running containers:
-```
-docker container stop liminal-postgress liminal-webserver liminal-scheduler```
-```
-
-And don't forget to deactivate the python virtualenv, just write:
-
-```
-deactivate
-```