You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@livy.apache.org by js...@apache.org on 2017/07/27 03:12:24 UTC
incubator-livy git commit: [LIVY-376][DOCS] Update README with new
naming and links and move Docs into separate files
Repository: incubator-livy
Updated Branches:
refs/heads/master dcd29a043 -> 60ee047e9
[LIVY-376][DOCS] Update README with new naming and links and move Docs into separate files
[LIVY-376](https://issues.cloudera.org/browse/LIVY-376)
* The README has been updated to match our new Apache naming and links.
* The API Documentation has been split into separate files
* Examples have been moved to the website https://github.com/apache/incubator-livy-website/pull/5
This PR will be followed up with a way to build out the Docs for the website
Author: Alex Bozarth <aj...@us.ibm.com>
Closes #21 from ajbozarth/readme.
Project: http://git-wip-us.apache.org/repos/asf/incubator-livy/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-livy/commit/60ee047e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-livy/tree/60ee047e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-livy/diff/60ee047e
Branch: refs/heads/master
Commit: 60ee047e98b93b50b2763594e9bd17ecc4d31563
Parents: dcd29a0
Author: Alex Bozarth <aj...@us.ibm.com>
Authored: Thu Jul 27 11:12:19 2017 +0800
Committer: jerryshao <ss...@hortonworks.com>
Committed: Thu Jul 27 11:12:19 2017 +0800
----------------------------------------------------------------------
NOTICE.txt | 2 +-
README.md | 82 ++++
README.rst | 982 ------------------------------------------
docs/programmatic-api.md | 89 ++++
docs/rest-api.md | 726 +++++++++++++++++++++++++++++++
5 files changed, 898 insertions(+), 983 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/60ee047e/NOTICE.txt
----------------------------------------------------------------------
diff --git a/NOTICE.txt b/NOTICE.txt
index 73ec690..e61d336 100644
--- a/NOTICE.txt
+++ b/NOTICE.txt
@@ -1,4 +1,4 @@
-Apache livy
+Apache Livy
Copyright 2017 The Apache Software Foundation
This product includes software developed at
http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/60ee047e/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..2dc33c2
--- /dev/null
+++ b/README.md
@@ -0,0 +1,82 @@
+# Apache Livy
+
+[](https://travis-ci.org/apache/incubator-livy)
+
+Apache Livy is an open source REST interface for interacting with
+[Apache Spark](http://spark.apache.org) from anywhere. It supports executing snippets of code or
+programs in a Spark context that runs locally or in
+[Apache Hadoop YARN](http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html).
+
+* Interactive Scala, Python and R shells
+* Batch submissions in Scala, Java, Python
+* Multiple users can share the same server (impersonation support)
+* Can be used for submitting jobs from anywhere with REST
+* Does not require any code change to your programs
+
+[Pull requests](https://github.com/apache/incubator-livy/pulls) are welcomed! But before you begin,
+please check out the [Contributing](http://livy.incubator.apache.org/community#Contributing)
+section on the [Community](http://livy.incubator.apache.org/community) page of our website.
+
+## Online Documentation
+
+Guides and documentation on getting started using Livy, example code snippets, and Livy API
+documentation can be found at [livy.incubator.apache.org](http://livy.incubator.apache.org).
+
+## Before Building Livy
+
+To build Livy, you will need:
+
+Debian/Ubuntu:
+ * mvn (from ``maven`` package or maven3 tarball)
+ * openjdk-7-jdk (or Oracle Java7 jdk)
+ * Python 2.6+
+ * R 3.x
+
+Redhat/CentOS:
+ * mvn (from ``maven`` package or maven3 tarball)
+ * java-1.7.0-openjdk (or Oracle Java7 jdk)
+ * Python 2.6+
+ * R 3.x
+
+MacOS:
+ * Xcode command line tools
+ * Oracle's JDK 1.7+
+ * Maven (Homebrew)
+ * Python 2.6+
+ * R 3.x
+
+Required python packages for building Livy:
+ * cloudpickle
+ * requests
+ * requests-kerberos
+ * flake8
+ * flaky
+ * pytest
+
+
+To run Livy, you will also need a Spark installation. You can get Spark releases at
+https://spark.apache.org/downloads.html.
+
+Livy requires at least Spark 1.6 and supports both Scala 2.10 and 2.11 builds of Spark, Livy
+will automatically pick repl dependencies through detecting the Scala version of Spark.
+
+Livy also supports Spark 2.0+ for both interactive and batch submission, you could seamlessly
+switch to different versions of Spark through ``SPARK_HOME`` configuration, without needing to
+rebuild Livy.
+
+
+## Building Livy
+
+Livy is built using [Apache Maven](http://maven.apache.org). To check out and build Livy, run:
+
+```
+git clone https://github.com/apache/incubator-livy.git
+cd livy
+mvn package
+```
+
+By default Livy is built against Apache Spark 1.6.2, but the version of Spark used when running
+Livy does not need to match the version used to build Livy. Livy internally uses reflection to
+mitigate the gaps between different Spark versions, also Livy package itself does not
+contain a Spark distribution, so it will work with any supported version of Spark (Spark 1.6+)
+without needing to rebuild against specific version of Spark.
http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/60ee047e/README.rst
----------------------------------------------------------------------
diff --git a/README.rst b/README.rst
deleted file mode 100644
index 9f14fa0..0000000
--- a/README.rst
+++ /dev/null
@@ -1,982 +0,0 @@
-Welcome to Livy
-===============
-
-.. image:: https://travis-ci.org/cloudera/livy.svg?branch=master
- :target: https://travis-ci.org/cloudera/livy
-
-Livy is an open source REST interface for interacting with `Apache Spark`_ from anywhere.
-It supports executing snippets of code or programs in a Spark context that runs locally or in `Apache Hadoop YARN`_.
-
-* Interactive Scala, Python and R shells
-* Batch submissions in Scala, Java, Python
-* Multiple users can share the same server (impersonation support)
-* Can be used for submitting jobs from anywhere with REST
-* Does not require any code change to your programs
-
-`Pull requests`_ are welcomed! But before you begin, please check out the `Wiki`_.
-
-.. _Apache Spark: http://spark.apache.org
-.. _Apache Hadoop YARN: http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html
-.. _Pull requests: https://github.com/cloudera/livy/pulls
-.. _Wiki: https://github.com/cloudera/livy/wiki/Contributing-to-Livy
-
-
-Prerequisites
-=============
-
-To build Livy, you will need:
-
-Debian/Ubuntu:
- * mvn (from ``maven`` package or maven3 tarball)
- * openjdk-7-jdk (or Oracle Java7 jdk)
- * Python 2.6+
- * R 3.x
-
-Redhat/CentOS:
- * mvn (from ``maven`` package or maven3 tarball)
- * java-1.7.0-openjdk (or Oracle Java7 jdk)
- * Python 2.6+
- * R 3.x
-
-MacOS:
- * Xcode command line tools
- * Oracle's JDK 1.7+
- * Maven (Homebrew)
- * Python 2.6+
- * R 3.x
-
-Required python packages for building Livy:
- * cloudpickle
- * requests
- * requests-kerberos
- * flake8
- * flaky
- * pytest
-
-
-To run Livy, you will also need a Spark installation. You can get Spark releases at
-https://spark.apache.org/downloads.html.
-
-Livy requires at least Spark 1.6 and supports both Scala 2.10 and 2.11 builds of Spark, Livy
-will automatically pick repl dependencies through detecting the Scala version of Spark.
-
-Livy also supports Spark 2.0+ for both interactive and batch submission, you could seamlessly
-switch to different versions of Spark through ``SPARK_HOME`` configuration, without needing to
-rebuild Livy.
-
-
-Building Livy
-=============
-
-Livy is built using `Apache Maven`_. To check out and build Livy, run:
-
-.. code:: shell
-
- git clone https://github.com/cloudera/livy.git
- cd livy
- mvn package
-
-By default Livy is built against Apache Spark 1.6.2, but the version of Spark used when running
-Livy does not need to match the version used to build Livy. Livy internally uses reflection to
-mitigate the gaps between different Spark versions, also Livy package itself does not
-contain a Spark distribution, so it will work with any supported version of Spark (Spark 1.6+)
-without needing to rebuild against specific version of Spark.
-
-.. _Apache Maven: http://maven.apache.org
-
-
-Running Livy
-============
-
-In order to run Livy with local sessions, first export these variables:
-
-.. code:: shell
-
- export SPARK_HOME=/usr/lib/spark
- export HADOOP_CONF_DIR=/etc/hadoop/conf
-
-Then start the server with:
-
-.. code:: shell
-
- ./bin/livy-server
-
-Livy uses the Spark configuration under ``SPARK_HOME`` by default. You can override the Spark configuration
-by setting the ``SPARK_CONF_DIR`` environment variable before starting Livy.
-
-It is strongly recommended to configure Spark to submit applications in YARN cluster mode.
-That makes sure that user sessions have their resources properly accounted for in the YARN cluster,
-and that the host running the Livy server doesn't become overloaded when multiple user sessions are
-running.
-
-
-Livy Configuration
-==================
-
-Livy uses a few configuration files under configuration the directory, which by default is the
-``conf`` directory under the Livy installation. An alternative configuration directory can be
-provided by setting the ``LIVY_CONF_DIR`` environment variable when starting Livy.
-
-The configuration files used by Livy are:
-
-* ``livy.conf``: contains the server configuration. The Livy distribution ships with a default
- configuration file listing available configuration keys and their default values.
-
-* ``spark-blacklist.conf``: list Spark configuration options that users are not allowed to override.
- These options will be restricted to either their default values, or the values set in the Spark
- configuration used by Livy.
-
-* ``log4j.properties``: configuration for Livy logging. Defines log levels and where log messages
- will be written to. The default configuration will print log messages to stderr.
-
-Upgrade from Livy 0.1
-=====================
-
-A few things changed between since Livy 0.1 that require manual intervention when upgrading.
-
-- Sessions that were active when the Livy 0.1 server was stopped may need to be killed
- manually. Use the tools from your cluster manager to achieve that (for example, the
- ``yarn`` command line tool).
-
-- The configuration file has been renamed from ``livy-defaults.conf`` to ``livy.conf``.
-
-- A few configuration values do not have any effect anymore. Notably:
-
- * ``livy.server.session.factory``: this config option has been replaced by the Spark
- configuration under ``SPARK_HOME``. If you wish to use a different Spark configuration
- for Livy, you can set ``SPARK_CONF_DIR`` in Livy's environment. To define the default
- file system root for sessions, set ``HADOOP_CONF_DIR`` to point at the Hadoop configuration
- to use. The default Hadoop file system will be used.
-
- * ``livy.yarn.jar``: this config has been replaced by separate configs listing specific
- archives for different Livy features. Refer to the default ``livy.conf`` file shipped
- with Livy for instructions.
-
- * ``livy.server.spark-submit``: replaced by the ``SPARK_HOME`` environment variable.
-
-
-Using the Programmatic API
-==========================
-
-Livy provides a programmatic Java/Scala and Python API that allows applications to run code inside
-Spark without having to maintain a local Spark context. Here shows how to use the Java API.
-
-Add the Cloudera repository to your application's POM:
-
-.. code:: xml
-
- <repositories>
- <repository>
- <id>cloudera.repo</id>
- <url>https://repository.cloudera.com/artifactory/cloudera-repos</url>
- <name>Cloudera Repositories</name>
- <snapshots>
- <enabled>false</enabled>
- </snapshots>
- </repository>
- </repositories>
-
-
-And add the Livy client dependency:
-
-.. code:: xml
-
- <dependency>
- <groupId>com.cloudera.livy</groupId>
- <artifactId>livy-client-http</artifactId>
- <version>0.2.0</version>
- </dependency>
-
-
-To be able to compile code that uses Spark APIs, also add the correspondent Spark dependencies.
-
-To run Spark jobs within your applications, extend ``org.apache.livy.Job`` and implement
-the functionality you need. Here's an example job that calculates an approximate value for Pi:
-
-.. code:: java
-
- import java.util.*;
-
- import org.apache.spark.api.java.*;
- import org.apache.spark.api.java.function.*;
-
- import com.cloudera.livy.*;
-
- public class PiJob implements Job<Double>, Function<Integer, Integer>,
- Function2<Integer, Integer, Integer> {
-
- private final int samples;
-
- public PiJob(int samples) {
- this.samples = samples;
- }
-
- @Override
- public Double call(JobContext ctx) throws Exception {
- List<Integer> sampleList = new ArrayList<Integer>();
- for (int i = 0; i < samples; i++) {
- sampleList.add(i + 1);
- }
-
- return 4.0d * ctx.sc().parallelize(sampleList).map(this).reduce(this) / samples;
- }
-
- @Override
- public Integer call(Integer v1) {
- double x = Math.random();
- double y = Math.random();
- return (x*x + y*y < 1) ? 1 : 0;
- }
-
- @Override
- public Integer call(Integer v1, Integer v2) {
- return v1 + v2;
- }
-
- }
-
-
-To submit this code using Livy, create a LivyClient instance and upload your application code to
-the Spark context. Here's an example of code that submits the above job and prints the computed
-value:
-
-.. code:: java
-
- LivyClient client = new LivyClientBuilder()
- .setURI(new URI(livyUrl))
- .build();
-
- try {
- System.err.printf("Uploading %s to the Spark context...\n", piJar);
- client.uploadJar(new File(piJar)).get();
-
- System.err.printf("Running PiJob with %d samples...\n", samples);
- double pi = client.submit(new PiJob(samples)).get();
-
- System.out.println("Pi is roughly: " + pi);
- } finally {
- client.stop(true);
- }
-
-
-To learn about all the functionality available to applications, read the javadoc documentation for
-the classes under the ``api`` module.
-
-
-Spark Example
-=============
-
-Here's a step-by-step example of interacting with Livy in Python with the `Requests`_ library. By
-default Livy runs on port 8998 (which can be changed with the ``livy.server.port`` config option).
-We’ll start off with a Spark session that takes Scala code:
-
-.. code:: shell
-
- sudo pip install requests
-
-.. code:: python
-
- import json, pprint, requests, textwrap
- host = 'http://localhost:8998'
- data = {'kind': 'spark'}
- headers = {'Content-Type': 'application/json'}
- r = requests.post(host + '/sessions', data=json.dumps(data), headers=headers)
- r.json()
-
- {u'state': u'starting', u'id': 0, u'kind': u'spark'}
-
-Once the session has completed starting up, it transitions to the idle state:
-
-.. code:: python
-
- session_url = host + r.headers['location']
- r = requests.get(session_url, headers=headers)
- r.json()
-
- {u'state': u'idle', u'id': 0, u'kind': u'spark'}
-
-Now we can execute Scala by passing in a simple JSON command:
-
-.. code:: python
-
- statements_url = session_url + '/statements'
- data = {'code': '1 + 1'}
- r = requests.post(statements_url, data=json.dumps(data), headers=headers)
- r.json()
-
- {u'output': None, u'state': u'running', u'id': 0}
-
-If a statement takes longer than a few milliseconds to execute, Livy returns
-early and provides a statement URL that can be polled until it is complete:
-
-.. code:: python
-
- statement_url = host + r.headers['location']
- r = requests.get(statement_url, headers=headers)
- pprint.pprint(r.json())
-
- {u'id': 0,
- u'output': {u'data': {u'text/plain': u'res0: Int = 2'},
- u'execution_count': 0,
- u'status': u'ok'},
- u'state': u'available'}
-
-That was a pretty simple example. More interesting is using Spark to estimate
-Pi. This is from the `Spark Examples`_:
-
-.. code:: python
-
- data = {
- 'code': textwrap.dedent("""
- val NUM_SAMPLES = 100000;
- val count = sc.parallelize(1 to NUM_SAMPLES).map { i =>
- val x = Math.random();
- val y = Math.random();
- if (x*x + y*y < 1) 1 else 0
- }.reduce(_ + _);
- println(\"Pi is roughly \" + 4.0 * count / NUM_SAMPLES)
- """)
- }
-
- r = requests.post(statements_url, data=json.dumps(data), headers=headers)
- pprint.pprint(r.json())
-
- statement_url = host + r.headers['location']
- r = requests.get(statement_url, headers=headers)
- pprint.pprint(r.json())
-
- {u'id': 1,
- u'output': {u'data': {u'text/plain': u'Pi is roughly 3.14004\nNUM_SAMPLES: Int = 100000\ncount: Int = 78501'},
- u'execution_count': 1,
- u'status': u'ok'},
- u'state': u'available'}
-
-Finally, close the session:
-
-.. code:: python
-
- session_url = 'http://localhost:8998/sessions/0'
- requests.delete(session_url, headers=headers)
-
- <Response [204]>
-
-.. _Requests: http://docs.python-requests.org/en/latest/
-.. _Spark Examples: https://spark.apache.org/examples.html
-
-
-PySpark Example
-===============
-
-PySpark has the same API, just with a different initial request:
-
-.. code:: python
-
- data = {'kind': 'pyspark'}
- r = requests.post(host + '/sessions', data=json.dumps(data), headers=headers)
- r.json()
-
- {u'id': 1, u'state': u'idle'}
-
-The Pi example from before then can be run as:
-
-.. code:: python
-
- data = {
- 'code': textwrap.dedent("""
- import random
- NUM_SAMPLES = 100000
- def sample(p):
- x, y = random.random(), random.random()
- return 1 if x*x + y*y < 1 else 0
-
- count = sc.parallelize(xrange(0, NUM_SAMPLES)).map(sample).reduce(lambda a, b: a + b)
- print "Pi is roughly %f" % (4.0 * count / NUM_SAMPLES)
- """)
- }
-
- r = requests.post(statements_url, data=json.dumps(data), headers=headers)
- pprint.pprint(r.json())
-
- {u'id': 12,
- u'output': {u'data': {u'text/plain': u'Pi is roughly 3.136000'},
- u'execution_count': 12,
- u'status': u'ok'},
- u'state': u'running'}
-
-
-SparkR Example
-==============
-
-SparkR has the same API:
-
-.. code:: python
-
- data = {'kind': 'sparkr'}
- r = requests.post(host + '/sessions', data=json.dumps(data), headers=headers)
- r.json()
-
- {u'id': 1, u'state': u'idle'}
-
-The Pi example from before then can be run as:
-
-.. code:: python
-
- data = {
- 'code': textwrap.dedent("""
- n <- 100000
- piFunc <- function(elem) {
- rands <- runif(n = 2, min = -1, max = 1)
- val <- ifelse((rands[1]^2 + rands[2]^2) < 1, 1.0, 0.0)
- val
- }
- piFuncVec <- function(elems) {
- message(length(elems))
- rands1 <- runif(n = length(elems), min = -1, max = 1)
- rands2 <- runif(n = length(elems), min = -1, max = 1)
- val <- ifelse((rands1^2 + rands2^2) < 1, 1.0, 0.0)
- sum(val)
- }
- rdd <- parallelize(sc, 1:n, slices)
- count <- reduce(lapplyPartition(rdd, piFuncVec), sum)
- cat("Pi is roughly", 4.0 * count / n, "\n")
- """)
- }
-
- r = requests.post(statements_url, data=json.dumps(data), headers=headers)
- pprint.pprint(r.json())
-
- {u'id': 12,
- u'output': {u'data': {u'text/plain': u'Pi is roughly 3.136000'},
- u'execution_count': 12,
- u'status': u'ok'},
- u'state': u'running'}
-
-
-Community
-=========
-
- * User group: http://groups.google.com/a/cloudera.org/group/livy-user
- * Dev group: http://groups.google.com/a/cloudera.org/group/livy-dev
- * Dev slack: https://livy-dev.slack.com.
-
- To join: http://livy-slack-invite.azurewebsites.net. Invite token: ``I'm not a bot``.
- * JIRA: https://issues.cloudera.org/browse/LIVY
- * Pull requests: https://github.com/cloudera/livy/pulls
-
-
-REST API
-========
-
-GET /sessions
--------------
-
-Returns all the active interactive sessions.
-
-Request Parameters
-^^^^^^^^^^^^^^^^^^
-
-+------+-----------------------------------+------+
-| name | description | type |
-+======+===================================+======+
-| from | The start index to fetch sessions | int |
-+------+-----------------------------------+------+
-| size | Number of sessions to fetch | int |
-+------+-----------------------------------+------+
-
-Response Body
-^^^^^^^^^^^^^
-
-+----------+-------------------------------------+------+
-| name | description | type |
-+==========+=====================================+======+
-| from | The start index of fetched sessions | int |
-+----------+-------------------------------------+------+
-| total | Number of sessions fetched | int |
-+----------+-------------------------------------+------+
-| sessions | `Session`_ list | list |
-+----------+-------------------------------------+------+
-
-
-POST /sessions
---------------
-
-Creates a new interactive Scala, Python, or R shell in the cluster.
-
-Request Body
-^^^^^^^^^^^^
-
-+--------------------------+------------------------------------------------+-----------------+
-| name | description | type |
-+==========================+================================================+=================+
-| kind | The session kind (required) | `session kind`_ |
-+--------------------------+------------------------------------------------+-----------------+
-| proxyUser | User to impersonate when starting the session | string |
-+--------------------------+------------------------------------------------+-----------------+
-| jars | jars to be used in this session | List of string |
-+--------------------------+------------------------------------------------+-----------------+
-| pyFiles | Python files to be used in this session | List of string |
-+--------------------------+------------------------------------------------+-----------------+
-| files | files to be used in this session | List of string |
-+--------------------------+------------------------------------------------+-----------------+
-| driverMemory | Amount of memory to use for the driver process | string |
-+--------------------------+------------------------------------------------+-----------------+
-| driverCores | Number of cores to use for the driver process | int |
-+--------------------------+------------------------------------------------+-----------------+
-| executorMemory | Amount of memory to use per executor process | string |
-+--------------------------+------------------------------------------------+-----------------+
-| executorCores | Number of cores to use for each executor | int |
-+--------------------------+------------------------------------------------+-----------------+
-| numExecutors | Number of executors to launch for this session | int |
-+--------------------------+------------------------------------------------+-----------------+
-| archives | Archives to be used in this session | List of string |
-+--------------------------+------------------------------------------------+-----------------+
-| queue | The name of the YARN queue to which submitted | string |
-+--------------------------+------------------------------------------------+-----------------+
-| name | The name of this session | string |
-+--------------------------+------------------------------------------------+-----------------+
-| conf | Spark configuration properties | Map of key=val |
-+--------------------------+------------------------------------------------+-----------------+
-| heartbeatTimeoutInSecond | Timeout in second to which session be orphaned | int |
-+--------------------------+------------------------------------------------+-----------------+
-
-
-Response Body
-^^^^^^^^^^^^^
-
-The created `Session`_.
-
-
-GET /sessions/{sessionId}
--------------------------
-
-Returns the session information.
-
-Response Body
-^^^^^^^^^^^^^
-
-The `Session`_.
-
-
-GET /sessions/{sessionId}/state
--------------------------------
-
-Returns the state of session
-
-Response
-^^^^^^^^
-
-+-------+-----------------------------------+--------+
-| name | description | type |
-+=======+===================================+========+
-| id | Session id | int |
-+-------+-----------------------------------+--------+
-| state | The current state of session | string |
-+-------+-----------------------------------+--------+
-
-
-DELETE /sessions/{sessionId}
-----------------------------
-
-Kills the `Session`_ job.
-
-
-GET /sessions/{sessionId}/log
-------------------------------
-
-Gets the log lines from this session.
-
-Request Parameters
-^^^^^^^^^^^^^^^^^^
-
-+------+-----------------------------------+------+
-| name | description | type |
-+======+===================================+======+
-| from | Offset | int |
-+------+-----------------------------------+------+
-| size | Max number of log lines to return | int |
-+------+-----------------------------------+------+
-
-Response Body
-^^^^^^^^^^^^^
-
-+------+--------------------------+-----------------+
-| name | description | type |
-+======+==========================+=================+
-| id | The session id | int |
-+------+--------------------------+-----------------+
-| from | Offset from start of log | int |
-+------+--------------------------+-----------------+
-| size | Number of log lines | int |
-+------+--------------------------+-----------------+
-| log | The log lines | list of strings |
-+------+--------------------------+-----------------+
-
-
-GET /sessions/{sessionId}/statements
-------------------------------------
-
-Returns all the statements in a session.
-
-Response Body
-^^^^^^^^^^^^^
-
-+------------+-------------------+------+
-| name | description | type |
-+============+===================+======+
-| statements | `statement`_ list | list |
-+------------+-------------------+------+
-
-
-POST /sessions/{sessionId}/statements
--------------------------------------
-
-Runs a statement in a session.
-
-Request Body
-^^^^^^^^^^^^
-
-+------+---------------------+--------+
-| name | description | type |
-+======+=====================+========+
-| code | The code to execute | string |
-+------+---------------------+--------+
-
-Response Body
-^^^^^^^^^^^^^
-
-The `statement`_ object.
-
-
-GET /sessions/{sessionId}/statements/{statementId}
---------------------------------------------------
-
-Returns a specified statement in a session.
-
-Response Body
-^^^^^^^^^^^^^
-
-The `statement`_ object.
-
-
-POST /sessions/{sessionId}/statements/{statementId}/cancel
-----------------------------------------------------------
-
-Cancel the specified statement in this session.
-
-Response Body
-^^^^^^^^^^^^^
-
-+------+----------------------------+--------+
-| name | description | type |
-+======+============================+========+
-| msg | is always "cancelled" | string |
-+------+----------------------------+--------+
-
-
-GET /batches
--------------
-
-Returns all the active batch sessions.
-
-Request Parameters
-^^^^^^^^^^^^^^^^^^
-
-+------+-----------------------------------+------+
-| name | description | type |
-+======+===================================+======+
-| from | The start index to fetch sessions | int |
-+------+-----------------------------------+------+
-| size | Number of sessions to fetch | int |
-+------+-----------------------------------+------+
-
-Response Body
-^^^^^^^^^^^^^
-
-+----------+-------------------------------------+------+
-| name | description | type |
-+==========+=====================================+======+
-| from | The start index of fetched sessions | int |
-+----------+-------------------------------------+------+
-| total | Number of sessions fetched | int |
-+----------+-------------------------------------+------+
-| sessions | `Batch`_ list | list |
-+----------+-------------------------------------+------+
-
-
-POST /batches
--------------
-
-Request Body
-^^^^^^^^^^^^
-
-+----------------+---------------------------------------------------+-----------------+
-| name | description | type |
-+================+===================================================+=================+
-| file | File containing the application to execute | path (required) |
-+----------------+---------------------------------------------------+-----------------+
-| proxyUser | User to impersonate when running the job | string |
-+----------------+---------------------------------------------------+-----------------+
-| className | Application Java/Spark main class | string |
-+----------------+---------------------------------------------------+-----------------+
-| args | Command line arguments for the application | list of strings |
-+----------------+---------------------------------------------------+-----------------+
-| jars | jars to be used in this session | List of string |
-+----------------+---------------------------------------------------+-----------------+
-| pyFiles | Python files to be used in this session | List of string |
-+----------------+---------------------------------------------------+-----------------+
-| files | files to be used in this session | List of string |
-+----------------+---------------------------------------------------+-----------------+
-| driverMemory | Amount of memory to use for the driver process | string |
-+----------------+---------------------------------------------------+-----------------+
-| driverCores | Number of cores to use for the driver process | int |
-+----------------+---------------------------------------------------+-----------------+
-| executorMemory | Amount of memory to use per executor process | string |
-+----------------+---------------------------------------------------+-----------------+
-| executorCores | Number of cores to use for each executor | int |
-+----------------+---------------------------------------------------+-----------------+
-| numExecutors | Number of executors to launch for this session | int |
-+----------------+---------------------------------------------------+-----------------+
-| archives | Archives to be used in this session | List of string |
-+----------------+---------------------------------------------------+-----------------+
-| queue | The name of the YARN queue to which submitted | string |
-+----------------+---------------------------------------------------+-----------------+
-| name | The name of this session | string |
-+----------------+---------------------------------------------------+-----------------+
-| conf | Spark configuration properties | Map of key=val |
-+----------------+---------------------------------------------------+-----------------+
-
-Response Body
-^^^^^^^^^^^^^
-
-The created `Batch`_ object.
-
-
-GET /batches/{batchId}
-----------------------
-
-Returns the batch session information.
-
-Response Body
-^^^^^^^^^^^^^
-
-The `Batch`_.
-
-
-GET /batches/{batchId}/state
-----------------------------
-
-Returns the state of batch session
-
-Response
-^^^^^^^^
-
-+-------+-----------------------------------+--------+
-| name | description | type |
-+=======+===================================+========+
-| id | Batch session id | int |
-+-------+-----------------------------------+--------+
-| state | The current state of batch session| string |
-+-------+-----------------------------------+--------+
-
-
-DELETE /batches/{batchId}
--------------------------
-
-Kills the `Batch`_ job.
-
-
-GET /batches/{batchId}/log
---------------------------
-
-Gets the log lines from this batch.
-
-Request Parameters
-^^^^^^^^^^^^^^^^^^
-
-+------+-----------------------------------+------+
-| name | description | type |
-+======+===================================+======+
-| from | Offset | int |
-+------+-----------------------------------+------+
-| size | Max number of log lines to return | int |
-+------+-----------------------------------+------+
-
-Response Body
-^^^^^^^^^^^^^
-
-+------+--------------------------+-----------------+
-| name | description | type |
-+======+==========================+=================+
-| id | The batch id | int |
-+------+--------------------------+-----------------+
-| from | Offset from start of log | int |
-+------+--------------------------+-----------------+
-| size | Number of log lines | int |
-+------+--------------------------+-----------------+
-| log | The log lines | list of strings |
-+------+--------------------------+-----------------+
-
-
-REST Objects
-============
-
-Session
--------
-
-A session represents an interactive shell.
-
-+----------------+------------------------------------------+----------------------------+
-| name | description | type |
-+================+==========================================+============================+
-| id | The session id | int |
-+----------------+------------------------------------------+----------------------------+
-| appId | The application id of this session | String |
-+----------------+------------------------------------------+----------------------------+
-| owner | Remote user who submitted this session | String |
-+----------------+------------------------------------------+----------------------------+
-| proxyUser | User to impersonate when running | String |
-+----------------+------------------------------------------+----------------------------+
-| kind | Session kind (spark, pyspark, or sparkr) | `session kind`_ |
-+----------------+------------------------------------------+----------------------------+
-| log | The log lines | list of strings |
-+----------------+------------------------------------------+----------------------------+
-| state | The session state | string |
-+----------------+------------------------------------------+----------------------------+
-| appInfo | The detailed application info | Map of key=val |
-+----------------+------------------------------------------+----------------------------+
-
-
-Session State
-^^^^^^^^^^^^^
-
-+---------------+----------------------------------+
-| value | description |
-+===============+==================================+
-| not_started | Session has not been started |
-+---------------+----------------------------------+
-| starting | Session is starting |
-+---------------+----------------------------------+
-| idle | Session is waiting for input |
-+---------------+----------------------------------+
-| busy | Session is executing a statement |
-+---------------+----------------------------------+
-| shutting_down | Session is shutting down |
-+---------------+----------------------------------+
-| error | Session errored out |
-+---------------+----------------------------------+
-| dead | Session has exited |
-+---------------+----------------------------------+
-| success | Session is successfully stopped |
-+---------------+----------------------------------+
-
-Session Kind
-^^^^^^^^^^^^
-
-+-------------+------------------------------------+
-| value | description |
-+=============+====================================+
-| spark | Interactive Scala Spark session |
-+-------------+------------------------------------+
-| `pyspark`_ | Interactive Python 2 Spark session |
-+-------------+------------------------------------+
-| `pyspark3`_ | Interactive Python 3 Spark session |
-+-------------+------------------------------------+
-| sparkr | Interactive R Spark session |
-+-------------+------------------------------------+
-
-pyspark
-^^^^^^^
-To change the Python executable the session uses, Livy reads the path from environment variable
-``PYSPARK_PYTHON`` (Same as pyspark).
-
-Like pyspark, if Livy is running in ``local`` mode, just set the environment variable.
-If the session is running in ``yarn-cluster`` mode, please set
-``spark.yarn.appMasterEnv.PYSPARK_PYTHON`` in SparkConf so the environment variable is passed to
-the driver.
-
-pyspark3
-^^^^^^^^
-To change the Python executable the session uses, Livy reads the path from environment variable
-``PYSPARK3_PYTHON``.
-
-Like pyspark, if Livy is running in ``local`` mode, just set the environment variable.
-If the session is running in ``yarn-cluster`` mode, please set
-``spark.yarn.appMasterEnv.PYSPARK3_PYTHON`` in SparkConf so the environment variable is passed to
-the driver.
-
-Statement
----------
-
-A statement represents the result of an execution statement.
-
-+--------+----------------------+---------------------+
-| name | description | type |
-+========+======================+=====================+
-| id | The statement id | integer |
-+--------+----------------------+---------------------+
-| code | The execution code | string |
-+--------+----------------------+---------------------+
-| state | The execution state | statement state |
-+--------+----------------------+---------------------+
-| output | The execution output | statement output |
-+--------+----------------------+---------------------+
-
-Statement State
-^^^^^^^^^^^^^^^
-
-+------------+----------------------------------------------------+
-| value | description |
-+============+====================================================+
-| waiting | Statement is enqueued but execution hasn't started |
-+------------+----------------------------------------------------+
-| running | Statement is currently running |
-+------------+----------------------------------------------------+
-| available | Statement has a response ready |
-+------------+----------------------------------------------------+
-| error | Statement failed |
-+------------+----------------------------------------------------+
-| cancelling | Statement is being cancelling |
-+------------+----------------------------------------------------+
-| cancelled | Statement is cancelled |
-+------------+----------------------------------------------------+
-
-Statement Output
-^^^^^^^^^^^^^^^^
-
-+-----------------+-------------------+----------------------------------+
-| name | description | type |
-+=================+===================+==================================+
-| status | Execution status | string |
-+-----------------+-------------------+----------------------------------+
-| execution_count | A monotonically | integer |
-| | increasing number | |
-+-----------------+-------------------+----------------------------------+
-| data | Statement output | An object mapping a mime type to |
-| | | the result. If the mime type is |
-| | | ``application/json``, the value |
-| | | is a JSON value. |
-+-----------------+-------------------+----------------------------------+
-
-Batch
------
-
-+----------------+-------------------------------------+-----------------+
-| name | description | type |
-+================+=====================================+=================+
-| id | The session id | int |
-+----------------+-------------------------------------+-----------------+
-| appId | The application id of this session | String |
-+----------------+-------------------------------------+-----------------+
-| appInfo | The detailed application info | Map of key=val |
-+----------------+-------------------------------------+-----------------+
-| log | The log lines | list of strings |
-+----------------+-------------------------------------+-----------------+
-| state | The batch state | string |
-+----------------+-------------------------------------+-----------------+
-
-
-License
-=======
-
-Apache License, Version 2.0
-http://www.apache.org/licenses/LICENSE-2.0
http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/60ee047e/docs/programmatic-api.md
----------------------------------------------------------------------
diff --git a/docs/programmatic-api.md b/docs/programmatic-api.md
new file mode 100644
index 0000000..faec319
--- /dev/null
+++ b/docs/programmatic-api.md
@@ -0,0 +1,89 @@
+## Using the Programmatic API
+
+Livy provides a programmatic Java/Scala and Python API that allows applications to run code inside
+Spark without having to maintain a local Spark context. Here shows how to use the Java API.
+
+Add the Livy client dependency to your application's POM:
+
+```xml
+<dependency>
+ <groupId>org.apache.livy</groupId>
+ <artifactId>livy-client-http</artifactId>
+ <version>0.4.0-SNAPSHOT</version>
+</dependency>
+```
+
+Note: Until Livy's first Apache release you will have to
+install the livy artifacts locally using `mvn install`.
+
+To be able to compile code that uses Spark APIs, also add the correspondent Spark dependencies.
+
+To run Spark jobs within your applications, extend ``org.apache.livy.Job`` and implement
+the functionality you need. Here's an example job that calculates an approximate value for Pi:
+
+```java
+import java.util.*;
+
+import org.apache.spark.api.java.*;
+import org.apache.spark.api.java.function.*;
+
+import org.apache.livy.*;
+
+public class PiJob implements Job<Double>, Function<Integer, Integer>,
+ Function2<Integer, Integer, Integer> {
+
+ private final int samples;
+
+ public PiJob(int samples) {
+ this.samples = samples;
+ }
+
+ @Override
+ public Double call(JobContext ctx) throws Exception {
+ List<Integer> sampleList = new ArrayList<Integer>();
+ for (int i = 0; i < samples; i++) {
+ sampleList.add(i + 1);
+ }
+
+ return 4.0d * ctx.sc().parallelize(sampleList).map(this).reduce(this) / samples;
+ }
+
+ @Override
+ public Integer call(Integer v1) {
+ double x = Math.random();
+ double y = Math.random();
+ return (x*x + y*y < 1) ? 1 : 0;
+ }
+
+ @Override
+ public Integer call(Integer v1, Integer v2) {
+ return v1 + v2;
+ }
+
+}
+```
+
+To submit this code using Livy, create a LivyClient instance and upload your application code to
+the Spark context. Here's an example of code that submits the above job and prints the computed
+value:
+
+```java
+LivyClient client = new LivyClientBuilder()
+ .setURI(new URI(livyUrl))
+ .build();
+
+try {
+ System.err.printf("Uploading %s to the Spark context...\n", piJar);
+ client.uploadJar(new File(piJar)).get();
+
+ System.err.printf("Running PiJob with %d samples...\n", samples);
+ double pi = client.submit(new PiJob(samples)).get();
+
+ System.out.println("Pi is roughly: " + pi);
+} finally {
+ client.stop(true);
+}
+```
+
+To learn about all the functionality available to applications, read the javadoc documentation for
+the classes under the ``api`` module.
http://git-wip-us.apache.org/repos/asf/incubator-livy/blob/60ee047e/docs/rest-api.md
----------------------------------------------------------------------
diff --git a/docs/rest-api.md b/docs/rest-api.md
new file mode 100644
index 0000000..c7d2fcc
--- /dev/null
+++ b/docs/rest-api.md
@@ -0,0 +1,726 @@
+## REST API
+
+### GET /sessions
+
+Returns all the active interactive sessions.
+
+
+#### Request Parameters
+
+<table class="table">
+ <tr><th>Name</th><th>Description</th><th>Type</th></tr>
+ <tr>
+ <td>from</td>
+ <td>The start index to fetch sessions</td>
+ <td>int</td>
+ </tr>
+ <tr>
+ <td>size</td>
+ <td>Number of sessions to fetch</td>
+ <td>int</td>
+ </tr>
+</table>
+
+#### Response Body
+
+<table class="table">
+ <tr><th>Name</th><th>Description</th><th>Type</th></tr>
+ <tr>
+ <td>from</td>
+ <td>The start index to fetch sessions</td>
+ <td>int</td>
+ </tr>
+ <tr>
+ <td>size</td>
+ <td>Number of sessions to fetch</td>
+ <td>int</td>
+ </tr>
+ <tr>
+ <td>sessions</td>
+ <td><a href="#session">Session</a> list</td>
+ <td>list</td>
+ </tr>
+</table>
+
+### POST /sessions
+
+Creates a new interactive Scala, Python, or R shell in the cluster.
+
+#### Request Body
+
+<table class="table">
+ <tr><th>Name</th><th>Description</th><th>Type</th></tr>
+ <tr>
+ <td>kind</td>
+ <td>The session kind (required)</td>
+ <td><a href="#session-kind">session kind</a></td>
+ </tr>
+ <tr>
+ <td>proxyUser</td>
+ <td>User to impersonate when starting the session</td>
+ <td>string</td>
+ </tr>
+ <tr>
+ <td>jars</td>
+ <td>jars to be used in this session</td>
+ <td>List of string</td>
+ </tr>
+ <tr>
+ <td>pyFiles</td>
+ <td>Python files to be used in this session</td>
+ <td>List of string</td>
+ </tr>
+ <tr>
+ <td>files</td>
+ <td>files to be used in this session</td>
+ <td>List of string</td>
+ </tr>
+ <tr>
+ <td>driverMemory</td>
+ <td>Amount of memory to use for the driver process</td>
+ <td>string</td>
+ </tr>
+ <tr>
+ <td>driverCores</td>
+ <td>Number of cores to use for the driver process</td>
+ <td>int</td>
+ </tr>
+ <tr>
+ <td>executorMemory</td>
+ <td>Amount of memory to use per executor process</td>
+ <td>string</td>
+ </tr>
+ <tr>
+ <td>executorCores</td>
+ <td>Number of cores to use for each executor</td>
+ <td>int</td>
+ </tr>
+ <tr>
+ <td>numExecutors</td>
+ <td>Number of executors to launch for this session</td>
+ <td>int</td>
+ </tr>
+ <tr>
+ <td>archives</td>
+ <td>Archives to be used in this session</td>
+ <td>List of string</td>
+ </tr>
+ <tr>
+ <td>queue</td>
+ <td>The name of the YARN queue to which submitted</td>
+ <td>string</td>
+ </tr>
+ <tr>
+ <td>name</td>
+ <td>The name of this session</td>
+ <td>string</td>
+ </tr>
+ <tr>
+ <td>conf</td>
+ <td>Spark configuration properties</td>
+ <td>Map of key=val</td>
+ </tr>
+ <tr>
+ <td>heartbeatTimeoutInSecond</td>
+ <td>Timeout in second to which session be orphaned</td>
+ <td>int</td>
+ </tr>
+</table>
+
+#### Response Body
+
+The created <a href="#session">Session</a>.
+
+
+### GET /sessions/{sessionId}
+
+Returns the session information.
+
+#### Response Body
+
+The <a href="#session">Session</a>.
+
+
+### GET /sessions/{sessionId}/state
+
+Returns the state of session
+
+#### Response
+
+<table class="table">
+ <tr><th>Name</th><th>Description</th><th>Type</th></tr>
+ <tr>
+ <td>id</td>
+ <td>Session id</td>
+ <td>int</td>
+ </tr>
+ <tr>
+ <td>state</td>
+ <td>The current state of session</td>
+ <td>string</td>
+ </tr>
+</table>
+
+### DELETE /sessions/{sessionId}
+
+Kills the <a href="#session">Session</a> job.
+
+
+### GET /sessions/{sessionId}/log
+
+Gets the log lines from this session.
+
+#### Request Parameters
+
+<table class="table">
+ <tr><th>Name</th><th>Description</th><th>Type</th></tr>
+ <tr>
+ <td>from</td>
+ <td>Offset</td>
+ <td>int</td>
+ </tr>
+ <tr>
+ <td>size</td>
+ <td>Max number of log lines to return</td>
+ <td>int</td>
+ </tr>
+</table>
+
+#### Response Body
+
+<table class="table">
+ <tr><th>Name</th><th>Description</th><th>Type</th></tr>
+ <tr>
+ <td>id</td>
+ <td>The session id</td>
+ <td>int</td>
+ </tr>
+ <tr>
+ <td>from</td>
+ <td>Offset from start of log</td>
+ <td>int</td>
+ </tr>
+ <tr>
+ <td>size</td>
+ <td>Max number of log lines</td>
+ <td>int</td>
+ </tr>
+ <tr>
+ <td>log</td>
+ <td>The log lines</td>
+ <td>list of strings</td>
+ </tr>
+</table>
+
+### GET /sessions/{sessionId}/statements
+
+Returns all the statements in a session.
+
+#### Response Body
+
+<table class="table">
+ <tr><th>Name</th><th>Description</th><th>Type</th></tr>
+ <tr>
+ <td>statements</td>
+ <td><a href="#statement">statement</a> list</td>
+ <td>list</td>
+ </tr>
+</table>
+
+### POST /sessions/{sessionId}/statements
+
+Runs a statement in a session.
+
+#### Request Body
+
+<table class="table">
+ <tr><th>Name</th><th>Description</th><th>Type</th></tr>
+ <tr>
+ <td>code</td>
+ <td>The code to execute</td>
+ <td>string</td>
+ </tr>
+</table>
+
+#### Response Body
+
+The <a href="#statement">statement</a> object.
+
+
+### GET /sessions/{sessionId}/statements/{statementId}
+
+Returns a specified statement in a session.
+
+#### Response Body
+
+The <a href="#statement">statement</a> object.
+
+
+### POST /sessions/{sessionId}/statements/{statementId}/cancel
+
+Cancel the specified statement in this session.
+
+#### Response Body
+
+<table class="table">
+ <tr><th>Name</th><th>Description</th><th>Type</th></tr>
+ <tr>
+ <td>msg</td>
+ <td>is always "cancelled"</td>
+ <td>string</td>
+ </tr>
+</table>
+
+### GET /batches
+
+Returns all the active batch sessions.
+
+#### Request Parameters
+
+<table class="table">
+ <tr><th>Name</th><th>Description</th><th>Type</th></tr>
+ <tr>
+ <td>from</td>
+ <td>The start index to fetch sessions</td>
+ <td>int</td>
+ </tr>
+ <tr>
+ <td>size</td>
+ <td>Number of sessions to fetch</td>
+ <td>int</td>
+ </tr>
+</table>
+
+#### Response Body
+
+<table class="table">
+ <tr><th>Name</th><th>Description</th><th>Type</th></tr>
+ <tr>
+ <td>from</td>
+ <td>The start index of fetched sessions</td>
+ <td>int</td>
+ </tr>
+ <tr>
+ <td>total</td>
+ <td>Number of sessions fetched</td>
+ <td>int</td>
+ </tr>
+ <tr>
+ <td>sessions</td>
+ <td><a href="#batch">Batch</a> list</td>
+ <td>list</td>
+ </tr>
+</table>
+
+### POST /batches
+
+#### Request Body
+
+<table class="table">
+ <tr><th>Name</th><th>Description</th><th>Type</th></tr>
+ <tr>
+ <td>file</td>
+ <td>File containing the application to execute</td>
+ <td>path (required)</td>
+ </tr>
+ <tr>
+ <td>proxyUser</td>
+ <td>User to impersonate when running the job</td>
+ <td>string</td>
+ </tr>
+ <tr>
+ <td>className</td>
+ <td>Application Java/Spark main class</td>
+ <td>string</td>
+ </tr>
+ <tr>
+ <td>args</td>
+ <td>Command line arguments for the application</td>
+ <td>list of strings</td>
+ </tr>
+ <tr>
+ <td>jars</td>
+ <td>jars to be used in this session</td>
+ <td>list of strings</td>
+ </tr>
+ <tr>
+ <td>pyFiles</td>
+ <td>Python files to be used in this session</td>
+ <td>list of strings</td>
+ </tr>
+ <tr>
+ <td>files</td>
+ <td>files to be used in this session</td>
+ <td>list of strings</td>
+ </tr>
+ <tr>
+ <td>driverMemory</td>
+ <td>Amount of memory to use for the driver process</td>
+ <td>string</td>
+ </tr>
+ <tr>
+ <td>driverCores</td>
+ <td>Number of cores to use for the driver process</td>
+ <td>int</td>
+ </tr>
+ <tr>
+ <td>executorMemory</td>
+ <td>Amount of memory to use per executor process</td>
+ <td>string</td>
+ </tr>
+ <tr>
+ <td>executorCores</td>
+ <td>Number of cores to use for each executor</td>
+ <td>int</td>
+ </tr>
+ <tr>
+ <td>numExecutors</td>
+ <td>Number of executors to launch for this session</td>
+ <td>int</td>
+ </tr>
+ <tr>
+ <td>archives</td>
+ <td>Archives to be used in this session</td>
+ <td>List of string</td>
+ </tr>
+ <tr>
+ <td>queue</td>
+ <td>The name of the YARN queue to which submitted</td>
+ <td>string</td>
+ </tr>
+ <tr>
+ <td>name</td>
+ <td>The name of this session</td>
+ <td>string</td>
+ </tr>
+ <tr>
+ <td>conf</td>
+ <td>Spark configuration properties</td>
+ <td>Map of key=val</td>
+ </tr>
+</table>
+
+#### Response Body
+
+The created <a href="#batch">Batch</a> object.
+
+
+### GET /batches/{batchId}
+
+Returns the batch session information.
+
+#### Response Body
+
+The <a href="#batch">Batch</a>.
+
+
+### GET /batches/{batchId}/state
+
+Returns the state of batch session
+
+#### Response
+
+<table class="table">
+ <tr><th>Name</th><th>Description</th><th>Type</th></tr>
+ <tr>
+ <td>id</td>
+ <td>Batch session id</td>
+ <td>int</td>
+ </tr>
+ <tr>
+ <td>state</td>
+ <td>The current state of batch session</td>
+ <td>string</td>
+ </tr>
+</table>
+
+### DELETE /batches/{batchId}
+
+Kills the <a href="#batch">Batch</a> job.
+
+
+### GET /batches/{batchId}/log
+
+Gets the log lines from this batch.
+
+#### Request Parameters
+
+<table class="table">
+ <tr><th>Name</th><th>Description</th><th>Type</th></tr>
+ <tr>
+ <td>from</td>
+ <td>Offset</td>
+ <td>Offset</td>
+ </tr>
+ <tr>
+ <td>size</td>
+ <td>Max number of log lines to return</td>
+ <td>Offset</td>
+ </tr>
+</table>
+
+#### Response Body
+
+<table class="table">
+ <tr><th>Name</th><th>Description</th><th>Type</th></tr>
+ <tr>
+ <td>id</td>
+ <td>The batch id</td>
+ <td>id</td>
+ </tr>
+ <tr>
+ <td>from</td>
+ <td>Offset from start of log</td>
+ <td>id</td>
+ </tr>
+ <tr>
+ <td>size</td>
+ <td>Number of log lines</td>
+ <td>id</td>
+ </tr>
+ <tr>
+ <td>log</td>
+ <td>The log lines</td>
+ <td>list of strings</td>
+ </tr>
+</table>
+
+## REST Objects
+
+### Session
+
+A session represents an interactive shell.
+
+<table class="table">
+ <tr><th>Name</th><th>Description</th><th>Type</th></tr>
+ <tr>
+ <td>id</td>
+ <td>The session id</td>
+ <td>int</td>
+ </tr>
+ <tr>
+ <td>appId</td>
+ <td>The application id of this session</td>
+ <td>string</td>
+ </tr>
+ <tr>
+ <td>owner</td>
+ <td>Remote user who submitted this session</td>
+ <td>string</td>
+ </tr>
+ <tr>
+ <td>proxyUser</td>
+ <td>User to impersonate when running</td>
+ <td>string</td>
+ </tr>
+ <tr>
+ <td>kind</td>
+ <td>Session kind (spark, pyspark, or sparkr)</td>
+ <td><a href="#session-kind">session kind</a></td>
+ </tr>
+ <tr>
+ <td>log</td>
+ <td>The log lines</td>
+ <td>list of strings</td>
+ </tr>
+ <tr>
+ <td>state</td>
+ <td>The session state</td>
+ <td>string</td>
+ </tr>
+ <tr>
+ <td>appInfo</td>
+ <td>The detailed application info</td>
+ <td>Map of key=val</td>
+ </tr>
+</table>
+
+
+#### Session State
+
+<table class="table">
+ <tr><th>Value</th><th>Description</th></tr>
+ <tr>
+ <td>not_started</td>
+ <td>Session has not been started</td>
+ </tr>
+ <tr>
+ <td>starting</td>
+ <td>Session is starting</td>
+ </tr>
+ <tr>
+ <td>idle</td>
+ <td>Session is waiting for input</td>
+ </tr>
+ <tr>
+ <td>busy</td>
+ <td>Session is executing a statement</td>
+ </tr>
+ <tr>
+ <td>shutting_down</td>
+ <td>Session is shutting down</td>
+ </tr>
+ <tr>
+ <td>error</td>
+ <td>Session errored out</td>
+ </tr>
+ <tr>
+ <td>dead</td>
+ <td>Session has exited</td>
+ </tr>
+ <tr>
+ <td>success</td>
+ <td>Session is successfully stopped</td>
+ </tr>
+</table>
+
+#### Session Kind
+
+<table class="table">
+ <tr><th>Value</th><th>Description</th></tr>
+ <tr>
+ <td>spark</td>
+ <td>Interactive Scala Spark session</td>
+ </tr>
+ <tr>
+ <td><a href="#pyspark">pyspark</a></td>
+ <td>Interactive Python 2 Spark session</td>
+ </tr>
+ <tr>
+ <td><a href="#pyspark3">pyspark3</a></td>
+ <td>Interactive Python 3 Spark session</td>
+ </tr>
+ <tr>
+ <td>sparkr</td>
+ <td>Interactive R Spark session</td>
+ </tr>
+</table>
+
+#### pyspark
+To change the Python executable the session uses, Livy reads the path from environment variable
+``PYSPARK_PYTHON`` (Same as pyspark).
+
+Like pyspark, if Livy is running in ``local`` mode, just set the environment variable.
+If the session is running in ``yarn-cluster`` mode, please set
+``spark.yarn.appMasterEnv.PYSPARK_PYTHON`` in SparkConf so the environment variable is passed to
+the driver.
+
+#### pyspark3
+To change the Python executable the session uses, Livy reads the path from environment variable
+``PYSPARK3_PYTHON``.
+
+Like pyspark, if Livy is running in ``local`` mode, just set the environment variable.
+If the session is running in ``yarn-cluster`` mode, please set
+``spark.yarn.appMasterEnv.PYSPARK3_PYTHON`` in SparkConf so the environment variable is passed to
+the driver.
+
+### Statement
+
+A statement represents the result of an execution statement.
+
+<table class="table">
+ <tr><th>Name</th><th>Description</th><th>Type</th></tr>
+ <tr>
+ <td>id</td>
+ <td>The statement id</td>
+ <td>integer</td>
+ </tr>
+ <tr>
+ <td>code</td>
+ <td>The execution code</td>
+ <td>string</td>
+ </tr>
+ <tr>
+ <td>state</td>
+ <td>The execution state</td>
+ <td>statement state</td>
+ </tr>
+ <tr>
+ <td>output</td>
+ <td>The execution output</td>
+ <td>statement output</td>
+ </tr>
+</table>
+
+#### Statement State
+
+<table class="table">
+ <tr><th>Value</th><th>Description</th></tr>
+ <tr>
+ <td>waiting</td>
+ <td>Statement is enqueued but execution hasn't started</td>
+ </tr>
+ <tr>
+ <td>running</td>
+ <td>Statement is currently running</td>
+ </tr>
+ <tr>
+ <td>available</td>
+ <td>Statement has a response ready</td>
+ </tr>
+ <tr>
+ <td>error</td>
+ <td>Statement failed</td>
+ </tr>
+ <tr>
+ <td>cancelling</td>
+ <td>Statement is being cancelling</td>
+ </tr>
+ <tr>
+ <td>cancelled</td>
+ <td>Statement is cancelled</td>
+ </tr>
+</table>
+
+#### Statement Output
+
+<table class="table">
+ <tr><th>Name</th><th>Description</th><th>Type</th></tr>
+ <tr>
+ <td>status</td>
+ <td>Execution status</td>
+ <td>string</td>
+ </tr>
+ <tr>
+ <td>execution_count</td>
+ <td>A monotonically increasing number</td>
+ <td>integer</td>
+ </tr>
+ <tr>
+ <td>data</td>
+ <td>Statement output</td>
+ <td>An object mapping a mime type to the result. If the mime type is
+ ``application/json``, the value is a JSON value.</td>
+ </tr>
+</table>
+
+### Batch
+
+<table class="table">
+ <tr><th>Name</th><th>Description</th><th>Type</th></tr>
+ <tr>
+ <td>id</td>
+ <td>The session id</td>
+ <td>int</td>
+ </tr>
+ <tr>
+ <td>appId</td>
+ <td>The application id of this session</td>
+ <td>string</td>
+ </tr>
+ <tr>
+ <td>appInfo</td>
+ <td>The detailed application info</td>
+ <td>Map of key=val</td>
+ </tr>
+ <tr>
+ <td>log</td>
+ <td>The log lines</td>
+ <td>list of strings</td>
+ </tr>
+ <tr>
+ <td>state</td>
+ <td>The batch state</td>
+ <td>string</td>
+ </tr>
+</table>