You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@lens.apache.org by am...@apache.org on 2015/08/19 07:09:10 UTC
incubator-lens git commit: LENS-351 : Document db resources feature
in user guide
Repository: incubator-lens
Updated Branches:
refs/heads/master df5567b78 -> c36859568
LENS-351 : Document db resources feature in user guide
Project: http://git-wip-us.apache.org/repos/asf/incubator-lens/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-lens/commit/c3685956
Tree: http://git-wip-us.apache.org/repos/asf/incubator-lens/tree/c3685956
Diff: http://git-wip-us.apache.org/repos/asf/incubator-lens/diff/c3685956
Branch: refs/heads/master
Commit: c36859568fce4b4f9ffcf880e405d5f370309b70
Parents: df5567b
Author: Rajat Khandelwal <pr...@apache.org>
Authored: Wed Aug 19 10:38:37 2015 +0530
Committer: Amareshwari Sriramadasu <am...@apache.org>
Committed: Wed Aug 19 10:38:37 2015 +0530
----------------------------------------------------------------------
src/site/apt/user/index.apt | 94 ++++++++++++++++++++++++++++------------
1 file changed, 67 insertions(+), 27 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-lens/blob/c3685956/src/site/apt/user/index.apt
----------------------------------------------------------------------
diff --git a/src/site/apt/user/index.apt b/src/site/apt/user/index.apt
index d48c734..6a86b1b 100644
--- a/src/site/apt/user/index.apt
+++ b/src/site/apt/user/index.apt
@@ -25,29 +25,29 @@ Lens User Guide
Lens Server runs several services which can be used from their REST endpoints.
This document covers some of the important services, their purpose and key API calls.
-
- Lens server provides metastore service for managing metadata. Metadata exposed by lens
+
+ Lens server provides metastore service for managing metadata. Metadata exposed by lens
is inspired by OLAP data cubes. See {{{./olap-cube.html#Metastore_model} Metastore Model}} for metastore constructs
that lens provides.
-
+
Lens server also provides query service for querying data exposed by lens. See
{{{./olap-cube.html#Query_Language} Query Language}} for the grammar of the query.
-
- To access any service on lens, user should be working in a session. User can also pass
+
+ To access any service on lens, user should be working in a session. User can also pass
various configuration parameters from client.
-
- The sections below give more details on each service and configuration.
+
+ The sections below give more details on each service and configuration.
* Configuration
Client configuration can be overridden in lens-client-site.xml.
See {{{./client-config.html}client configuration}} for all the configuration parameters
available and their default values.
-
+
* Session Service
- To use any Lens service the user must first create a session. Each Lens session is associated with a unique session handle, which must be passed when making queries, or doing metadata operations in the same session. To check if the session service is deployed in the Lens Server, user can send a GET request to /session. An OK response means that the session service is deployed.
-
+ To use any Lens service the user must first create a session. Each Lens session is associated with a unique session handle, which must be passed when making queries, or doing metadata operations in the same session. To check if the session service is deployed in the Lens Server, user can send a GET request to /session. An OK response means that the session service is deployed.
+
Sessions also allow users to set configuration or resources which could be shared across a group of queries. For example, if a group of queries need to call a UDF available in some specific jar, then the jar file can be added as a resource in the session. All queries started within the same session can make use of the jar file.
@@ -56,18 +56,58 @@ Lens User Guide
and {{{./cli.html#Session_management} CLI}} for doing all session level operations.
The important API calls exposed by the session resource are -
-
- * {{{../resource_SessionResource.html#path__session.html}/session}} - Endpoint to create or delete a session.
+
+ * {{{../resource_SessionResource.html#path__session.html}/session}} - Endpoint to create or delete a session.
* {{{../resource_SessionResource.html#path__session_params.html}/session/params}} - Endpoint to maintain session settings
* {{{../resource_SessionResource.html#path__session_resources_add.html}/session/resources}} - Adding or removing resources from the session
+
+ While adding resources through client, you have to provide a path to the resource file and type. The type can either
+ be <<<jar>>> or <<<file>>>. The path has to be a path accessible by the lens server. E.g. It can be
+ a hdfs path or a local path on lens server's file system. But it can't be a path only accessible on the client
+ machine. Lens client doesn't upload the jar to the server, just passes the path through. The path can be a a path
+ to a file/jar, a path to a directory or a regex path. Given path will be converted to a list of paths of the
+ provided type and all those resources will be added.
+
+ ** If Given path is a path to a file which exists, the list will be a singleton list containing only that file.
+
+ ** If Given path is a directory path, server will see if a <<<jar_order>>>(for type jar) or a <<<glob_order>>>(for type file) file exists.
+
+ *** If yes, that file is read. The order file is supposed to list the order for adding resources. Resources will be
+ added in that order.
+
+ *** else, the directory is globbed and resources will be added in glob order.
+
+ ** If given path is a regex path, regex path is resolved to list of paths. Each path is then resolved to a list of
+ paths since they can either be file or directory.
+
+ ** order file can also contain regexes which are handled like the above point.
+
+
+* Database Resource Service
+
+ Lens server provides a service which automatically adds jars to a session
+ depending on which database the user is using. The idea is, that each database can store one collection of schemas
+ in it, and might have some required resources for them to work properly. So all the resources for the database
+ can be provided to the server at deployment time and server will automatically add all those to every session that
+ switches to that database.
+
+ DB resources currently only deals with jars. To use this, first decide on a directory where the jars for each db
+ will be stored. You can set the value in <<<lens.server.database.resource.dir>>> in <<<lens-site.xml>>> as
+ described in {{{../admin/config.html}Server Config}}. The default
+ is <<</tmp/lens/resources>>>. I'll explain the workings using the default directory. so /tmp/lens/resources/dbname
+ is supposed to store all jars for db <dbname>. After the user switches to <dbname>, the first command after the
+ switch will add all jars to the session. Now sometimes the order in which jars are to be added can also be
+ important. For that, you can supply a <<<jar_order>>> file as described in previous section.
+
+
[]
* Query Execution Service
-
- The {{{../resource_QueryServiceResource.html}Query Execution Service}} is used to query data exposed by Lens.
+
+ The {{{../resource_QueryServiceResource.html}Query Execution Service}} is used to query data exposed by Lens.
** Query Submission Workflow
@@ -86,7 +126,7 @@ Lens User Guide
To summarize, given below are steps for batch (async) queries
[[1]] Create session, note returned session handle.
-
+
[[2]] Create query by passing session handle, note returned query handle.
[[3]] Poll for query status by passing session handle and query handle.
@@ -94,20 +134,20 @@ Lens User Guide
[[4]] If query is SUCCESSFUL, get results.
[]
-
+
Steps for interactive queries.
[[1]] Create a session.
[[2]] Create a query by setting <<<op=EXECUTE_WITH_TIMEOUT>>>, Also set the <<<timeout>>> value.
- [[3]] Check the response, if it contains only the query handle, then poll for status as is the case in async queries. If it contains both query handle and result set, then that means query did complete successfully within the timeout.
+ [[3]] Check the response, if it contains only the query handle, then poll for status as is the case in async queries. If it contains both query handle and result set, then that means query did complete successfully within the timeout.
[]
** Getting query results
-
- A query can be run once, but its results can be fetched any number of times,
+
+ A query can be run once, but its results can be fetched any number of times,
if the results are persisted, until its purged from server memory. Results
can be obtained by sending a GET to <<</queryapi/queries/{queryhandle}/resultset>>>.
This endpoint takes optional <<<fromindex>>> and <<<fetchsize>>> parameters
@@ -118,39 +158,39 @@ Lens User Guide
** Life of a Query in the Lens Server
- The following diagram shows query state transition in the Lens Server
+ The following diagram shows query state transition in the Lens Server
[/images/querystate.png] Query States in Lens
- When user submits a query to the Lens Server, its starts in the <<NEW>> state. After the query is submitted, it moves into the <<QUEUED>> state. Until Lens server is free to take up the query, it remains in the <<QUEUED>> state. As soon as Lens server starts processing the query, it enters the <<LAUNCHED>> state. At this stage Lens has decided which backend engine will be used to execute the query.
+ When user submits a query to the Lens Server, its starts in the <<NEW>> state. After the query is submitted, it moves into the <<QUEUED>> state. Until Lens server is free to take up the query, it remains in the <<QUEUED>> state. As soon as Lens server starts processing the query, it enters the <<LAUNCHED>> state. At this stage Lens has decided which backend engine will be used to execute the query.
For each query Lens server will poll the chosen backend engine for query status. A GET on the query endpoint returns the latest status of the query.
- The <<RUNNING>> state indicates that the query is currently being processed by the query backend.
+ The <<RUNNING>> state indicates that the query is currently being processed by the query backend.
After the <<RUNNING>> state, the query can enter either the <<EXECUTED>> or <<FAILED>> states, depending on the result of query execution.
If the execution is successful, then server would format the result if required and then set the state to
<<SUCCESSFUL>> or <<FAILED>> if formatting fails.
If the query is <<SUCCESSFUL>>, its result set can be retrieved using the result set API call, by passing the session handle and query handle. The query can be executed once, and its results can be fetched multiple times unless the query has been purged from Lens server state.
- In any state, if the user requests that the query be cancelled, the query will enter into <<CANCELLED>> state. Query can be cancelled by sending a DELETE at the query endpoint.
+ In any state, if the user requests that the query be cancelled, the query will enter into <<CANCELLED>> state. Query can be cancelled by sending a DELETE at the query endpoint.
<<FAILED>>, <<SUCCESSFUL>> and <<CANCELLED>> are end states for a query. Once a query reaches these states, it becomes eligible to purging. The query is purged when its purge delay expires, after which it is not possible to retrieve results of the query. This purge delay is configurable. After purging the query enters the <<CLOSED>> state.
** Prepared queries
A query can be prepared for execution. Once prepared the query can be submitted for execution as many times as required. When prepared query is no longer required, it should be destroyed. REST api, JavaClient api and CLI commands are available for all the operations supported.
-
+
* {{{../resource_QueryServiceResource.html#path__queryapi_preparedqueries.html}Rest api for prepared queries}}
* {{{../resource_QueryServiceResource.html#path__queryapi_preparedqueries_-prepareHandle-.html}Rest api for handling a prepared query}}
-
+
* {{{../apidocs/org/apache/lens/client/LensStatement.html} Java client api}}
-
+
* {{{./cli.html#Query_Management} CLI query management}}
-
+
* Metastore service
The Metastore service is used for DDL operations like creating, updating cubes, fact tables and dimensions. It also pprovides endpoints to create storage tables and to add partitions to a storage table. For more detailed information see the {{{../resource_MetastoreResource.html}metastore service resource}} documentation.