You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@drill.apache.org by ts...@apache.org on 2015/05/30 07:03:43 UTC
[17/26] drill git commit: clarify DFS vs local FS,
resolve pull request 97798bf4e555db85ba75e1bc8d18d3298cff989b, typos
clarify DFS vs local FS, resolve pull request 97798bf4e555db85ba75e1bc8d18d3298cff989b, typos
Project: http://git-wip-us.apache.org/repos/asf/drill/repo
Commit: http://git-wip-us.apache.org/repos/asf/drill/commit/9a2f1ba9
Tree: http://git-wip-us.apache.org/repos/asf/drill/tree/9a2f1ba9
Diff: http://git-wip-us.apache.org/repos/asf/drill/diff/9a2f1ba9
Branch: refs/heads/gh-pages
Commit: 9a2f1ba9243c66144bd715a70a707f0de491aa75
Parents: 806f24f
Author: Kristine Hahn <kh...@maprtech.com>
Authored: Fri May 29 10:32:25 2015 -0700
Committer: Kristine Hahn <kh...@maprtech.com>
Committed: Fri May 29 10:32:25 2015 -0700
----------------------------------------------------------------------
_data/docs.json | 112 +++++++++----------
.../020-configuring-drill-memory.md | 4 +-
.../010-connect-a-data-source-introduction.md | 9 +-
.../040-file-system-storage-plugin.md | 105 +++++++++++++++++
_docs/connect-a-data-source/040-workspaces.md | 76 -------------
.../050-file-system-storage-plugin.md | 64 -----------
_docs/connect-a-data-source/050-workspaces.md | 33 ++++++
.../connect-a-data-source/100-mapr-db-format.md | 3 +-
.../050-json-data-model.md | 4 +-
.../020-develop-a-simple-function.md | 4 +-
.../030-developing-an-aggregate-function.md | 18 +++
.../060-custom-function-interfaces.md | 14 +--
.../design-docs/050-value-vectors.md | 2 +-
_docs/img/connect-plugin.png | Bin 36731 -> 41222 bytes
.../data-types/010-supported-data-types.md | 7 ++
15 files changed, 239 insertions(+), 216 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/drill/blob/9a2f1ba9/_data/docs.json
----------------------------------------------------------------------
diff --git a/_data/docs.json b/_data/docs.json
index b9c3f83..25cc5ff 100644
--- a/_data/docs.json
+++ b/_data/docs.json
@@ -1435,8 +1435,8 @@
}
],
"children": [],
- "next_title": "Workspaces",
- "next_url": "/docs/workspaces/",
+ "next_title": "File System Storage Plugin",
+ "next_url": "/docs/file-system-storage-plugin/",
"parent": "Storage Plugin Configuration",
"previous_title": "Storage Plugin Configuration",
"previous_url": "/docs/storage-plugin-configuration/",
@@ -1456,14 +1456,14 @@
}
],
"children": [],
- "next_title": "File System Storage Plugin",
- "next_url": "/docs/file-system-storage-plugin/",
+ "next_title": "Workspaces",
+ "next_url": "/docs/workspaces/",
"parent": "Storage Plugin Configuration",
"previous_title": "Plugin Configuration Introduction",
"previous_url": "/docs/plugin-configuration-introduction/",
- "relative_path": "_docs/connect-a-data-source/040-workspaces.md",
- "title": "Workspaces",
- "url": "/docs/workspaces/"
+ "relative_path": "_docs/connect-a-data-source/040-file-system-storage-plugin.md",
+ "title": "File System Storage Plugin",
+ "url": "/docs/file-system-storage-plugin/"
},
{
"breadcrumbs": [
@@ -1480,11 +1480,11 @@
"next_title": "HBase Storage Plugin",
"next_url": "/docs/hbase-storage-plugin/",
"parent": "Storage Plugin Configuration",
- "previous_title": "Workspaces",
- "previous_url": "/docs/workspaces/",
- "relative_path": "_docs/connect-a-data-source/050-file-system-storage-plugin.md",
- "title": "File System Storage Plugin",
- "url": "/docs/file-system-storage-plugin/"
+ "previous_title": "File System Storage Plugin",
+ "previous_url": "/docs/file-system-storage-plugin/",
+ "relative_path": "_docs/connect-a-data-source/050-workspaces.md",
+ "title": "Workspaces",
+ "url": "/docs/workspaces/"
},
{
"breadcrumbs": [
@@ -1501,8 +1501,8 @@
"next_title": "Hive Storage Plugin",
"next_url": "/docs/hive-storage-plugin/",
"parent": "Storage Plugin Configuration",
- "previous_title": "File System Storage Plugin",
- "previous_url": "/docs/file-system-storage-plugin/",
+ "previous_title": "Workspaces",
+ "previous_url": "/docs/workspaces/",
"relative_path": "_docs/connect-a-data-source/060-hbase-storage-plugin.md",
"title": "HBase Storage Plugin",
"url": "/docs/hbase-storage-plugin/"
@@ -2965,12 +2965,12 @@
}
],
"children": [],
- "next_title": "HBase Storage Plugin",
- "next_url": "/docs/hbase-storage-plugin/",
+ "next_title": "Workspaces",
+ "next_url": "/docs/workspaces/",
"parent": "Storage Plugin Configuration",
- "previous_title": "Workspaces",
- "previous_url": "/docs/workspaces/",
- "relative_path": "_docs/connect-a-data-source/050-file-system-storage-plugin.md",
+ "previous_title": "Plugin Configuration Introduction",
+ "previous_url": "/docs/plugin-configuration-introduction/",
+ "relative_path": "_docs/connect-a-data-source/040-file-system-storage-plugin.md",
"title": "File System Storage Plugin",
"url": "/docs/file-system-storage-plugin/"
},
@@ -3116,8 +3116,8 @@
"next_title": "Hive Storage Plugin",
"next_url": "/docs/hive-storage-plugin/",
"parent": "Storage Plugin Configuration",
- "previous_title": "File System Storage Plugin",
- "previous_url": "/docs/file-system-storage-plugin/",
+ "previous_title": "Workspaces",
+ "previous_url": "/docs/workspaces/",
"relative_path": "_docs/connect-a-data-source/060-hbase-storage-plugin.md",
"title": "HBase Storage Plugin",
"url": "/docs/hbase-storage-plugin/"
@@ -5564,8 +5564,8 @@
}
],
"children": [],
- "next_title": "Workspaces",
- "next_url": "/docs/workspaces/",
+ "next_title": "File System Storage Plugin",
+ "next_url": "/docs/file-system-storage-plugin/",
"parent": "Storage Plugin Configuration",
"previous_title": "Storage Plugin Configuration",
"previous_url": "/docs/storage-plugin-configuration/",
@@ -9339,8 +9339,8 @@
}
],
"children": [],
- "next_title": "Workspaces",
- "next_url": "/docs/workspaces/",
+ "next_title": "File System Storage Plugin",
+ "next_url": "/docs/file-system-storage-plugin/",
"parent": "Storage Plugin Configuration",
"previous_title": "Storage Plugin Configuration",
"previous_url": "/docs/storage-plugin-configuration/",
@@ -9360,14 +9360,14 @@
}
],
"children": [],
- "next_title": "File System Storage Plugin",
- "next_url": "/docs/file-system-storage-plugin/",
+ "next_title": "Workspaces",
+ "next_url": "/docs/workspaces/",
"parent": "Storage Plugin Configuration",
"previous_title": "Plugin Configuration Introduction",
"previous_url": "/docs/plugin-configuration-introduction/",
- "relative_path": "_docs/connect-a-data-source/040-workspaces.md",
- "title": "Workspaces",
- "url": "/docs/workspaces/"
+ "relative_path": "_docs/connect-a-data-source/040-file-system-storage-plugin.md",
+ "title": "File System Storage Plugin",
+ "url": "/docs/file-system-storage-plugin/"
},
{
"breadcrumbs": [
@@ -9384,11 +9384,11 @@
"next_title": "HBase Storage Plugin",
"next_url": "/docs/hbase-storage-plugin/",
"parent": "Storage Plugin Configuration",
- "previous_title": "Workspaces",
- "previous_url": "/docs/workspaces/",
- "relative_path": "_docs/connect-a-data-source/050-file-system-storage-plugin.md",
- "title": "File System Storage Plugin",
- "url": "/docs/file-system-storage-plugin/"
+ "previous_title": "File System Storage Plugin",
+ "previous_url": "/docs/file-system-storage-plugin/",
+ "relative_path": "_docs/connect-a-data-source/050-workspaces.md",
+ "title": "Workspaces",
+ "url": "/docs/workspaces/"
},
{
"breadcrumbs": [
@@ -9405,8 +9405,8 @@
"next_title": "Hive Storage Plugin",
"next_url": "/docs/hive-storage-plugin/",
"parent": "Storage Plugin Configuration",
- "previous_title": "File System Storage Plugin",
- "previous_url": "/docs/file-system-storage-plugin/",
+ "previous_title": "Workspaces",
+ "previous_url": "/docs/workspaces/",
"relative_path": "_docs/connect-a-data-source/060-hbase-storage-plugin.md",
"title": "HBase Storage Plugin",
"url": "/docs/hbase-storage-plugin/"
@@ -10498,12 +10498,12 @@
}
],
"children": [],
- "next_title": "File System Storage Plugin",
- "next_url": "/docs/file-system-storage-plugin/",
+ "next_title": "HBase Storage Plugin",
+ "next_url": "/docs/hbase-storage-plugin/",
"parent": "Storage Plugin Configuration",
- "previous_title": "Plugin Configuration Introduction",
- "previous_url": "/docs/plugin-configuration-introduction/",
- "relative_path": "_docs/connect-a-data-source/040-workspaces.md",
+ "previous_title": "File System Storage Plugin",
+ "previous_url": "/docs/file-system-storage-plugin/",
+ "relative_path": "_docs/connect-a-data-source/050-workspaces.md",
"title": "Workspaces",
"url": "/docs/workspaces/"
}
@@ -11489,8 +11489,8 @@
}
],
"children": [],
- "next_title": "Workspaces",
- "next_url": "/docs/workspaces/",
+ "next_title": "File System Storage Plugin",
+ "next_url": "/docs/file-system-storage-plugin/",
"parent": "Storage Plugin Configuration",
"previous_title": "Storage Plugin Configuration",
"previous_url": "/docs/storage-plugin-configuration/",
@@ -11510,14 +11510,14 @@
}
],
"children": [],
- "next_title": "File System Storage Plugin",
- "next_url": "/docs/file-system-storage-plugin/",
+ "next_title": "Workspaces",
+ "next_url": "/docs/workspaces/",
"parent": "Storage Plugin Configuration",
"previous_title": "Plugin Configuration Introduction",
"previous_url": "/docs/plugin-configuration-introduction/",
- "relative_path": "_docs/connect-a-data-source/040-workspaces.md",
- "title": "Workspaces",
- "url": "/docs/workspaces/"
+ "relative_path": "_docs/connect-a-data-source/040-file-system-storage-plugin.md",
+ "title": "File System Storage Plugin",
+ "url": "/docs/file-system-storage-plugin/"
},
{
"breadcrumbs": [
@@ -11534,11 +11534,11 @@
"next_title": "HBase Storage Plugin",
"next_url": "/docs/hbase-storage-plugin/",
"parent": "Storage Plugin Configuration",
- "previous_title": "Workspaces",
- "previous_url": "/docs/workspaces/",
- "relative_path": "_docs/connect-a-data-source/050-file-system-storage-plugin.md",
- "title": "File System Storage Plugin",
- "url": "/docs/file-system-storage-plugin/"
+ "previous_title": "File System Storage Plugin",
+ "previous_url": "/docs/file-system-storage-plugin/",
+ "relative_path": "_docs/connect-a-data-source/050-workspaces.md",
+ "title": "Workspaces",
+ "url": "/docs/workspaces/"
},
{
"breadcrumbs": [
@@ -11555,8 +11555,8 @@
"next_title": "Hive Storage Plugin",
"next_url": "/docs/hive-storage-plugin/",
"parent": "Storage Plugin Configuration",
- "previous_title": "File System Storage Plugin",
- "previous_url": "/docs/file-system-storage-plugin/",
+ "previous_title": "Workspaces",
+ "previous_url": "/docs/workspaces/",
"relative_path": "_docs/connect-a-data-source/060-hbase-storage-plugin.md",
"title": "HBase Storage Plugin",
"url": "/docs/hbase-storage-plugin/"
http://git-wip-us.apache.org/repos/asf/drill/blob/9a2f1ba9/_docs/configure-drill/020-configuring-drill-memory.md
----------------------------------------------------------------------
diff --git a/_docs/configure-drill/020-configuring-drill-memory.md b/_docs/configure-drill/020-configuring-drill-memory.md
index 30d5121..ad46997 100644
--- a/_docs/configure-drill/020-configuring-drill-memory.md
+++ b/_docs/configure-drill/020-configuring-drill-memory.md
@@ -33,8 +33,8 @@ The `drill-env.sh` file contains the following options:
export DRILL_JAVA_OPTS="-Xms1G -Xmx$DRILL_MAX_HEAP -XX:MaxDirectMemorySize=$DRILL_MAX_DIRECT_MEMORY -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=1G -ea"
-* DRILL_MAX_DIRECT_MEMORY is the Java direct memory.
-* DRILL_MAX_HEAP is the maximum theoretical heap limit for the JVM.
+* DRILL_MAX_DIRECT_MEMORY is the Java direct memory limit per node.
+* DRILL_MAX_HEAP is the maximum theoretical heap limit for the JVM per node.
* Xmx specifies the maximum memory allocation pool for a Java Virtual Machine (JVM).
* Xms specifies the initial memory allocation pool.
http://git-wip-us.apache.org/repos/asf/drill/blob/9a2f1ba9/_docs/connect-a-data-source/010-connect-a-data-source-introduction.md
----------------------------------------------------------------------
diff --git a/_docs/connect-a-data-source/010-connect-a-data-source-introduction.md b/_docs/connect-a-data-source/010-connect-a-data-source-introduction.md
index 29133c0..b86542b 100644
--- a/_docs/connect-a-data-source/010-connect-a-data-source-introduction.md
+++ b/_docs/connect-a-data-source/010-connect-a-data-source-introduction.md
@@ -7,10 +7,13 @@ A storage plugin provides the following information to Drill:
* Interfaces that Drill can use to read from and write to data sources.
* A set of storage plugin optimization rules that assist with efficient and faster execution of Drill queries, such as pushdowns, statistics, and partition awareness.
-Apache Drill connects to a data source, such as a file on the file system or a Hive metastore, through a storage plugin. When you execute a query, Drill gets the plugin name you provide in FROM clause of your query or from the default you specify in the USE.<plugin name> command that precedes the query.
-.
+Through the storage plugin, Drill connects to a data source, such as a database, a file on a local or distributed file system, or a Hive metastore. When you execute a query, Drill gets the plugin name in one of several ways:
-In addition to the connection string, the storage plugin configures the workspace and file formats for reading data, as described in subsequent sections.
+* The FROM clause of the query can identify the plugin to use.
+* The USE <plugin name> command can precede the query.
+* You can specify the storage plugin when starting Drill.
+
+In addition to providing a the connection string to the data source, the storage plugin configures the workspace and file formats for reading data, as described in subsequent sections.
## Storage Plugins Internals
The following image represents the storage plugin layer between Drill and a
http://git-wip-us.apache.org/repos/asf/drill/blob/9a2f1ba9/_docs/connect-a-data-source/040-file-system-storage-plugin.md
----------------------------------------------------------------------
diff --git a/_docs/connect-a-data-source/040-file-system-storage-plugin.md b/_docs/connect-a-data-source/040-file-system-storage-plugin.md
new file mode 100644
index 0000000..9f16bde
--- /dev/null
+++ b/_docs/connect-a-data-source/040-file-system-storage-plugin.md
@@ -0,0 +1,105 @@
+---
+title: "File System Storage Plugin"
+parent: "Storage Plugin Configuration"
+---
+You can register a storage plugin instance that connects Drill to a local file system or to a distributed file system registered in `core-site.xml`, such as S3
+or HDFS. By
+default, Drill includes an instance named `dfs` that points to the local file
+system on your machine.
+
+## Connecting Drill to a File System
+
+In a Drill cluster, you typically do not query the local file system, but instead place files on the distributed file system. You configure the connection property of the storage plugin workspace to connect Drill to a distributed file system. For example, the following connection properties connect Drill to an HDFS, MapR-FS, or Mongo-DB cluster:
+
+* HDFS
+ `"connection": "hdfs://<IP Address>:<Port>/"`
+* MapR-FS Remote Cluster
+ `"connection": "maprfs://<IP Address>/"`
+* Mongo-DB Cluster
+ `"connection": "mongodb://<IP Address>:<Port>/"
+
+The Drill installation includes a [Mongo-DB storage plugin]({{site.baseurl}}/docs/mongodb-plugin-for-apache-drill).
+
+To register a local or a distributed file system with Apache Drill, complete
+the following steps:
+
+ 1. Navigate to `[http://localhost:8047](http://localhost:8047/)`, and select the **Storage** tab.
+ 2. In the New Storage Plugin window, enter a unique name and then click **Create**.
+ 3. In the Configuration window, provide the following configuration information for the type of file system that you are configuring as a data source.
+ * Local file system example:
+
+ {
+ "type": "file",
+ "enabled": true,
+ "connection": "file:///",
+ "workspaces": {
+ "root": {
+ "location": "/user/max/donuts",
+ "writable": false,
+ "defaultinputformat": null
+ }
+ },
+ "formats" : {
+ "json" : {
+ "type" : "json"
+ }
+ }
+ }
+ * Distributed file system example:
+
+ {
+ "type" : "file",
+ "enabled" : true,
+ "connection" : "hdfs://10.10.30.156:8020/",
+ "workspaces" : {
+ "root : {
+ "location" : "/user/root/drill",
+ "writable" : true,
+ "defaultinputformat" : "null"
+ }
+ },
+ "formats" : {
+ "json" : {
+ "type" : "json"
+ }
+ }
+ }
+
+ To connect to a Hadoop file system, you include the IP address of the
+name node and the port number.
+ 4. Click **Enable**.
+
+Once you have configured a storage plugin instance for the file system, you
+can issue Drill queries against it.
+
+The following example shows an instance of a file type storage plugin with a
+workspace named `json_files` configured to point Drill to the
+`/users/max/drill/json/` directory in the local file system `(dfs)`:
+
+ {
+ "type" : "file",
+ "enabled" : true,
+ "connection" : "file:///",
+ "workspaces" : {
+ "json_files" : {
+ "location" : "/users/max/drill/json/",
+ "writable" : false,
+ "defaultinputformat" : json
+ }
+ },
+
+{% include startnote.html %}The `connection` parameter in the configuration above is "`file:///`", connecting Drill to the local file system (`dfs`).{% include endnote.html %}
+
+To query a file in the example `json_files` workspace, you can issue the `USE`
+command to tell Drill to use the `json_files` workspace configured in the `dfs`
+instance for each query that you issue:
+
+**Example**
+
+ USE dfs.json_files;
+ SELECT * FROM dfs.json_files.`donuts.json` WHERE type='frosted'
+
+If the `json_files` workspace did not exist, the query would have to include the
+full path to the `donuts.json` file:
+
+ SELECT * FROM dfs.`/users/max/drill/json/donuts.json` WHERE type='frosted';
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/9a2f1ba9/_docs/connect-a-data-source/040-workspaces.md
----------------------------------------------------------------------
diff --git a/_docs/connect-a-data-source/040-workspaces.md b/_docs/connect-a-data-source/040-workspaces.md
deleted file mode 100644
index 864c7ce..0000000
--- a/_docs/connect-a-data-source/040-workspaces.md
+++ /dev/null
@@ -1,76 +0,0 @@
----
-title: "Workspaces"
-parent: "Storage Plugin Configuration"
----
-When you register an instance of a file system data source, you can configure
-one or more workspaces for the instance. A workspace is a directory within the
-file system that you define. Drill searches the workspace to locate data when
-you run a query.
-
-Each workspace that you register defines a schema that you can connect to and
-query. Configuring workspaces is useful when you want to run multiple queries
-on files or tables in a specific directory. You cannot create workspaces for
-`hive` and `hbase` instances, though Hive databases show up as workspaces in
-Drill.
-
-The following example shows an instance of a file type storage plugin with a
-workspace named `json` configured to point Drill to the
-`/users/max/drill/json/` directory in the local file system `(dfs)`:
-
- {
- "type" : "file",
- "enabled" : true,
- "connection" : "file:///",
- "workspaces" : {
- "json" : {
- "location" : "/users/max/drill/json/",
- "writable" : false,
- "defaultinputformat" : json
- }
- },
-
-{% include startnote.html %}The `connection` parameter in the configuration above is "`file:///`", connecting Drill to the local file system (`dfs`).{% include endnote.html %}
-To connect to a Hadoop or MapR file system the `connection` parameter would be "`hdfs:///" `or` "maprfs:///", `respectively.
-
-To query a file in the example `json` workspace, you can issue the `USE`
-command to tell Drill to use the `json` workspace configured in the `dfs`
-instance for each query that you issue:
-
-**Example**
-
- USE dfs.json;
- SELECT * FROM dfs.json.`donuts.json` WHERE type='frosted'
-
-If the `json` workspace did not exist, the query would have to include the
-full path to the `donuts.json` file:
-
- SELECT * FROM dfs.`/users/max/drill/json/donuts.json` WHERE type='frosted';
-
-Using a workspace alleviates the need to repeatedly enter the directory path
-in subsequent queries on the directory.
-
-### Default Workspaces
-
-Each `file` and `hive` instance includes a `default` workspace. The `default`
-workspace points to the file system or to the Hive metastore. When you query
-files and tables in the `file` or `hive default` workspaces, you can omit the
-workspace name from the query.
-
-For example, you can issue a query on a Hive table in the `default workspace`
-using either of the following formats and get the same results:
-
-**Example**
-
- SELECT * FROM hive.customers LIMIT 10;
- SELECT * FROM hive.`default`.customers LIMIT 10;
-
-{% include startnote.html %}Default is a reserved word. You must enclose reserved words in back ticks.{% include endnote.html %}
-
-
-Because HBase instances do not have workspaces, you can use the following
-format to query a table in HBase:
-
- SELECT * FROM hbase.customers LIMIT 10;
-
-After you register a data source as a storage plugin instance with Drill, and
-optionally configure workspaces, you can query the data source.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/9a2f1ba9/_docs/connect-a-data-source/050-file-system-storage-plugin.md
----------------------------------------------------------------------
diff --git a/_docs/connect-a-data-source/050-file-system-storage-plugin.md b/_docs/connect-a-data-source/050-file-system-storage-plugin.md
deleted file mode 100644
index 2b3e287..0000000
--- a/_docs/connect-a-data-source/050-file-system-storage-plugin.md
+++ /dev/null
@@ -1,64 +0,0 @@
----
-title: "File System Storage Plugin"
-parent: "Storage Plugin Configuration"
----
-You can register a storage plugin instance that connects Drill to a local file
-system or a distributed file system registered in `core-site.xml`, such as S3
-or HDFS. When you register a storage plugin instance for a file system,
-provide a unique name for the instance, and identify the type as “`file`”. By
-default, Drill includes an instance named `dfs` that points to the local file
-system on your machine. You can update this configuration to point to a
-distributed file system or you can create a new instance to point to a
-distributed file system.
-
-To register a local or a distributed file system with Apache Drill, complete
-the following steps:
-
- 1. Navigate to `[http://localhost:8047](http://localhost:8047/)`, and select the **Storage** tab.
- 2. In the New Storage Plugin window, enter a unique name and then click **Create**.
- 3. In the Configuration window, provide the following configuration information for the type of file system that you are configuring as a data source.
- 1. Local file system example:
-
- {
- "type": "file",
- "enabled": true,
- "connection": "file:///",
- "workspaces": {
- "root": {
- "location": "/user/max/donuts",
- "writable": false,
- "defaultinputformat": null
- }
- },
- "formats" : {
- "json" : {
- "type" : "json"
- }
- }
- }
- 2. Distributed file system example:
-
- {
- "type" : "file",
- "enabled" : true,
- "connection" : "hdfs://10.10.30.156:8020/",
- "workspaces" : {
- "root : {
- "location" : "/user/root/drill",
- "writable" : true,
- "defaultinputformat" : "null"
- }
- },
- "formats" : {
- "json" : {
- "type" : "json"
- }
- }
- }
-
- To connect to a Hadoop file system, you must include the IP address of the
-name node and the port number.
- 4. Click **Enable**.
-
-Once you have configured a storage plugin instance for the file system, you
-can issue Drill queries against it.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/9a2f1ba9/_docs/connect-a-data-source/050-workspaces.md
----------------------------------------------------------------------
diff --git a/_docs/connect-a-data-source/050-workspaces.md b/_docs/connect-a-data-source/050-workspaces.md
new file mode 100644
index 0000000..361bfec
--- /dev/null
+++ b/_docs/connect-a-data-source/050-workspaces.md
@@ -0,0 +1,33 @@
+---
+title: "Workspaces"
+parent: "Storage Plugin Configuration"
+---
+When you register an instance of a file system data source, you can configure
+one or more workspaces for the instance. The workspace defines the default directory location of files in a local or distributed file system. The `default`
+workspace points to the root of the file system. Drill searches the workspace to locate data when
+you run a query.
+
+You cannot create workspaces for
+`hive` and `hbase` storage plugins, though Hive databases show up as workspaces in
+Drill. Each `hive` instance includes a `default` workspace that points to the Hive metastore. When you query
+files and tables in the `hive default` workspaces, you can omit the
+workspace name from the query.
+
+For example, you can issue a query on a Hive table in the `default workspace`
+using either of the following formats and get the same results:
+
+**Example**
+
+ SELECT * FROM hive.customers LIMIT 10;
+ SELECT * FROM hive.`default`.customers LIMIT 10;
+
+{% include startnote.html %}Default is a reserved word. You must enclose reserved words in back ticks.{% include endnote.html %}
+
+Because HBase instances do not have workspaces, you can use the following
+format to query a table in HBase:
+
+ SELECT * FROM hbase.customers LIMIT 10;
+
+After you register a data source as a storage plugin instance with Drill, and
+optionally configure workspaces, you can query the data source.
+
http://git-wip-us.apache.org/repos/asf/drill/blob/9a2f1ba9/_docs/connect-a-data-source/100-mapr-db-format.md
----------------------------------------------------------------------
diff --git a/_docs/connect-a-data-source/100-mapr-db-format.md b/_docs/connect-a-data-source/100-mapr-db-format.md
index f101dfa..b091f8e 100644
--- a/_docs/connect-a-data-source/100-mapr-db-format.md
+++ b/_docs/connect-a-data-source/100-mapr-db-format.md
@@ -2,8 +2,7 @@
title: "MapR-DB Format"
parent: "Connect a Data Source"
---
-The MapR-DB format is not included in apache drill release. Drill includes a `maprdb` format for MapR-DB that is defined within the
-default `dfs` storage plugin instance when you install Drill from the `mapr-drill` package on a MapR node. The `maprdb` format improves the
+The MapR-DB format is not included in Apache Drill release. If you install Drill from the `mapr-drill` package on a MapR node, the MapR-DB format appears in the `dfs` storage plugin instance. The `maprdb` format improves the
estimated number of rows that Drill uses to plan a query. It also enables you
to query tables like you would query files in a file system because MapR-DB
and MapR-FS share the same namespace.
http://git-wip-us.apache.org/repos/asf/drill/blob/9a2f1ba9/_docs/data-sources-and-file-formats/050-json-data-model.md
----------------------------------------------------------------------
diff --git a/_docs/data-sources-and-file-formats/050-json-data-model.md b/_docs/data-sources-and-file-formats/050-json-data-model.md
index 1b1660d..1bc4cec 100644
--- a/_docs/data-sources-and-file-formats/050-json-data-model.md
+++ b/_docs/data-sources-and-file-formats/050-json-data-model.md
@@ -126,7 +126,7 @@ Using the following techniques, you can query complex, nested JSON:
* Generate key/value pairs for loosely structured data
## Example: Flatten and Generate Key Values for Complex JSON
-This example uses the following data that represents unit sales of tickets to events that were sold over a period of for several days in December:
+This example uses the following data that represents unit sales of tickets to events that were sold over a period of several days in December:
### ticket_sales.json Contents
@@ -214,7 +214,7 @@ Sum the ticket sales by combining the `SUM`, `FLATTEN`, and `KVGEN` functions in
1 row selected (0.244 seconds)
### Example: Aggregate and Sort Data
-Sum the ticket sales by state and group by day and sort in ascending order.
+Sum and group the ticket sales by date and sort in ascending order of total tickets sold.
SELECT `right`(tkt.tot_sales.key,2) `December Date`,
SUM(tkt.tot_sales.`value`) AS TotalSales
http://git-wip-us.apache.org/repos/asf/drill/blob/9a2f1ba9/_docs/develop-custom-functions/020-develop-a-simple-function.md
----------------------------------------------------------------------
diff --git a/_docs/develop-custom-functions/020-develop-a-simple-function.md b/_docs/develop-custom-functions/020-develop-a-simple-function.md
index 794182c..4a4250c 100644
--- a/_docs/develop-custom-functions/020-develop-a-simple-function.md
+++ b/_docs/develop-custom-functions/020-develop-a-simple-function.md
@@ -37,10 +37,10 @@ function interface:
**Example**
- public void setup(RecordBatch b) {
+ public void setup() {
}
public void eval() {
- out.value = in1.value + in2.value;
+ out.value = in1.value + in2.value;
}
5. Use the maven-source-plugin to compile the sources and classes JAR files. Verify that an empty `drill-module.conf` is included in the resources folder of the JARs.
http://git-wip-us.apache.org/repos/asf/drill/blob/9a2f1ba9/_docs/develop-custom-functions/030-developing-an-aggregate-function.md
----------------------------------------------------------------------
diff --git a/_docs/develop-custom-functions/030-developing-an-aggregate-function.md b/_docs/develop-custom-functions/030-developing-an-aggregate-function.md
index 76a9cfe..ac28d9e 100644
--- a/_docs/develop-custom-functions/030-developing-an-aggregate-function.md
+++ b/_docs/develop-custom-functions/030-developing-an-aggregate-function.md
@@ -30,6 +30,24 @@ Complete the following steps to create an aggregate function:
@Workspace BitHolder value;
@Output BitHolder out;
4. Include the `setup(), add(), output(),` and `reset()` methods.
+
+ **Example**
+ public void setup() {
+ value = new BitHolder();
+ value.value = 0;
+ }
+
+ @Override
+ public void add() {
+ value.value++;
+ }
+ @Override
+ public void output() {
+ out.value = value.value;
+ }
+ @Override
+ public void reset() {
+ value.value = 0;
5. Use the maven-source-plugin to compile the sources and classes JAR files. Verify that an empty `drill-module.conf` is included in the resources folder of the JARs.
Drill searches this module during classpath scanning. If the file is not
included in the resources folder, you can add it to the JAR file or add it to
http://git-wip-us.apache.org/repos/asf/drill/blob/9a2f1ba9/_docs/develop-custom-functions/060-custom-function-interfaces.md
----------------------------------------------------------------------
diff --git a/_docs/develop-custom-functions/060-custom-function-interfaces.md b/_docs/develop-custom-functions/060-custom-function-interfaces.md
index af46c4a..4183ac8 100644
--- a/_docs/develop-custom-functions/060-custom-function-interfaces.md
+++ b/_docs/develop-custom-functions/060-custom-function-interfaces.md
@@ -55,7 +55,6 @@ The following example shows the program created for the `myaddints` function:
import org.apache.drill.exec.expr.holders.Float8Holder;
import org.apache.drill.exec.expr.holders.IntHolder;
import org.apache.drill.exec.expr.holders.VarCharHolder;
- import org.apache.drill.exec.record.RecordBatch;
public class MyUdfs {
@@ -65,7 +64,7 @@ The following example shows the program created for the `myaddints` function:
@Param BigIntHolder input1;
@Param BigIntHolder input2;
@Output BigIntHolder out;
- public void setup(RecordBatch b){}
+ public void setup(){}
public void eval(){
out.value = input1.value + input2.value;
@@ -118,7 +117,6 @@ The following example shows the program created for the `mysecondmin` function:
import org.apache.drill.exec.expr.holders.Float8Holder;
import org.apache.drill.exec.expr.holders.IntHolder;
import org.apache.drill.exec.expr.holders.VarCharHolder;
- import org.apache.drill.exec.record.RecordBatch;
public class MyUdfs {
@@ -128,9 +126,9 @@ The following example shows the program created for the `mysecondmin` function:
@Workspace BigIntHolder min;
@Workspace BigIntHolder secondMin;
@Output BigIntHolder out;
- public void setup(RecordBatch b) {
- min = new BigIntHolder();
- secondMin = new BigIntHolder();
+ public void setup() {
+ min = new BigIntHolder();
+ secondMin = new BigIntHolder();
min.value = 999999999;
secondMin.value = 999999999;
}
@@ -139,8 +137,8 @@ The following example shows the program created for the `mysecondmin` function:
public void add() {
if (in.value < min.value) {
- min.value = in.value;
- secondMin.value = min.value;
+ min.value = in.value;
+ secondMin.value = min.value;
}
}
http://git-wip-us.apache.org/repos/asf/drill/blob/9a2f1ba9/_docs/developer-information/design-docs/050-value-vectors.md
----------------------------------------------------------------------
diff --git a/_docs/developer-information/design-docs/050-value-vectors.md b/_docs/developer-information/design-docs/050-value-vectors.md
index 828376a..87bc82d 100644
--- a/_docs/developer-information/design-docs/050-value-vectors.md
+++ b/_docs/developer-information/design-docs/050-value-vectors.md
@@ -20,7 +20,7 @@ Reading a random element from a ValueVector must be a constant time operation.
To accomodate, elements are identified by their offset from the start of the
buffer. Repeated, nullable and variable width ValueVectors utilize in an
additional fixed width value vector to index each element. Write access is not
-supported once the ValueVector has been constructed by the RecordBatch.
+supported once the ValueVector has been constructed.
### Efficient Subsets of Value Vectors
http://git-wip-us.apache.org/repos/asf/drill/blob/9a2f1ba9/_docs/img/connect-plugin.png
----------------------------------------------------------------------
diff --git a/_docs/img/connect-plugin.png b/_docs/img/connect-plugin.png
index db3a3ec..702da8a 100644
Binary files a/_docs/img/connect-plugin.png and b/_docs/img/connect-plugin.png differ
http://git-wip-us.apache.org/repos/asf/drill/blob/9a2f1ba9/_docs/sql-reference/data-types/010-supported-data-types.md
----------------------------------------------------------------------
diff --git a/_docs/sql-reference/data-types/010-supported-data-types.md b/_docs/sql-reference/data-types/010-supported-data-types.md
index 48397d0..3fda60b 100644
--- a/_docs/sql-reference/data-types/010-supported-data-types.md
+++ b/_docs/sql-reference/data-types/010-supported-data-types.md
@@ -61,6 +61,13 @@ The section [“Query Complex Data”]({{ site.baseurl }}/docs/querying-complex-
* ["KVGEN"]({{site.baseurl}}/docs/kvgen/)
* ["FLATTEN"]({{site.baseurl}}/docs/flatten/)
+## ANY Type
+The ANY type is a key technological advance in Drill that enables it to address late typing problems. Drill uses the ANY type internally and you might see references to ANY in the output of the DESCRIBE or other commands. You cannot cast a value to the ANY type in this release.
+
+Using the ANY type, the parser postpones the problem of resolving the type of some value until the query is actually running. At that point, Drill has an empirical schema available for each record batch to use for final code
+generation and optimization. If the empirical schema changes due to
+changes in the data processing, Drill regenerates the code as necessary.
+
## Casting and Converting Data Types
In Drill, you cast or convert data to the required type for moving data from one data source to another or to make the data readable.